HP 3PAR and support for ODX

Over the weekend my fellow MVP Flemming Riis from Denmark contacted me about a new customer bulletin (c04205854) from HP about HP 3PAR StoreServ Storage with HP 3PAR OS 3.1.2 which warns for a serious issue when used with Windows Server 2012 and Windows Server 2012 R2. In the past year we saw and heard of several cases with Live Storage Migration causing corruption of VHDX caused by a bad implementation of ODX in HP 3PAR’s firmware. Previously we advised users to disable ODX at the operating system level as a temporary workaround. Now it seems, HP has found the issue and offers a patch for HP 3PAR OS.

Description

An issue has been discovered with the HP 3PAR OS and the use of Windows Server 2012 Off-loaded Data Transfer (ODX) commands, which may result in a number of blocks incorrectly zeroed beyond the requested range under certain conditions. This issue is not observed with ODX disabled.

Detailed analysis

When the Write Using Token requests using Block Device Zero Token is greater than 16 MBs in size, and the size is not a multiple of 16 MBs, a number of blocks may be zeroed beyond the end of the requested range. Disabling ODX eliminates the use of Write Using Token requests using Block Device Zero Token.

Scope

Windows Server 2012 or Windows Server 2012 R2 hosts with ODX in use with HP 3PAR StoreServ Storage running HP 3PAR OS version 3.1.2 GA, 3.1.2 MU1, 3.1.2 MU2, 3.1.2 EMU2, or 3.1.2 MU3.

Resolution

Upgrade the HP 3PAR OS on the HP 3PAR StoreServ Storage to 3.1.2 MU2 or later if running a lower HP 3PAR OS version. Next apply the patch as follows:

  • For 3.1.2 MU2 and 3.1.2 EMU2, apply Patch 11 followed by Patch 36.
  • For 3.1.2 MU3, apply Patch 30.

Workaround

Disable ODX on the Windows 2012 or Windows 2012 R2 hosts. To disable ODX:

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name “FilterSupportedFeaturesMode” -Value 1

A server reboot is required for all servers in which the registry value is modified.

If you have an opportunity to test this patched version of HP 3PAR in your own environment, I would strongly advise you to do this first!

You can find the customer advisory here

8 Comments

  1. March 26, 2014    

    Thanks guys, I’ve been plagued by corrupt VMs for months; we’ve got the patch install scheduled for our 3Par, hopefully this will sort it.
    One thing we have been forced to find out is just how well our new DPM platform is running thanks to this issue!

    • March 26, 2014    

      Hi Ross, good luck with the update. Let us know what the impact was.
      And certainly the effect it has on DPM and host level backups.
      Keep in touch!
      Hans

      • April 15, 2014    

        Hi Hans,

        I thought I’d let you know that we (finally) got HP to upgrade our 3par to resolve this issue, and it does appear to be fixed.

        We haven’t seen any corruption since the upgrade and backups with DPM are running very well. A restore of a 20GB vhdx to a new location takes 3 minutes, which we like!
        We have other odd problems though, which I’ll comment about elsewhere :-)

  2. March 31, 2014    

    Hello,
    last week 3PAR firmware 3.1.3 was released which fixes some ODX issues and also solves a problem that I/O is better
    distributed over the available cores. I have at least one customer who can no longer reproduce his ODX problem (locking up when copying one file from one 3PAR volume to another) is resolved.

    Cheers,
    Thomas

    • March 31, 2014    

      Hi Thomas,
      That’s very good news and thanks for reporting back.
      Best regards, Hans

  3. April 4, 2014    

    Hello,

    we are experiencing the same problem (just after we solved one problem with LiveMigration due to Emulex) and could also track it down to ODX. It occurs every time the DPM 2012 R2 does it’s snapshots, disabling ODX and rebooting every cluster node solves it. But we do not want to go for production with the handbrake pulled up.

    Now we are still waiting for the HP 3PAR SPS to get our storage updated. Until it’s done we try to keep calm and wait. Seems to be annoying for many customers as they did not get it done within one week since ticket creation.

    Kind regards from Switzerland,

    Matthias Petz

    • April 4, 2014    

      Hi Matthias,
      I’m afraid there are a lot of customers asking for this update. Hopefully your patience will be rewarded.
      -H

  4. Masih's Gravatar Masih
    April 29, 2014    

    We too are awaiting HP to upgrade our FW to 3.1.3 but kinda in a dilemma over whether to go for patching the existing 3.1.2 MU3 with Patch 30 or to go for the entirely fresh 3.1.3 FW.

    We have exactly the same issue where our Cluster Shared Volumes running on a Windows Server 2012 go offline, causing a “status I/O timeout “c00000b5)” in Events Log. It happens when DPM 2012 is performing snapshot backup.

    Any inputs would be helpful.

  1. Virtualizing Intensive Workloads on Hyper-V, Can It Be Done? | Working Hard In IT on June 26, 2014 at 10:04

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>