BLOG

BLOG

Fix VMWare ESXi 6.* slow disk performance on HP b120i controller

May 27, 2017

Even though this post is fairly dated, I will leave this post online just in case anyone still has an HP ProLiant ML310e or MicroServer Gen8 server with terrible disk performance. It had always been working fine until I started using ESXi 6 and 6.5 on it. For some reason the disk performance was suddenly just atrocious. Imagine getting only about 25 megabytes per second of write speed on it. I'm on a gigabit network and copying files from my Windows computer to a Linux Samba instance, and it started out at 110MB/s, then tanked to 25MB/s after half a minute. Horrible. And totally unexpected, because with ESXi 5 and 5.5, it worked just fine, staying a steady 110MB/s all the way through. So what gives?

Turns out that for some reason the HPVSA driver for ESXi has some issues. I have no idea why as I haven't dug into it in detail. All I do know is that when I replace it with an older driver, my server's disk performance is fine again. This isn't a fluke. This is a situation I can reproduce on multiple servers with various disk configurations. For example one of my HP MicroServer gen8 servers has 2x8TB WD RED disks in RAID-1, and gave me around 30MB/s performance at best. When I uninstalled the new HPVSA driver (version .101 or 102 or so I believe) and just installed the old driver (version .88), I was able to get around 170MB/s write performance. No other settings were changed at all.

Affected configurations

I don't know if this issue exists in other configurations but what I've seen so far it affects these configurations;

  • VMWare ESXi 6.5.0
  • HP MicroServer gen8
  • HP ProLiant ML310e gen8
  • both servers run controller: HP ProLiant b120i SmartArray

Let me just say that, yes, I know these are entry-level servers. But with a different driver I get normal performance and with an up-to-date driver I get shit performance. Whatever the problem is, below is how to solve it.

Files you'll need

First of all, I take no responsibility for any damage or problems if you follow these instructions. They come as-is without any guarantees. I've used them myself and it works perfectly, though.

Before you continue check that you have ESXi 6.5 Build 5310538. I've noticed if you have an older version, that uninstalling/replacing the driver may make the HP VSA driver fail to load. Then you will still be able to access your hard drives but only through VMWare's own AHCI/SATA driver, and will ignore the RAID configuration. This is bad if you have data on the RAID drives already; you can't access them then. Also, the RAID won't actually be used then. This can be fixed however by upgrading ESXi and uninstalling/reinstall the driver again if you run into this issue. See this page on instructions on how to upgrade. This guide can be used to upgrade ESXi HPE 6.5 older build to ESXi 6.5 20170404001-standard which is the one that corresponds to build 5310538. If you had replaced the HP VSA driver before doing the upgrade to ESXi, you'll need to uninstall/reinstall the .88 driver again and reboot again before it works again.

You need the old driver. You can get it from this official Hewlett Packard Enterprise page here:
http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_dcfe1ad1b6ae442e8ed665c595

It's easiest if you can just download the file directly to your ESXi server. The direct download link is below. Note that I stripped https:// and replaced it with http://. The reason for this is that the 'wget' program which allows you to download files directly off the Internet to the ESXi server is simplistic; it only does http:// and doesn't support https://.
http://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1749737361/v98767/hpvsa-5.5.0-88.zip

Sometimes they change their website and the link doesn't work anymore. If it doesn't try looking for hpvsa-5.5.0-88.zip on Google. Or use the local copy I have stored here on my blog. My local copy forces https:// though so you can't direct download it to your ESXi server. But you can download it to your desktop computer and then copy it to your ESXi server using a program such as SCP or WinSCP.
https://www.johandraaisma.nl/wp-content/uploads/hpvsa-5.5.0-88.zip

Enable SSH on the ESXi installation. If you have ESXi 6.5 you can do it by going to the web interface and logging on, and then rightclicking "Host" and selecting "Services" > "Enable Secure Shell (SSH)". Or you can do it at the console by pressing F2, logging in, and going to "Troubleshooting options" and enabling it there.

Stop all virtual machines before you continue.

Step-by-step solution

  1. Open an SSH session to your ESXi server installation and log on as root. Same password as for the web interface / console.
     
  2. Go into "Maintenance mode". Note that all virtual machines must be stopped first.
    esxcli system maintenanceMode set --enable true
  3. Allow outgoing HTTP requests through the ESXi firewall, to retrieve the necessary file.
    esxcli network firewall ruleset set -e true -r httpClient
  4. Go into /tmp folder so we don't leave a mess.
    cd /tmp
  5. Use wget to download the .zip file from HP website. Note that I'm using http:// and not https://.
    You can also use a program such as WinSCP to copy the file directly from your desktop PC to the ESXi server.
    wget http://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1749737361/v98767/hpvsa-5.5.0-88.zip
  6. Unzip the file so we can get the .vib driver file we'll need.
    unzip hpvsa-5.5.0-88.zip
  7. Copy the file to a location where the installer can find it easily.
    cp scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib /var/log/vmware/
  8. Uninstall the currently installed scsi-hpvsa driver. Be patient. Takes a minute.
    esxcli software vib remove -n scsi-hpvsa -f
  9. Install the old driver we just downloaded. Be patient. Takes a minute.
    esxcli software vib install -v file:scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib --force --no-sig-check --maintenance-mode
  10. Now reboot your server.
    reboot
  11. Remember to disable "Maintenance mode". You can do this in vSphere Client or in the web interface.

 

Credit

I originally found the solution here, through a post on Dr. Sven Seiler's blog. I added a few steps that were missing:

https://seiler.it/solving-performance-issues-with-esxi-6-gen8

34 Comments

  1. Johan

    If you run into a problem where the RAID arrays are not visible anymore; don’t worry, the data is still there. It’s just a matter of convincing the hpvsa driver to load properly. One issue I’ve found is that when I look under Storage > Adapters, that my hard drives were listed there, but not as RAID drives. Just as standard SATA/AHCI drives. In the column ‘drivers’ I saw that it wasn’t using the hpvsa driver that I downgraded, but was using vmw-ahci instead. What I did to resolve that was disable the vmw-ahci driver with this command:

    esxcli system module set –enabled=false –module=vmw_ahci

    (ps: above command is with two dashes not a single dash – wordpress messing up the formatting)

    And uninstall the hpvsa .88 driver and rebooted.
    Then I reinstalled the hpvsa .88 driver and rebooted again.

    Then hpvsa loaded properly and my datastores became visible again.

    Also, I found that I had to update to 20170404001-standard (build 5310538) for this to work at all; on an older build it wouldn’t load the hpvsa driver no matter what I did.

    • Karsten

      Thank you, this did the trick for me. “hdparm -tT” went up from 20 MB/s to over 100. I used HP’s custom version of ESXi 6.5.0 as a basis. After installing the hpvsa .88 driver, the adapter showed up without any storage, but after disabling vmw_ahci it worked on my gen 8 microserver.

    • CS

      This actually works.
      Thanks!

    • Fischje

      Yes this worked for me too.

    • woocash

      You saved me a lot of time Sir. Thank You!

  2. David Waters

    Thank you for an excellent solution. I was getting only about 10MB/s transfers over Gigabit Ethernet from the SSD’s in my ESXi host using the HP-supplied hpvsa-5.5.0-100 on ESXi 6.0 U2. Now it’s up to 95MB/s using the -88 driver. You can’t get much faster over 1GbE! Thanks again.

  3. Alex

    The downgrade to scsi-hpvsa-5.5.0-88 didn’t worked for me on 6.5.0 Update 1 (Build 5969303). Performance is still poor :(

  4. dennis

    I read today that:
    6.5.0 Update 1 (Build 5969303) should solve the issues
    see website http://www.nxhut.com/2016/11/fix-slow-disk-performance-vmwahci.html

    I had issues as well on my HP Microserver gen8 and a quick check showed me that downloading something onto a VM was downloaded with 18MB/s with a capping at 10MB/s it seems indeed solved. But I am not sure yet ;)

    • Johan

      Let’s hope it actually resolves it, I’m going to be finding out shortly.

      • Gary Brooks

        So I assume the hpvsa issue still exists and requires a backrev driver even on 6.5U1 or were you able to get it to work otherwise? I have not.

        • Johan

          No for some reason my usual methods of upgrading ESXi from one version to the next just fails with some lack of disk space message, even though there’s terabytes free. I was never able to update to that version and actually test it. I will revisit this eventually.

    • Bastian

      Not solving this specific issue, cause the hp ms gen8 doesnt use the vmw-ahci driver since it got its own driver from HP. The discribed solution by Johan workes perfectly fine for the new hp esxi 6.5U1 image. Tested it today!

  5. Joost

    Not exactly the same config (B140i ML110 Gen9), but same performance issues on the latest HPE ESXi 6.5. I think I’ve figured out whats going on..

    When you first write to a thick provisioned disk, it’s very slow (the mentioned max 10MB/s and the guest become very sluggish). But as soon as you rewrite the same amount of data, it will do at very decent speeds (70-100MB/s sequential write on 2x WDC Gold 2TB RAID1, seems reasonable). Even after a reboot of the host that specific part of the virtual disk keeps being written fast, so there is caching involved. Not sure why this is, must be something like (image) sectors being allocated on first write, which seems a very costly operation in ESXi 6.5 / latest driver?

    I also thought it had something to do with snapshots, but that’s not the case, clean installs behave the same. With thin provisioned disks the performance is always bad, that has probably something to do with the shrinking/expanding behavior. So I think you’d better use thick disks and just need to wait for them to be written, or just write all empty space once before usage.

    Btw I tested with both Windows and Linux guests, there is no real difference.

    • Johan

      Thanks for the additional information, Joost, it is interesting to see the differences in thin/thick and eager/lazy zeroed performance. This however was already known to me. But what you describe with being slow at first, and then speeding up, is not what is happening here on ESXi 6.5 with these HP machines. The performance is in fact the other way around; until the cache fills up, write performance is fine. Then the cache is full (about a gig or so) and then suddenly performance stays at a constant low level. There’s no way to improve that low performance at all, not by removing and rewriting. The only solution is to mess with the drivers here.

  6. Ilie

    The article was very useful for me. Thank You very much!!!

  7. Alex

    Thanks a lot !!!!!! This solve my issue.

  8. Maurice

    Beste Johan,
    Heel hartelijk dank voor de goede beschrijving.
    Met vriendelijke groet,
    Maurice.

    Dear Johan,
    Thank you very much for the very good description.
    Kind regards,

  9. Ben

    Thank you for this guide, unfortunately it does not work for my installation:

    You are running HPE Customized Image ESXi 6.5.0 version 650.10.1.5 released on October 2017 and based on ESXi 6.5.0 Vmkernel Release Build 5310538.
    Version 6.5.0 Update 1 (Build 6765664)

    I follow exactly the steps as you named them… ESXi seems to be able to both remove the 102 and install the 88 driver… at least it says at both steps 8 and 9: “The update completed successfully, but the system needs to be rebooted for the changes to be effective”.

    After that I reboot and during Startup get a purple diagnostic screen with “The system has found a problem on you machine and cannot continue. Two filesystems with the same UUID have been detected. Make sure you do not have two ESXi installations”.

    Of course I don’t have two installations but it seems as ESXi sees both disks from my RAID1 instead of the mirror and thinks that there are two installations.

    Then I reset the server, ESXi boots up again – but with the 102 driver and slow performance again.

    Any ideas what goes wrong?
    Thanks in advance for you assistance :)

    • Johan

      I see you are running the Update 1 version.. when I wrote this post it was with the ordinary 6.5 release, no Update 1. It’s possible that things have changed on Update 1.

      In any case, yes, I agree that it seems that perhaps it’s seeing both drives from the mirror, and that would most likely be caused by the vmw_ahci driver loading before the HPVSA driver. Interestingly when this happened to me on ESXi 6.5 it didn’t cause a purple screen. This might be a change in Update 1. To resolve this, I believe you should disable the vmw_ahci driver first before loading the new (old) HPVSA driver. If you look at my very first comment on this post you’ll see how to do this.

  10. Dave

    This worked for me as well. Thanks so much for taking the time to publish this article!

  11. Shawn

    Hi Johan,

    The Question pertained to:

    service lirc stop >/dev/null 2>&1
    sleep 1 >/dev/null 2>&1
    modprobe -r lirc_rpi >/dev/null 2>&1
    modprobe lirc_rpi gpio_in_pin=12 gpio_out_pin=16 >/dev/null 2>&1
    service lirc start >/dev/null 2>&1

    “The above script I named ir1.sh and made executable with chmod +x. Now whenever I run it, my transmitter GPIO port is GPIO16. I have an ir2.sh and ir3.sh as well that have as gpio_out_pin the numbers 20 and 21.”

    If you look at the GPIO overview of the Raspberry Pi 2 you can see that they’re right next to eachother (pins 36, 38, and 40, with a ground on pin 34).

    I’m trying to figure out how you were able to name your scripts and where did you place them?

    LIRC has certain commands. For example:
    irsend SEND_ONCE ___ KEY_POWER etc. so how would you connect your script to the commands? I can work LIRC no problem, the issue is that i want to control multiple devices using multiple GPIO pins like you did in the article?

    Please email me when you get a chance. Thanks.

    • Johan

      Right so as you can see in those instructions, I am unloading the entire lirc_rpi module, and then reloading it on a specific pin. So whenever I need to switch, I unload the module, and reload it on the appropriate pin, and then run lirc. I know, fugly as hell, but, it works, and I was not able to get it working any other way with multiple different pins at the time. Perhaps the situation has improved now since then.

  12. Ian Hadlington

    Thanks Johan

    This seems to have worked well for me [HP ML310e v2 gen 8 with Esxi 6.0].

    It’s early days yet, but datastore latency seems to have dramatically improved. All everything has booted up!!

    Thanks for the info and directions

  13. Dmitry

    Thanks Johan!

    It’s work for me for esxi 5.5 and Microserv Gen8 SmartArray b120i (raid10 4*3Tb)

    Speed increased from 22MB/s to 75MB/s!

  14. Gerard Achieve

    Hi I have this server running Win Server 2012 2 x 500GB 7.2k Drives Mirrored. The Storgae is down to 66GB, so we were advised to purchase 2 x 2TB Drives 658102-001.

    How do I add these new drives to the server?

    • Johan

      Kind of an odd question to ask here but I happen to know the answer. You’ll likely want RAID-1 fault tolerance on those 2x2TB drives so you end up with a total of 2TB extra usable space. I won’t provide you with ongoing support on this but I can at least tell you the steps:

      Shut down the server normally.
      Open the front cover.
      Find the 2 empty bays to put the new drives in, and slot them in.
      Power the server on, and either use iLO to see the console, or attach a monitor/keyboard/mouse to see the console of the server.

      During the boot sequence, a message about ProLiant SmartArray B120i will show up, and will briefly show a message to launch the ACU (Array Configuration Utility). Usually you have to press F5 to launch it. It should then boot into the ACU and you will be able to configure the storage controller. Add a RAID-1 array and assign the two unused 2TB drives to it. You can accept default settings, they’re usually fine. Once that’s done you can exit the utility and boot the server normally.

      Now it’s just an additional storage volume you can format and start to use. If you’re using ESXi then you can go to the storage section and add new storage, then format this 2TB storage, and add an extra harddrive to the Windows Server 2012 virtual machine. You can then use Disk Management in Windows Server 2012 to format the drive and give it a drive letter. If you don’t use ESXi but Windows Server 2012 runs directly on the server then just go to Disk Management and format the drive and give it a drive letter.

      Alright, good luck.

  15. favas

    Hello All!

    I am interested in buying a Proliant ML310e G8 v2, which, as far as I am concerned, has the same storage adapter.

    I was doing some research on how this server performs with ESXi, and to my big surprise I found this thread. I was expecting no issues at all, but here we are… Thanks to Johan btw for the workaround!

    Has anyone tested with ESXi 6.7?
    If yes, is the default disk performance the same?

    • Johan

      Sorry, haven’t tested 6.7 on it yet, although I have a server at home that I could test it on. I might do that. In any case, there is a solution and difference between 6.5 and 6.7 isn’t so big so same fix will probably work if it is necessary.

      • Johan

        I tested 6.7 HPE custom image. It doesn’t even have the driver anymore. I had to disable vmw_ahci and install the HP driver. Then it worked fine.

  16. Hamid Ghouli

    Thank you for your great post.

    My Question is the latest esxi is 6.0 u3 for ProLiant ML310e Gen8 v2, Then how could you install esx 6.7 on it?

    can you guide me?
    and can you give me a link for download compatible esxi 6.7 (latest update) for ProLiant ML310e Gen8 v2?

    • Johan

      I always install ESXi on a USB drive in the server, and keep it completely separate from the datastores. That way, I can install the HPE ESXi 6.7 image available from here on the USB drive, and then later add any necessary drivers to access the datastore. Technically I believe VMWare doesn’t provide support on this but I don’t care, it works fine.

      • Hamid Ghouli

        Thanks for guide, Could you tell me what are necessary drivers and how to add them ?

        • Johan

          Just the one in this guide, links are already in this post.