vSphere and physical device hot plug

If you don’t care about physical hardware I/O hot remove / hot add / hot plug support in VMware hypervisor software, please move along, there’s nothing of interest to you here. If you want to know about VMware hot plug support for PCI and PCIe option cards and other physical devices, please continue reading.

I’m a software engineer for a gigantic computer software corporation that also designs and sells enterprise servers. We support VMware vSphere virtualization software on this hardware. It’s my job to ensure the latest versions of ESX and ESXi install, boot, and otherwise work correctly on our latest hardware. I talk with engineers and architects at VMware on a regular basis. I see their roadmaps, along with the roadmaps of our hardware partners. We work together to ensure chipsets that will be released next year and that we’re designing with now will have software support when everything is released six months from now.

I also frequently get questions from colleagues and assorted random tech nerds about VMware support for physical device hot plug. When I answer “VMware does not support physical device hot plug,” they always respond with “Send me a link to a web page for proof.” Never mind that I am an authority on the topic. They want a web page, so here’s your web page. VMware makes no statement of support for physical hot plug, but neither do they say they don’t support it. It’s not secret information, but neither does VMware brag about this lack of support.

VMware supports virtual device hot plug for their virtual machines. They even support virtual CPU and virtual RAM hot add.

VMware does not support hot remove and hot add of physical I/O devices. They do not support hot remove and hot add of physical memory, or physical processor sockets.

We go through the substantial engineering effort to create hot swap hardware to enhance hardware maintenance. You, as the system administrator, get a predictive failure alert telling you the network controller will fail soon. It’s a cool trick for the OS to remaining running as you swap out the defective I/O hardware and replace it with something brand new. Another cool trick is adding additional I/O while the OS is running. Windows, Solaris, and the various enterprise distros of Linux all support hot plug to one degree or another (assuming, also, that the hardware supports hot swap). VMware, alas, doesn’t support hot plug at all.

I’m not sure VMware will ever support hot plug for physical I/O devices. Here’s why: vMotion. Why deal with the disruption of changing hardware when you can quickly vMotion your running virtual machines to a completely new physical host? It’s a cool feature that works like magic, and VMware makes a big deal about it (rightly so, IMO), plus another machine means more licenses for VMware to sell. The software engineering to make I/O hot plug work is kind of tricky, and there are support issues dealing with BIOSes, servers, and I/O devices from myriad vendors. Why deal with the headache, especially when success means less revenue in licensing fees?

VMware does support hot swap of ancillary hardware such as storage (disks, etc), power supplies, fans and so forth. To replace PCI and PCIe option cards, however, you must first power off the system, pull out the old, push in the new, and power back on.

3 Comments

  • October 5, 2011 - 4:40 pm | Permalink

    Thank you, and well said. I must be one of those bad consumers, because I don’t wonder why ESXi doesn’t support that. Then again, I am so excited that ESXi works as awesomely as it does that I don’t really question its functionality. As a software engineer, I am often asked why my app doesn’t support x functionality. Trying to delve into the tradeoffs is just speaking Greek to them.

    That was a long-winded way of saying: I know your pain, brother.

  • Andrew Downes
    December 8, 2011 - 7:22 am | Permalink

    Thanks Richard, great blog item, much appreciated. Another reason we need more competition in x86 hypervisors.

  • David Egolf
    July 23, 2014 - 3:16 pm | Permalink

    As of July 2014 is this still the case? Has ESXi added any support for hot plug with recent versions?

  • Leave a Reply