Why You Shouldn't Update vCenter If Using ESXi… Yet!

UPDATE A patch that resolves this issue has now been released – read more about it here.

Although I did enjoy a brief moment of smugness when the HP related ESX 4 Update 1 problems arose, it was cut short by another issue effecting vCenter 4.0 Update 1 and ESXi.

VMware haven’t povided much information about the cause of the problem thus far, there is simply a disclaimer on the Support & Downloads page saying not to apply vCenter 4.0 Update 1 if ESXi hosts are being managed, and pointing to a KB article that doesn’t go into much depth. So allow me to elaborate on the cause of the problem, so we can all be a little more enlightened.

The problem boils down to the way updates are performed on ESXi, and are a side of effect of it’s non-reliance upon a local disk based filesystem. Basically, anytime a piece of non-OS software is updated on ESXi, it is done so via a 64MB ramdisk. Such updates could be for the VC agent, the HA agent, or even 3rd party modules such as PowerPath or the Nexus 1000V.

By default the vmkernel tries to keep 6% of physical memory free at all times. When free physical memory drops below this level, it starts employing various memory management techniques in order to reclaim physical memory from guests. So having 64MB of free memory for agent updates isn’t so much the problem – the catch is that this memory has to be contiguous in order for the ramdisk to be created.

Memory fragmentation can occur on systems even under moderate load, as we found out when we tried to apply vCenter 4.0 Update 1 in our labs and encountered this problem. The hosts that had little or no load were completely unaffected.

Due to the synchronous nature in which vCenter attempts to update the vpxa binary on any hosts it is managing, there really isn’t a viable workaround for production environments at this stage, hence VMware’s recommendation to not apply vCenter 4.0 Update 1 if there are ESXi hosts being managed. One possible workaround is to place the hosts into maintenance mode and reboot them prior to the vCenter update, but you would obviously have to offline all the guests until the agent updates were complete.

I am told this issue will be fixed in the next set of patches (_not_ Update 2) released for ESXi, which shouldnt be too far away. Shout outs to my excellent VMware account team for helping to bring this information to light.


5 Responses to “Why You Shouldn't Update vCenter If Using ESXi… Yet!”

  1. Virtualization Short Take #32 - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers Says:

    […] provides a thorough explanation of why VMware is recommending not to install vCenter Server 4.0 Update 1 when managing VMware ESXi […]

  2. JC Says:

    A bit of an overreaction, in my opinion, when the work around is a reboot of the ESXi host.

    • stu Says:

      @JC Overreaction!?!? Upgrading VC causes _parallel_ vc agent upgrades on all hosts managed by the VC – it’s not simply a case of rebooting each host 1 at a time before applying the VC update, you would need to reboot your entire ESXi infrastructure before starting the VC upgrade, and not start any VM’s until after the agent upgrades completed. Advising customers to wait is the only course of action, it’s not an overreaction at all IMHO.

  3. Allen Crawford Says:

    Interesting, I was just told that they are NOT fixing this until Update 2. In our environment we went ahead and rebooted each host (using maintenance mode of course) and then did update 01 with no problems. In theory this should free up enough contiguous memory to do the upgrade, but we aren’t very far along with our upgrade from ESX 3.5u4 to ESXi 4, so there weren’t that many to reboot yet. Might not be very feasible for large ESXi 4 environments, so it will be interesting to see when they fix this.

    • stu Says:

      That would be disappointing! I really can’t see how that would be possible, I will double check with my sources when VMware is back in the office next week.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: