ESXi 4.0 Security

I’ve been wanting to put up a post on ESXi security for some time now, and 2 recent posts have kicked me into action – this one from Scott Lowe about setting the root password on ESX and this announcement about the vSphere blogging contest :D. So have your caffeinated beverages and pizzas at the ready, we might be here a while!

Strangely, and disappointingly might I add, there is no vSphere 4 Security Hardening Guide available currently (UPDATE: A draft version has now been published – I’d like to think the release had something to do with this post, but in reality I’m sure it is merely a coincidence!) – the only published security guidance we have from VMware is the old one for VI 3.5 and the Security section in the ESXi Configuration Guide. While this situation sends a negative message to security types who may wonder if this lack of documentation is indicative of VMware’s approach to security (vSphere has been out for nearly 9 months now), when it comes to ESXi 4 not a lot has changed so most of the ESXi parts in the old 3.5 document apply equally to ESXi 4 and are covered in the ESXi Config Guide anyway. Before anyone flames me about that comment, I don’t believe VMware has a weak stance with regards to security and I am NOT one of the aforementioned security types – I have had to deal with them however, so I am speaking from experience.

I’ll try to break the post up a bit by talking about 3 specific areas: Root Password Management, Logging, and Monitoring. I am not going to regurgitate what is already documented by VMware (ie port requirements, config files to monitor etc) or stuff that my grandma knows (like configuring NTP, using low privilege accounts wherever possible etc) and I’m firmly in the camp that believes only the foolish enable SSH on ESXi in production environments so I’m not going anywhere near that. And remember the focus here is ESXi security – not vCenter or Virtual Machine security. Let’s get started.

Root Password Management
With ESXi, VMware introduced a new feature called “Root Lockdown Mode”, which strangely enough chops all remote access via the root account. Not that there is much you can access remotely… it’s basically only the API and CIM services (remember, SSH on ESXi is a no-no!).

Whether this is a good or bad thing really boils down to the tradeoff between enhanced security and operational pain. In my humble opinion, there really is no circumstance under which anyone needs to logon as root after an ESXi host has been added to vCenter. In the rare occurrence of the host having issues, it’s not too painful to logon as root directly at the console via HP iLO / Dell DRAC / IBM RSA / IP KVM etc and do the necessary. There however many tools and scripts out there that only work via remote root access, and there has been the occasional bug associated with the enablement of Lockdown Mode, so the decision to enable it or not might not be straight forward.

But if you really can’t enable lockdown mode, then how do you manage the root passwords like you might on *nix based platforms? This is potentially the reason lockdown mode came into existence. Every large enterprise I have worked at has had a policy that required local admin / root passwords to be changed frequently. The degree of automation varied though – at worst it was script based, at best the company took security seriously and implemented the proper tools. But throwing ESXi into the mix introduces some complexity as the only option to change the root password is via the API. So I wrote a tool to help with this. Of course I know you can easily do this via PowerCLI, however there are times when it may not be possible to use it. So I wrote a little console utility that you can use with batch files or systems that have their own esoteric scripting language whereby calling a standalone executable is just easier. Using all the creative powers in my possession, I came up with a name of…

You can download esxipasswd here be sure to read the “installation” instructions and just run the executable with no options to see how to use it. Any feedback is most welcome, either ping me on Twitter (@vinternals) or email vinternals at gmail dot com.

So if the only reason you have enabled Lockdown mode is because you didn’t have an easy way of managing the ESXi root passwords remotely, now you can. And I highly recommend changing the root password frequently, regardless of the size of your ESXi deployment.

Like all *nix based environments, ESXi provides syslog capabilities. By default ESXi is only configured to output locally, but pointing it at a remote syslog server is very easy using the Set-VMHostSysLogServer PowerCLI command (last time I checked, this only works with ESXi – you have to configure syslog on fat ESX like you would on a standard *nix box). Forwarding syslog to a central server should be considered mandatory in any environment.

Of critical importance is logging root or other privileged account logins, both failed and successful. Monitoring failed logins only is not good enough… how do you know what happened after the last failed attempt – did they give up or get in? There are several key messages to lookout for in syslog, the following is not an exhaustive list but it is a good start:

Remote root login via API (vSphere client, PowerCLI etc)
Local4.Warning Hostd: Rejected password for user root from XXX.XXX.XXX.XXX
Local4.Info Hostd: Accepted password for user root from XXX.XXX.XXX.XXX

Tech Support Mode Invoked
Auth.Error getty[9729]: VMware Tech Support Mode successfully accessed

Root login via Tech Support Mode on local console
System0.Notice DCUI: pam_unix(dcui:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=root
System0.Info login[9729]: pam_unix(login:session): session opened for user root by (uid=0)

Root login via Direct Console User Interface (DCUI) on local console
Auth.Error DCUI: authentication of user root failed
Auth.Error DCUI: authentication of user root succeeded

Unknown user login attempt via API (vSphere client, PowerCLI etc)
Local4.Warning Hostd: Rejected password for user erwerw from XXX.XXX.XXX.XXX
Local4.Info Hostd: [info β€˜Vmomi’] Throw vim.fault.InvalidLogin

If you’re a Windows kind of guy, you can use the free Kiwi Syslog server to investigate what syslog messages are relevant to you with ESXi. If you’re among the eunuchs UNIX folk… well I’m sure you know what to do ;).

Event Monitoring in vCenter
Syslog provides a certain level of monitoring, but any malicious user worth their salt will disable things like syslog as soon as they get root access. But it’s less likely that they would think to try and disable any monitoring that is occurring via the VMware API. There are a plethora of host events that can be tracked in VC – the obvious ones are UserPasswordChanged, AccountCreatedEvent and UserAssignedToGroup.

Configuring alarms in vCenter to track specific events like these isn’t straight forward however, you may need to do some PowerCLI hacking to get the events you want into an Alarm. LucD‘s blog is one of the best sources for this kind of PowerCLI wizardry, and as has happened so many times in the past few years he has beaten me to scripting up something – in this case, creating event based alarms for specific events that aren’t necessarily available via the UI. So have a read of this post on event based alarms to get a feel for what is involved. And don’t forget to set an appropriate trigger – ideally when a security related alarm goes off, you want someone other than the VI admins to know about it. Separation of concerns and all that.

Well you’ve made it this far, congratulations. Of course there is more to ESXi security than what I have covered above, but as I said from the outset – my aim was not to regurgitate what is already documented by VMware. Hopefully you have found some of the info useful, I’ll add the contents of this post to my ESXi Mastery section in due time where I’ll also be able to expand on it.


7 Responses to “ESXi 4.0 Security”

  1. Duncan Says:

    Great stuff Stu!

  2. ESXi 4.0 Security | Virtualization Spotlight Says:

    […] We try to keep an eye on all the community blogs here at Virtualization-Spotlight. Today our eye was caught by the vinternals blog’s article ESXi 4.0 Security. […]

  3. Greg Says:

    A “global root password changer” would be a good feature to add to vCenter. I work for a financial and I am still running 3.5 U4. I have spent more time hardening the console and fixing things that come out of compliance after patching then I care for . I am seriously considering standardizing on “I” when I migrate to vShpere and leveraging PowerCLI.

  4. Jaleh Rezaei Says:

    Hi Stu,

    This is a very insightful article. Thanks for taking the time to post it.

    ESXi product marketing manager, VMware

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: