Over the last couple of days I’ve been performing ESXi 4.0 to 4.1 build 320137 upgrades. During the upgrade process today one of my servers had a hiccup and the web services was not responding. After a couple of reboots and non-responsive web services I iLO’d in (it’s an HP server) so I could get to the ESXi DCUI (direct console user interface), AKA the yellow screen. To my shock when I pressed F2 I got:
Authentication Denied: Direct console access has been disabled by the administrator for contoso.net.
At first I thought OK, maybe someone enabled lockdown mode and I didn’t know it. Checked a few things, nope, no lockdown mode. After more poking and prodding, I found the root cause of the problem. But it’s a mystery to me why this is occurring. The only consistent theme is that the server was on ESXi 4.0 and they were upgraded to ESXi 4.1 build 320137.
So what was the problem? New to ESXi 4.1 is the Security Profile configuration screen. Here you can stop/start several low-level system services. On 25% of my upgraded boxes the “Direct Console UI” service was in the stopped state as shown below.
The solution is to reconfigure the service to Start and Stop with host, which is the ESXi 4.1 default configuration. After I started the service, viola, DCUI access was restored!
Since this happened on several boxes, but not all, I’ll chalk it up to another VMware bug. So in my upgrade procedures I’m adding a check to verify the service status before we bless an upgrade as being complete.
