Resetting lost ESXi root password with Nutanix

The other day I was at a customer for a fresh installation of Nutanix using vSphere 6.5. And for whatever reason, when they were resetting the ESXi root password to their default, it was fat fingered. When they went to add the hosts to vCenter, they couldn’t add them since the password was wrong. So what to do? If this was a non-Nutanix environment, the only supported ESXi method of resetting a lost password is re-imaging the server. But, Nutanix has a CVM running on each node that is configured with SSH keys to access the ESXi host. We can use a private IP address and the embedded SSH keys to successfully reset.

The full process to reset a lost ESXi root password on Nutanix is:

1. ssh into the CVM on the host that has the lost ESXi root password, using the Nutanix account name.

2. Enter: ssh root@192.168.5.1

3. ESXi console: passwd root

4. If the account is locked out: pam_tally2 –user root –reset

If you then run the add host wizard in vCenter and your password doesn’t work, try rebooting the ESXi host. This procedure saved us from re-phoenixing the ESXi host.

Downgrade your ESXi Host

On rare occasions you may need to downgrade your ESXi host. In my case I was working on a VMware certification test and my host was using a newer build than what was called for. Or, maybe you run into a situation like the NFS bug earlier this year and needed to downgrade back to a stable ESXi version. Either way, it’s a fairly simple process do downgrade your ESXi host, all without losing your settings.

1. On a computer with PowerCLI installed run the following command. From the long list of profiles, pick the profile which you want to downgrade to. Some profiles have build numbers, while others have dates. So it may take a little digging to downgrade to exactly the build level you want. In my case I wanted ESXi 5.1 GA.

Get-EsxImageProfile | Sort-Object “ModifiedTime” -Descending | format-table -property Name,CreationTime

 

2014-08-12_15-01-12

2. Enable SSH on your ESXi host and then enter the following command, but replace the profile name with the one you want to downgrade to. After the profile downloads and installs, reboot the ESXi host.

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.1.0-799733-standard –allow-downgrades

And that’s it! I don’t know the official support stance on this, so just don’t go doing it on random production servers. But it saved my bacon today. If you are a Nutanix customer, this command will also work and preserves our CVM and custom ESXi host configuration information. Also remember that you might be able to use “shift r” during ESXi boot to restore your ESXi host to a prior installed version. This could negate the need to re-download the profile that you want. In my case the host was freshly imaged with a newer build so the “shift r” at boot was not of assistance.

VMware has a KB article on a similar procedure, and a support statement that you can check out here.

VMware ESXi 5.1 Patches Released

VMwareHot off the presses are some ESXi 5.1 patches. This build of ESXi 5.1 (1157734) fixes several bugs and more importantly addresses some security issues. As always in any environment, please test out the patches thoroughly before putting them into production. Each environment is unique, and issues may surface that could cause you some headaches. These bug fixes aren’t earth shattering, so I would not suggest rushing them out to production systems.

ESXi 5.1 Build 1157734

Highlights of the patch bundle included in this release are:

  • Black frames might appear around text boxes in an application running on Virtual Machine Hardware Version 8 or later. This issue occurs on virtual machines with Windows 7 guest operating system and View 5.0 PCoIP.
  • For two ESXi hosts with different host names, identical machine names are generated in the domain controller under certain conditions. As a result, the ctive Directory functionality is lost for one of the two ESXi hosts.
  • After you upgrade to ESXi 5.1 from an earlier version, attempts to power on a virtual machine with static MAC address outside the allowed range (00:50:56:[00-3f] or 00:50:56:[80-BF]) fail with the following error message: The MAC address entered is not in the valid range.
  • If a physical NIC is named using non-standard naming conventions (other than vmnic#) and is added to a vSwitch, host profile creation fails with the following error message: Invalid value chosen for active NICs.
  • ESXi 5.1 hosts might get disconnected randomly from the vCenter Server system. This issue might occur if the heartbeat thread in the vpxa agent does not receive a response from the futex_wait system call. As a result, the heartbeat thread stops responding, and the vCenter Server does not receive heartbeat messages from the ESXi hosts for several hours.
  • Upon reboot, ESXi 5.1 hosts configured to obtain DNS configuration and host name from a DHCP server displays its host name as localhost in syslog rather than displaying the host name obtained from the DHCP server. As a result, for a remote syslog collector, all ESXi hosts appear to be the same, with the same host name.
  • To prevent buffer overflow, the HPSA proc node truncates LUN details on an ESXi host.
  • This patch updates the esx-base VIB to resolve a stability issue.

As always, you can down the ESXi patches from here. The full KB article for the patch bundle is here.

VMware vSphere 5.1 Update 1..is it for you?

VMware vSphere 5.1 update 1 is probably one of the most recently anticipated updates of the VMware stack and it has finally hit the streets. For those of you following the release of vSphere 5.1, you have seen the GA release last fall, followed by 5.1.0a then a couple of months later 5.1.0b, all addressing bugs and ironing out critical installation issues.

VMware vSphere 5.1 Update 1 has a laundry list of improvements, support for new Microsoft products, and a lot of bug fixes. If you are still on vSphere 5.0 or tried 5.1 in the past and ran into problems, you definitely need to check out vSphere 5.1 Update 1. If you want a complete vSphere 5.1 installation guide, check out my 15-part blog series here. I will be updating it in the near future for Update 1. If you are running vSphere 5.1, there are a number of security vulnerabilities addressed in the update so start planning your upgrade.

VMwareKnown Issues with vSphere 5.1 Update 1

Today VMware posted a new KB warning about a vSphere 5.1 Update 1 bug, which may affect customers. The problem prevents you from logging into the vSphere Web Client using an AD account, if you AD account is a member of approximately 19 or more domain groups and the SSO service is configured with multiple domains. The KB states until a hotfix is released, DO NOT upgrade to vSphere 5.1 Update 1. In many enterprise environments a vSphere administrator may be in dozens of groups, depending on how access is controlled within the domain. Fewer customers will probably have SSO configured for multiple domains, so the impact of this issue is probably limited to larger enterprises. Additional issues include:

  • If you are using the vSphere Storage Appliance, you MUST upgrade to vSA 5.1.3 after you upgrade the rest of your infrastructure to vSphere 5.1 Update 1. vSA 5.1.1 is NOT compatible with vSphere 5.1 Update 1.
  • You can NOT use the simple installer to upgrade from prior 5.1 versions to 5.1 Update 1. You must utilize the individual installers.
  • Windows Server 2012 failover clusters are NOT supported on ESXi 5.1 Update 1. The cluster validation wizard gets stuck in an endless loop and you are unable for form the cluster.

What got updated in vCloud Suite 5.1 Update 1?

  • ESXi 5.1 Update 1 Build 1065491
  • vCenter Server 5.1 Update 1 Build 1065152
  • vSphere Data Protection 5.1.10.32
  • vSphere Replication 5.1.1
  • vSphere Storage Appliance 5.1.3
  • vCenter Orchestrator Appliance 5.1.1
  • vCloud Director 5.1.2
  • vCenter Site Recovery Manager 5.1.1
  • vSphere 5.1 Update 1 Virtual Disk Development Kit
  • vSphere CLI 5.1 Update 1
  • VMware Converter Standalone 5.1 (Download here)
  • VMware vCenter Server Heartbeat 6.5 Update 1
  • VMware vSphere Management Assistant (5.1.0.1 – April 4, 2013)
  • HP Custom Image for ESXi 5.1.0 Update 1 Install CD

You can find all of these downloads in the usual place, My VMware. You can download the updated documentation archive ZIP bundle here. The full documentation page is here.

vCenter Server 5.1 Update Release Notes

vCenter 5.1 Update 1 is more than just bug and security fixes, it incorporates a number of newly supported operating systems and database back-ends. You can find the full release notes here. Below is just a tiny faction of the new features and bugfixes.

What’s New?

  • vCenter Server can be installed on Windows Server 2012
  • vCenter can use Microsoft SQL Server 2012 and SQL Server 2008 R2 SP2
  • Guest operating customization support for Windows 8, Windows Server 2012, Ubuntu 12.04 and RHEL 5.9
  • Removed vRAM usage limit of 192GB on vSphere Essentials and Essentials Plus

Resolved Issues

A lot of bug fixes are included, but a few highlights include:

  • Better error reporting when accidentally updating the Admin or STS service with incorrect protocol parameters. It will now tell you what you botched up.
  • Number of security patches including Java, tcServer, vCSA remote code vulnerability
  • Upgrade issues from 5.1.0a to 5.1.0b

VMware ESXi 5.1 Update 1 Release Notes

  • Mirrors the new guest OS support in vCenter 5.1 Update 1. Full 200+ page OS compatibility matrix is here.
  • Contains several security patches (glibc, libxslt, libxml2)
  • Resolved: Long running vMotion operations might result in unicast flooding
  • Windows Server 2012 failover clustering is not supported

You can find the ESXi 5.1 Update 1 full release notes here.

vCloud Director 5.1.2 Release Notes

Like vCenter 5.1 Update 1, vCloud Director has some new features and many resolved issues. Full release notes is here. The full vCloud Director 5.1.2 documentation set is here.

What’s new?

  • Ability to delegate creating, reverting, and removing snapshots
  • You can install vCloud Director on Red Hat Enterprise Linux 6.3
  • You can install vClould Director using Microsoft SQL Server 2012 databases
  • Supports customization of Windows Server 2012 guest operating systems

Resolved Issues

  • Security vulnerabilities addressed by updating Java to 1.6.0_37
  • Multiple bug fixes, see full release notes

vCenter Converter Standalone 5.1 Release Notes

The new version of Converter has added a number of great new features and broader operating system support. You can find the full release notes here.

  • Supports VM hardware version 9
  • Guest operating system support for Windows 8 and Windows Server 2012
  • Guest operating system support for Red Hat Enterprise Linux 6
  • Support for machine sources that use GPT partition tables
  • Support for systems that use UEFI
  • Support for EXT4 file system

vCenter Server Heartbeat 6.5 Update 1 Release Notes

No major changes here, but incremental support for the latest VMware products. Full release notes are here.

  • Support for vCenter 5.1 Update 1
  • Support for View Composer 5.2

vSphere Data Protection 5.1.20 Release Notes

More than just bug fixes, VMware added many new features to this build. Full release notes are here. A subset of the new features:

  • Integration with vCenter alarms and alerts notification system
  • Ability to clone backup jobs
  • New filters to restore tab
  • Expands capacity up to 8TB per appliance
  • Supports the ability to expand existing datastores
  • Supports guest-level backups of Microsoft SQL Servers
  • Supports guest-level backups of Microsoft Exchange Servers

vSphere Storage Appliance 5.1.2 Release Notes

Like vSphere Data Protection, the vSphere Storage Appliance has many new features. The full release notes are here.

  • Support multiple VSA clusters managed by a single vCenter Instance (about time)
  • Ability to run vCenter Server on a subnet different from the VSA cluster
  • Support for running the VSA on one of the ESXi hosts in the VSA cluster
  • Ability to install the VSA on an existing ESXi host that has running VMs
  • Ability to increase the storage capacity of a VSA cluster
  • Up to 24TB of storage per node
  • Multiple RAID types (RAID 5, RAID 6, RAID 10)

Summary

vSphere 5.1 Update 1 will be a welcomed upgrade to customers already running vSphere 5.1. After a rocky start of vSphere 5.1 GA, VMware has clearly been working on stability, bug fixes, and supporting the latest Microsoft operating systems and SQL databases. The vCloud Suite is ever expanding, so when you go to download all the components you will see over two dozen downloads you can choose from. If you’ve been hesitant to move up to vSphere 5.1, give 5.1 Update 1 a whirl in your lab and see if it’s stable enough for you.

vSphere 5.1 Suite

vCenter 5.1 Installation: Part 15 (ESXi SSL certificate)

Welcome to a post which will let you easily update your VMware ESXi host SSL certificates. Last year when I was writing my 15-part vSphere 5.1 installation and configuration series I didn’t include instructions on how to replace the ESXi SSL certificate. That process hasn’t changed for ages, so I put it on the back burner. Now the time has come to show you part 15 of the vSphere 5.1 install series, which is a semi-automated method to replace your ESXi SSL certificates.

vSphere 5.1 has relaxed the ESXi host certificate requirements a bit by not requiring the dataencipherment and nonrepudiation key properties, which previous versions required. However, I’ve included them in my script in case you have any 5.0 or 4.x hosts. It won’t hurt to have these properties enabled on a ESXi 5.1 certificate though.

Basic requirements for the script are:

  • ESXi 4.x or 5.x host(s)
  • OpenSSL installed (0.9.8 or higher, 32-bit or 64-bit)
  • Online Windows Enterprise Certificate Authority (2008 R2 or higher recommended)
  • vSphere CLI (I’ve tested with 5.x)
  • Properly configured Windows Certificate template (see blog post here)
  • DNS “A” record for your ESXi host
  • Existing D:\Certs directory

If your ESXi host is already managed by vCenter, the HA agent can get very confused by the new SSL certificate thumbprint. I would strongly suggest you first put your host in maintenance mode, remove it from the vCenter inventory, update the SSL certificate, reboot the ESXi host, then re-add it to the vCenter inventory.

Since the script includes the creation of the CSR, you will need to modify the basic attributes of the SSL certificate variables, as shown in red below. Once you’ve modified the variables for your environment, just open an elevated VMware vSphere CLI prompt (not just a regular command prompt) and type the script name followed by the FQDN of your ESXi server.

The script will create a CSR, submit the CSR to your MS online CA, download the new certificate, and upload it to your ESXi host. You will be prompted twice to enter the root credentials of your ESXi host. Now simply reboot your ESXi server, re-add it to your inventory, and you are done! Can’t get much easier than this folks. In my case the CA certificate life is shorter than what my certificate template requested, so I got a warning message.

The script has some error checking, but it’s not super robust. You might get tripped up on the Cert_Template and CA_Name variables, so let me explain them. The Cert_Template is “template name” NOT the “Template display name”. While they are the same in my example, the “Template name” usually has spaces removed.

The CA_Name is NOT simply the hostname of your CA, but is the hostname of the CA AND the CA name which was configured during the CA installation process. You can find the CA name by opening the Certification Authority MMC and looking at the left pane.

Congratulations! You have now made it through the whole vCenter 5.1 installation process using trusted SSL certificates. Probably took way longer than you expected, and much more tedious than it should be. I would hope in vSphere v.Next that they overhaul what seems like a complete mess of internal handing of certificates. How about certificate revocation? How about the ability to completely remove a compromised certificate from all keystores?


@Echo off
 REM Change these variables for your environment.
 REM Do not put spaces between the = sign
 REM SSL Certificate Properties
 REM Country name must be exactly two letters

 Set countryName=US
 Set state=CA
 Set locality=San Diego
 Set organization=Contoso

 REM Certifiate Authority Properties
 Set Cert_Template=VMware-SSL
 Set CA_Name=D001DC02\Contoso-D001DC02-CA

 REM Existing parent path for the ESXi certificate directory
 Set Cert_Path=D:\certs

 REM
 REM -- Don't change anything below here --
 REM

 set ESXiConfig=esxi.cfg
 if [%1]==[] ; GOTO :ERROR
 if not exist %Cert_Path%\ESXi mkdir %Cert_Path%\ESXi
 if exist "D:\program Files (x86)\VMware\Vmware vSphere CLI\bin" Set CLI=D:\program Files (x86)\VMware\Vmware vSphere CLI\bin
 if exist "C:\program Files (x86)\VMware\Vmware vSphere CLI\bin" Set CLI=C:\program Files (x86)\VMware\Vmware vSphere CLI\bin
 if exist "c:\OpenSSL-Win32\bin\openssl.exe" Set OpenSSL_BIN=c:\OpenSSL-Win32\bin\openssl.exe
 if exist "c:\OpenSSL-Win64\bin\openssl.exe" Set OpenSSL_BIN=c:\OpenSSL-Win64\bin\openssl.exe
 FOR /F "Tokens=1 delims=." %%A IN ("%1") DO SET Hostname=%%A
 (
 Echo [ req ]
 Echo default_bits = 2048
 Echo default_keyfile = rui.key
 Echo distinguished_name = req_distinguished_name
 Echo encrypt_key = no
 Echo prompt = no
 Echo string_mask = nombstr
 Echo req_extensions = v3_req
 Echo.
 Echo [ v3_req ]
 Echo basicConstraints = CA:FALSE
 Echo keyUsage = digitalSignature, keyEncipherment, dataEncipherment, nonRepudiation
 Echo extendedKeyUsage = serverAuth, clientAuth
 Echo subjectAltName = DNS:%1, DNS:%hostname%
 Echo.
 Echo [ req_distinguished_name ]
 Echo countryName = %countryName%
 Echo stateOrProvinceName = %state%
 Echo localityName = %locality%
 Echo 0.organizationName = %organization%
 Echo commonName = %1
 ) >%Cert_Path%\ESXi\%ESXiConfig%
 %OpenSSL_BIN% genrsa 2048 > %Cert_Path%\ESXi\rui.key
 %OpenSSL_BIN% req -out %Cert_Path%\ESXi\rui.csr -key %Cert_Path%\ESXi\rui.key -new -config %Cert_Path%\ESXi\%ESXiConfig%
 certreq -submit -f -config "%CA_NAME%" -attrib "CertificateTemplate:%Cert_Template%" %Cert_Path%\ESXi\rui.csr %Cert_Path%\ESXi\rui.crt
 "%CLI%\vifs.pl" --server %hostname% --put %Cert_Path%\ESXi\rui.key /host/ssl_key
 "%CLI%\vifs.pl" --server %hostname% --put %Cert_Path%\ESXi\rui.crt /host/ssl_cert
 Exit /B
 :ERROR
 Echo Please specify ESXi server FQDN (e.g. ESX01.domain.net).

Creating Cisco UCS Customized vSphere 5.0 U1 Bootable ISO

UPDATE 2 5/15/2012: Looks like VMware/Cisco pulled the 5.0 U1 custom ISO installation media. So follow my blog post below to create your own.


UPDATE 1 4/23/12: Cisco released a customized vSphere 5.0 U1 installation ISO with all of their latest drivers. You can download it here under OEM Customized Installer CDs. The instructions below are still valid, and would be good for incorporating future updates in your ISO image. 

Some vendors, like HP, produce customized VMware installation ISOs that have all of their drivers integrated. This is a great time saver, but unfortunately Cisco does not provide customized vSphere 5.0 installation media with the very latest drivers. Starting with vSphere 5.0 VMware gave users a method to build their own installation media and include updated packages, such as drivers. The procedure below creates a bootable ISO image using the very latest ESXi build (5.0 U1 plus the latest patches). Your machine must be connected to the internet, as it will pull down the latest bundles in real time. You do NOT need to start with an offline depot.

1) Open a PowerCLI window with Administrator rights and type the following command:

Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

2) At this point you can list all of the packages in the depot with the following command. A partial listing is shown below.
Get-EsxSoftwarePackage | select Name,Version,ReleaseDate | sort ReleaseDate
3)  Download the driver packages for your hardware from the following VMware URLs. Personally I would suggest you download all of them, so you don’t have to rebuild the image if you get a different server model.
4) You need to unzip each of the files that you downloaded, which will reveal another ZIP file and a VIB file, among others. We will be using the embedded bundle ZIP files. If you downloaded all of the drivers, unpacked them, and moved the bundled ZIPs to a single directory it should look like:
5) Repeat step 1 from above, but substitute the bundle zip files from the above screenshot. A sample is below:
add-esxsoftwaredepot E:\enic_driver_2.1.2.22-offline_bundle-564611.zip
add-esxsoftwaredepot E:\fnic_driver_1.5.0.7-offline_bundle-563432
6) Now you want to create a copy of the “latest” VMware profile and give it a unique name. To list all of the standard ESXi profiles use the following command:
Get-EsxImageProfile | Sort-Object “ModifiedTime” -Descending | format-table -property Name,CreationTime
7) You will notice that the latest profile has a date of 4/16/2012, but the build number is only 469512, which is far from the latest build. The latest build is actually ESXi-5.0.0-20120404001-standard. You can validate the latest patch build here. Update: Looks like the 4/16/2012 builds were a glitch, as the profile list on 4/17/2012 no longer showed the 4/16 builds and the latest was in fact the 3/16/2012 build.
8) Now you need to build a new profile based on the latest patch build. I called my new profile “ESXi-5.0.0-UCS-04152012“. The build profile name will be displayed during the boot selection process if you create an installable ISO file, so think about the name you use.
new-esximageprofile -cloneprofile ESXi-5.0.0-20120404001-standard
-name “ESXi-5.0.0-UCS-04152012”
9) After you create a new image profile, you now want to add the updated UCS drivers to the profile. To determine what software package name to use, look in your driver directory at the VIB filenames. The filename prefix (e.g. net-be2net) is what you will want to use when adding the driver files.

When I tried to update the scsi-megaraid-sas bundle it said it already existed, so I skipped that in example below.  To add the remaining drivers issue the following commands:

add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-enic
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 scsi-fnic
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 scsi-lpfc820
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-ixgbe
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-be2net
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 scsi-qla2xxx
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-qlcnic
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-qlge
10) To validate that your new profile in fact has the updated and new UCS drivers, use the following command:
compare-esximageprofile -comparisonprofile ESXi-5.0.0-UCS-04152012 -referenceprofile ESXi-5.0.0-20120404001-standard
As you can see in the screenshot below two new drivers were added to our custom image (net-qlge and net-qlcnic) while four others were upgraded. So yes, our custom image did get injected with the new drivers.
11) To create a customized bundle that you can use later, issue the following command:
export-esximageprofile -imageprofile ESXi-5.0.0-UCS-04152012 -exporttobundle -filepath e:\ESXi-5.0.0-UCS-04152012.zip
12) To create a customized bootable ISO image, issue the following command:
export-esximageprofile -imageprofile ESXi-5.0.0-UCS-04152012 -exporttoISO -filepath e:\ESXi-5.0.0-UCS-04152012.ISO
13) If all goes well, and you use the exact same bundles that I did, when you install ESXi 5.0 you should see build 623860.