This session was quite advanced and had a lot of troubleshooting examples which are hard to adequately capture without the slides, so I’ll just touch on some of the counters he used during some troubleshooting examples. Maybe in future posts I’ll focus on one subject like storage stats and recreate a couple of the presented examples since I thought he did a very good job of showing you want to look for when problems rear their ugly heads.
- There are a variety of management tools for ESXi
- vCenter Alarms
- vCenter Operations
- vCenter Charts
- esxtop – Live stats
- esxplot (free utility)
- Interpreting esxtop stats manual here
- New counters in ESXi 5.0
- CPU screen now shows the number of VMs and total vCPUs
- %VMWAIT stat
- Power: CPU pStates in Mhz (BIOS must be in OS controlled power mode)
- Failed disk I/Os
- VAAI block delete
- Low latency swap – LLSWR, LLSWW (broke in 5.0, look at stats in vCenter GUI)
- NHN – Wide NUMA indication
- CPU counters are misunderstood
- RDY – VM wants to run but can’t due to scheduling issues (bad)
- CSTP – Co-stopped state. Co-scheduling overhead due to multiple vCPUs
- Run – VM is using processor time
- (Note: At this point the speaker went into great depth about CPU utilization and even more states, so the stats above just scratch the surface.)
- Storage stats
- DAVG – Most important disk stat to monitor
- QAVG – Should be nearly, if not, zero all the time
- DQLEN – Driver queue length
So there you go..some very low-level counters that you can look at to start troubleshooting performance problems. If you use the vMA you can use resxtop to monitor real-time stats from an ESXi host, so you don’t need to SSH in and do it locally. Better for security, and easier to grab stats from multiple ESXi hosts at once. The esxplot utility is great for analyzing a lot of captured data and easily graph/search it for what you want.
