Hi,
I have been running vSAN 5.5U2 for circa 6 months in a lab configuration with no issues, but over the weekend, noticed everything had come to an abrupt halt. I restarted all three nodes (I use the same MacMini configuration as William Lam) and expected things to pick up where they left off - Note there were no changes to the setup at all.
The VMs began to power back on but as soon as I started to try and use them, noticed extremely poor performance.
A quick esxtop showed no signs of problems with the disks (D/AVG etc) but the VM latency was in the high hundreds. I started shutting down the few VMs I have running (about 10) and left nothing other than the VCSA up.
| ADAPTR PATH | NPTH CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd |
| vmhba0 - | 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 |
| vmhba32 - | 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 |
| vmhba33 - | 1 1.46 0.49 0.97 0.00 0.00 0.62 0.04 0.65 0.01 |
| vmhba34 - | 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 |
| vmhba35 - | 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 |
| vmhba36 - | 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 |
| vmhba37 - | 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 |
| vmhba38 - | 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 |
| GID VMNAME | VDEVNAME NVDISK CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s LAT/rd LAT/wr |
| 25112 LAB-VCA-V001 | - 2 3.36 0.00 3.36 0.00 0.02 0.00 1310.38 |
Eventually the VCSA ground to a complete halt to the point where eventually I couldn't even manage my cluster unless I was at the CLI on each of the hosts. I was still able to SSH to the VCSA to run some Ruby commands where the vsan health status came back "OK".
I checked the Network ports to make sure a crazy Sync wasn't underway as a consequence of the failure and that my network wasn't saturated (GB between each of the hosts) but the network stats were low and vsan health reports suggested there were no outstanding syncs.
Has anyone got any ideas where the Latency in this particular instance is coming from? VMkernel for example? How can I check this?
Update Edit:
Ok, so we're now in a slightly worse place than we were last night.
Having rebooted all three hosts, the vSAN appears to be stable again, HOWEVER, when I list the contents of the vsanDatastore, I'm missing 60% of my VMs (including the VCSA). Any tips for attempting to bring them back - like I say, its only homelab, but the troubleshooting bit is all part of the fun. Capacity wise, the datastore is reporting the same consumption before the failure so I imagine the data is there, I just can't see it!