Retrieving vCloud Director VM Chain Length with PowerCLI

In one of our vCloud Director environments I noticed that some datastores were reaching capacity even though there weren’t that many VMs on the datastores.  The VMs have a single 32 GB disk with no snapshots visible in the snapshot manager or API.  However, the vCenter storage reports showed that these VMs were using 300-400 GB of space.  I decide to browse the filesystem where the VMs reside and to my horror I saw that these VMs had hundreds of snapshots.  For some reason the backup software hasn’t been able to successfully commit snapshots after it backs up the VM.

Since the VMs didn’t report any snapshots through the vSphere API, I was wondering how I’d be able to find all the problem VMs and then I remembered that vCloud Director shows the chain length:

I decided to whip up a quick little script using PowerCLI to find the problem VMs and some other info including the owner since I’d most likely have to reach out to them to resolve the issue.  Here is the script:

get-org lab | get-civm | ? { $($_.ExtensionData.VCloudExtension.any.VirtualDisksMaxChainLength -as [int]) -gt 32 } | `
select name, `
vapp, `
org, `
@{N='Owner id';E={$_.vapp.owner}}, `
@{N='Owner Full Name';E={$_.vapp.owner.fullname}}, `
@{N='Chain Length';E={$_.ExtensionData.VCloudExtension.any.VirtualDisksMaxChainLength} } `
| ft -auto

Here are the results.  I’ve changed the VM and owner names:

Name    VApp               Org   Owner id   Owner Full Name Chain Length
----    ----               ---   --------   --------------- ------------
vm067   vApp_EG_402639_1   LAB   507541     Chris Greene    152
vm069   vApp_EG_7840_1     LAB   507541     Chris Greene    148
vm071   vApp_EG_344672_1   LAB   507541     Chris Greene    148
vm109   vApp_EG_514536_1   LAB   507541     Chris Greene    153
vm111   vApp_EG_359042_1   LAB   507541     Chris Greene    144
vm124   vApp_EG_7602_1     LAB   507541     Chris Greene    148
vm171   vApp_EG_397450_1   LAB   507541     Chris Greene    150
vm175   vApp_EG_83002_2    LAB   507541     Chris Greene    150
vm179   vApp_EG_513827_2   LAB   507541     Chris Greene    149
vm181   vApp_EG_361126_1   LAB   507541     Chris Greene    153
vm183   vApp_EG_340167_1   LAB   507541     Chris Greene    150
vm188   vApp_EG_502694_1   LAB   507541     Chris Greene    152
vm197   vApp_EG_467027_1   LAB   507541     Chris Greene    150

In vCenter you’ll see the following events on VMs with this issue:

Warning message from esx01.vmware.local: This virtual machine has more than 100 redo logs in a single branch of its snapshot tree. Deleting some of the snapshots or consolidating the redo logs will improve performance. The maximum number of redo logs supported is 255.

warning

11/19/2015 5:50:29 PM

vm1

vpxuser

I didn’t see an alarm for this but you could user PowerCLI to search the event logs for these messages:

 get-vm 'broken-vm' | Get-VIEvent | ? { $_.FullFormattedMessage -like "*redo logs in a single branch of its snapshot tree*" } 

If you wanted to see how many delta vmdk files a VM has:

 get-vm 'broken-vm' | $vm.ExtensionData.LayoutEx.File | ? { $_.name -like "*delta*" } | measure | select -expandproperty count

I wanted to pass this along because the issue can be deceiving as the VMs appear to have no snapshots.  Maybe one of these methods will help you if you run into the issue.