Retrieving vCloud Director VM Chain Length with PowerCLI

In one of our vCloud Director environments I noticed that some datastores were reaching capacity even though there weren’t that many VMs on the datastores.  The VMs have a single 32 GB disk with no snapshots visible in the snapshot manager or API.  However, the vCenter storage reports showed that these VMs were using 300-400 GB of space.  I decide to browse the filesystem where the VMs reside and to my horror I saw that these VMs had hundreds of snapshots.  For some reason the backup software hasn’t been able to successfully commit snapshots after it backs up the VM.

Since the VMs didn’t report any snapshots through the vSphere API, I was wondering how I’d be able to find all the problem VMs and then I remembered that vCloud Director shows the chain length:

2015-11-19_13-36-17.jpg
I decided to whip up a quick little script using PowerCLI to find the problem VMs and some other info including the owner since I’d most likely have to reach out to them to resolve the issue.  Here is the script:

get-org lab | get-civm | ? { $($_.ExtensionData.VCloudExtension.any.VirtualDisksMaxChainLength -as [int]) -gt 32 } | `
select name, `
vapp, `
org, `
@{N='Owner id';E={$_.vapp.owner}}, `
@{N='Owner Full Name';E={$_.vapp.owner.fullname}}, `
@{N='Chain Length';E={$_.ExtensionData.VCloudExtension.any.VirtualDisksMaxChainLength} } `
| ft -auto

Here are the results.  I’ve changed the VM and owner names:

Name    VApp               Org   Owner id   Owner Full Name Chain Length
----    ----               ---   --------   --------------- ------------
vm067   vApp_EG_402639_1   LAB   507541     Chris Greene    152
vm069   vApp_EG_7840_1     LAB   507541     Chris Greene    148
vm071   vApp_EG_344672_1   LAB   507541     Chris Greene    148
vm109   vApp_EG_514536_1   LAB   507541     Chris Greene    153
vm111   vApp_EG_359042_1   LAB   507541     Chris Greene    144
vm124   vApp_EG_7602_1     LAB   507541     Chris Greene    148
vm171   vApp_EG_397450_1   LAB   507541     Chris Greene    150
vm175   vApp_EG_83002_2    LAB   507541     Chris Greene    150
vm179   vApp_EG_513827_2   LAB   507541     Chris Greene    149
vm181   vApp_EG_361126_1   LAB   507541     Chris Greene    153
vm183   vApp_EG_340167_1   LAB   507541     Chris Greene    150
vm188   vApp_EG_502694_1   LAB   507541     Chris Greene    152
vm197   vApp_EG_467027_1   LAB   507541     Chris Greene    150

In vCenter you’ll see the following events on VMs with this issue:

Warning message from esx01.vmware.local: This virtual machine has more than 100 redo logs in a single branch of its snapshot tree. Deleting some of the snapshots or consolidating the redo logs will improve performance. The maximum number of redo logs supported is 255.

warning

11/19/2015 5:50:29 PM

vm1

vpxuser

I didn’t see an alarm for this but you could user PowerCLI to search the event logs for these messages:

 get-vm 'broken-vm' | Get-VIEvent | ? { $_.FullFormattedMessage -like "*redo logs in a single branch of its snapshot tree*" } 

If you wanted to see how many delta vmdk files a VM has:

 get-vm 'broken-vm' | $vm.ExtensionData.LayoutEx.File | ? { $_.name -like "*delta*" } | measure | select -expandproperty count

I wanted to pass this along because the issue can be deceiving as the VMs appear to have no snapshots.  Maybe one of these methods will help you if you run into the issue.

Advertisements

Gathering vSphere VM performance metrics with Perl, docker and VMware Photon

In a previous post I described how to gather vSphere performance metrics with Powershell.  In that post the main focus was getting vSphere virtual machine performance data concurrently.  After that post I wanted to use Ruby so that I could get some experience using Ruby threads, but there wasn’t much documentation on getting vSphere performance metrics with Ruby.  However, I remembered seeing some example scripts in Perl for getting vSphere performance metrics, but I really didn’t want to try to get up to speed with using threads in Perl.  At the time, I was also learning docker so I had the thought that instead of using multiple threads to get vSphere performance metrics concurrently, I could spin up a docker container for each virtual machine that I want to get data on.  This isn’t the same thing as using threads and I’m not even sure it’s a good idea, but it would give me more experience with docker.

Versions used:

Photon setup

I used the full ISO of Photon and installed it onto a vSphere VM using these instructions.

ssh into the VM and create the directory where we will build our docker image from.  My folder is named /root/docker-files/vsphere-perf-6.  This folder will need the following files, which I’ll cover in the rest of this post:

  1. Dockerfile
  2. VMware-vSphere-CLI-6.0.0-2503617.x86_64.tar.gz
  3. get-vm-metrics.pl

Dockerfile

I’m not going to cover docker too much in this post.  If you’d like to learn more about it, check out James Turnbull’s excellent The Docker Book.  The Dockerfile is a set of instructions that describe how to build our docker image and you can probably pick up most of what is going on.  Here are the contents of my Dockerfile:


FROM centos:centos6
MAINTAINER Chris Greene <chrisgreene@outlook.com>
ENV REFRESHED_AT 10-20-2015

RUN yum -y install tar perl kmod-14-10.el7 openssl-devel uuid e2fsprogs perl-XML-LibXML perl-Crypt-SSLeay util-linux-ng which perl-Time-HiRes

ADD VMware-vSphere-CLI-6.0.0-2503617.x86_64.tar.gz /tmp/
RUN yes | /tmp/vmware-vsphere-cli-distrib/vmware-install.pl --default

RUN mkdir /tmp/vsphere-perf-data/
ADD get-vm-metrics.pl /
RUN chmod 700 /get-vm-metrics.pl
ENTRYPOINT ["/get-vm-metrics.pl"]

I do want to cover a few of these options.

FROM specifies that the image will use CentOS 6 as a base.  In checking the vSphere CLI Release Notes you will see which OSes are supported.  If your curious, I tried CentOS 7 and had a variety of issues.

RUN runs the specified command in the docker image.

ADD copies the specified file into the image as well as unpack it if it’s an archive.

ENTRYPOINT specifies the file/command that we want to run when we enter the container.  Any additional arguments that we specify when starting the container will be passed into this command.  I use this so that I don’t need to specify the perl script each time I start a container.

vSphere CLI Archive

We will need to download and copy the VMware-vSphere-CLI-6.0.0-2503617.x86_64.tar.gz into this directory.  You can download the file at vSphere CLI 6.0 GA.  I used WinSCP to copy the file from my Windows machine to the Photon VM, but you can use any method to get the file into the VM.

get-vm-metrics.pl

This is the workhouse of the solution and can be found on github.  If you’re not familiar with vSphere performance data, and even maybe if you are, this script can be pretty cryptic.  The vSphere CLI comes with a couple of sample performance scripts and on Windows can be found in C:\Program Files (x86)\VMware\VMware vSphere CLI\Perl\samples\performance.  If you really want to find out how this works, I’d suggest experimenting with those scripts and using Perl’s Data::Dumper module to print out data structures such as $all_counters and %counters_by_name.  I printed these to files so I could see how the data was arrange.   I think I wrote this in June and can’t even remember what all data is in each data structure.  I can’t imagine too many people will really be interested in how the script works, but if anyone does, they can leave a comment.

The script simply outputs the performance metrics into a CSV file having the name of the VM’s UUID.

Building the docker image

Now that we have all of the necessary files we can build the docker image:

docker build -t “chrisgreene/vsphere-vm-perf” .

The -t is specified a tag and the dot at the end says to build the image from the current directory.  You should see a lot of info fly similar to:


Sending build context to Docker daemon  54.6 MB
Step 0 : FROM centos:centos6
---> 3bbbf0aca359
Step 1 : MAINTAINER Chris Greene <chrisgreene@outlook.com>
---> Using cache
---> 61988029d210
Step 2 : ENV REFRESHED_AT 10-20-2015
---> Using cache
---> e9cc98bcc314
Step 3 : RUN yum -y install tar perl kmod-14-10.el7 openssl-devel uuid e2fsprogs perl-XML-LibXML perl-Crypt-SSLeay util-linux-ng which perl-Time-HiRes
---> Using cache
---> cf7ae0f75ac1
Step 4 : ADD VMware-vSphere-CLI-6.0.0-2503617.x86_64.tar.gz /tmp/
---> Using cache
---> 5ac95667c360
Step 5 : RUN yes | /tmp/vmware-vsphere-cli-distrib/vmware-install.pl --default
---> Using cache
---> c18e84cda077
Step 6 : RUN mkdir /tmp/vsphere-perf-data/
---> Using cache
---> 525369fe6d26
Step 7 : ADD get-vm-metrics.pl /
---> Using cache
---> a121b473e867
Step 8 : RUN chmod 700 /get-vm-metrics.pl
---> Using cache
---> c54032fad81d
Step 9 : ENTRYPOINT /get-vm-metrics.pl
---> Using cache
---> 40de5fab2fe9
Successfully built 40de5fab2fe9

We can view our image by running the docker images command:


root [ ~/docker-files/vsphere-perf-6-]# docker images
REPOSITORY                    TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
chrisgreene/vsphere-vm-perf   latest              40de5fab2fe9        About a minute ago   729.1 MB

Running our container

After the docker image has been built, we can start containers based off of it.  As previously mentioned, the get-vm-metrics.pl script outputs a VM’s performance metrics into a CSV file having the name of the VM’s UUID.  In this example I’m going to have that CSV file created in a directory named /tmp/vsphere-perf-data on the docker host’s (Photon VM) filesystem.


docker run --rm -v /tmp/vsphere-perf-data:/tmp chrisgreene/vsphere-vm-perf --server vc5c.vmware.local --vm db3 --username 'vmware\api' --password 'vmware123'

Start: 2015-11-11T05:34:2100:00 - End: 2015-11-11T05:34:2400:00

–rm specifies to delete the container once it exits.

-v say that I want to map the /tmp/vsphere-perf-data directory on the docker host to the /tmp directory on the container.  The script creates the CSV file in the /tmp directory which is mapped to the  /tmp/vsphere-perf-data directory on the docker host.

chrisgreene/vsphere-vm-perf is the name of the image I want to start the container from.

The rest of the parameters are options that are passed into the get-vm-metrics.pl script.  I’m requesting to retrieve the performance metrics from a VM named db3 on vCenter vc5c.vmware.local.

If I look in the docker host’s /tmp/vsphere-perf-data/ directory, I’ll see the CSV file:


root [ ~/docker-files/vsphere-perf-6]# ll /tmp/vsphere-perf-data/
total 4864
-rw-r--r-- 1 root root 4979479 Nov 11 05:34 502a96ed-d2bb-be58-c079-43cd79503660.txt

Conclusion

I didn’t take this further since I was only using it as an intermediate step in the final goal of gathering the data with Ruby (rbvmomi).  I thought I’d use the provided examples written in Perl instead of trying to figure out everything from scratch, which I’d probably have to do if I used rbvmomi. I did figure out how to get the data using rbvmomi, but unfortunately the project that was requiring this data has been cancelled so I don’t know if I’ll get a chance to revisit it.  I’d still like to learn a little about multi-threading in Ruby or concurrency in Go so maybe I can use gathering VM performance data as an exercise.

The docker image can be found at https://hub.docker.com/r/chrisgreene/vsphere-vm-perf/