Retrieving vCloud Director VM Chain Length with PowerCLI

In one of our vCloud Director environments I noticed that some datastores were reaching capacity even though there weren’t that many VMs on the datastores.  The VMs have a single 32 GB disk with no snapshots visible in the snapshot manager or API.  However, the vCenter storage reports showed that these VMs were using 300-400 GB of space.  I decide to browse the filesystem where the VMs reside and to my horror I saw that these VMs had hundreds of snapshots.  For some reason the backup software hasn’t been able to successfully commit snapshots after it backs up the VM.

Since the VMs didn’t report any snapshots through the vSphere API, I was wondering how I’d be able to find all the problem VMs and then I remembered that vCloud Director shows the chain length:

2015-11-19_13-36-17.jpg
I decided to whip up a quick little script using PowerCLI to find the problem VMs and some other info including the owner since I’d most likely have to reach out to them to resolve the issue.  Here is the script:

get-org lab | get-civm | ? { $($_.ExtensionData.VCloudExtension.any.VirtualDisksMaxChainLength -as [int]) -gt 32 } | `
select name, `
vapp, `
org, `
@{N='Owner id';E={$_.vapp.owner}}, `
@{N='Owner Full Name';E={$_.vapp.owner.fullname}}, `
@{N='Chain Length';E={$_.ExtensionData.VCloudExtension.any.VirtualDisksMaxChainLength} } `
| ft -auto

Here are the results.  I’ve changed the VM and owner names:

Name    VApp               Org   Owner id   Owner Full Name Chain Length
----    ----               ---   --------   --------------- ------------
vm067   vApp_EG_402639_1   LAB   507541     Chris Greene    152
vm069   vApp_EG_7840_1     LAB   507541     Chris Greene    148
vm071   vApp_EG_344672_1   LAB   507541     Chris Greene    148
vm109   vApp_EG_514536_1   LAB   507541     Chris Greene    153
vm111   vApp_EG_359042_1   LAB   507541     Chris Greene    144
vm124   vApp_EG_7602_1     LAB   507541     Chris Greene    148
vm171   vApp_EG_397450_1   LAB   507541     Chris Greene    150
vm175   vApp_EG_83002_2    LAB   507541     Chris Greene    150
vm179   vApp_EG_513827_2   LAB   507541     Chris Greene    149
vm181   vApp_EG_361126_1   LAB   507541     Chris Greene    153
vm183   vApp_EG_340167_1   LAB   507541     Chris Greene    150
vm188   vApp_EG_502694_1   LAB   507541     Chris Greene    152
vm197   vApp_EG_467027_1   LAB   507541     Chris Greene    150

In vCenter you’ll see the following events on VMs with this issue:

Warning message from esx01.vmware.local: This virtual machine has more than 100 redo logs in a single branch of its snapshot tree. Deleting some of the snapshots or consolidating the redo logs will improve performance. The maximum number of redo logs supported is 255.

warning

11/19/2015 5:50:29 PM

vm1

vpxuser

I didn’t see an alarm for this but you could user PowerCLI to search the event logs for these messages:

 get-vm 'broken-vm' | Get-VIEvent | ? { $_.FullFormattedMessage -like "*redo logs in a single branch of its snapshot tree*" } 

If you wanted to see how many delta vmdk files a VM has:

 get-vm 'broken-vm' | $vm.ExtensionData.LayoutEx.File | ? { $_.name -like "*delta*" } | measure | select -expandproperty count

I wanted to pass this along because the issue can be deceiving as the VMs appear to have no snapshots.  Maybe one of these methods will help you if you run into the issue.


vCenter Orchestrator workflow to renew vCloud Director vApp leases

In vCloud Director (vCD), vApp leases are defined at the organization level and can’t be overridden at the vApp level.  In our dev/test cloud we set leases so that vApps will expire after a certain amount of time.  We do this so that unused vApps will expire and users can continue to deploy new vApps without the need for further increasing their vApp quota.  Users receive emails that their vApps are about to expire and they can renew them at any time, but I thought I’d look into a way of creating an exception list for vApps that should never expire.

I decided on using vCenter Ochestrator (vCO) to accomplish this for the following reasons:

  • We already use heavily in our vCD environment.
  • A vCO workflow has a graphical interface for adding/deleting vApps from the exception list.
  • The workflow will reside on the vCO server and not on some other server where it’s more likely to be forgotten.
  • I can use the built-in vCO scheduler to run the workflow.
  • If needed, I can make an API call to run the workflow externally.

The workflow

I’m not going to document each step of creating the workflow, but I’ll list what each workflow component looks like so you can see the inputs, outputs and code.

Workflow overview

2015-08-26_10-45-35


Workflow attributes

2015-08-26_10-46-18


vApps Left?

Inputs

2015-08-26_10-47-10

Scripting

2015-08-26_10-47-43


Get current vApp

Inputs

2015-08-26_10-48-33

Outputs

2015-08-26_10-49-11

Scripting

2015-08-26_11-11-24


vApp exist?

Inputs

2015-08-26_10-51-02

Scripting

2015-08-26_11-07-41


Update Leases

Inputs

2015-08-26_11-12-21

Outputs

2015-08-26_10-52-15

Scripting

2015-08-26_10-52-42

leaseSettingsSection = currentVapp.getLeaseSettingsSection()

System.log("Updating runtime lease for vApp: " + currentVapp.name + ". Current lease is: " + leaseSettingsSection.deploymentLeaseInSeconds)
System.log("Updating storage lease for vApp: " + currentVapp.name + ". Current lease is: " + leaseSettingsSection.storageLeaseInSeconds)

newDeploymentLeaseInSeconds = currentVapp.parent.parent.toAdminObject().settings.vAppLeaseSettings.deploymentLeaseSeconds
newStorageLeaseInSeconds    = currentVapp.parent.parent.toAdminObject().settings.vAppLeaseSettings.storageLeaseSeconds

leaseSettingsSection.deploymentLeaseInSeconds = newDeploymentLeaseInSeconds
leaseSettingsSection.storageLeaseInSeconds    = newStorageLeaseInSeconds

var task = currentVapp.updateSection(leaseSettingsSection)


VclWaitTaskEnd

Inputs

2015-08-26_10-53-17

Adding vApp(s) to the workflow

Edit the workflow:

2015-08-12_10-40-52

Select the vApp attribute’s value:

2015-08-12_10-42-53

2015-08-12_10-43-28

Browse to vCloud Director > vCD instance > Organizations > Org that your vApp(s) are in > vDCs > vDC your vApp(s) are in > vApps

2015-08-12_11-49-42

Add each of the vApps that you want to renew the leases on.

2015-08-12_11-53-47

Once you’re done, you’ll see the list of vApps in the vApps attribute array:

2015-08-12_11-54-51

Example of a successful run:

2015-08-26_10-54-48

Scheduling the workflow

2015-08-26_11-33-35

Select your workflow:

2015-08-12_11-58-41

Set the scheduling times for whatever makes sense in your environment:

2015-08-12_11-59-37

Finished schedule:

2015-08-12_12-00-05


Re-associating vCloud Director VMs with the correct resource pool

Our team was upgrading ESXi hosts in our vCloud Director environment and all of the VMs must have not have migrated when placing the hosts into maintenance mode so they manually migrated the VMs.  Since we have multiple org vDCs, we have multiple resource pools.  As you may know, when you try to migrate VMs that are in multiple resource pools, the migration wizard only allows you to place the VMs in a single resource pool.  I started seeing the following system alert on all of the vCloud Director VMs:

2015-01-15_20-52-07

I checked vCenter and sure enough all of the VMs were in the root resource pool (the cluster), and no longer in the correct resource pool.  If you migrate the VMs one at a time, the migration wizard automatically places the VM in the correct resource pool, but the team didn’t want to do this as it would take too much time.  If you do have to migrate the VMs manually, you can migrate VMs in bulk and have the migration wizard place the VMs in the correct resource pool if you use the following process:

  1. Select a resource pool.
  2. Select the Related Objects tab.
  3. Select Virtual Machines.
  4. Select all of the Virtual Machines.
  5. Select Migrate.

2015-01-15_21-03-23

Here you can see that the wizard automatically selected the correct resource pool:

2015-01-15_21-03-50

You still have to go through each resource pool and perform the migrations, but the VMs will retain the correct resource pool.  Of course, you could script it all as well.

So I was in the position of having all of the VMs in the wrong resource pool, which causes alarms in vCloud Director and could impact the performance of the virtual machines.  I could have went into vCloud Director and found each of the VMs individually, but this would have taken forever so I decided to use PowerCLI to re-associate the VMs with the correct resource pool.

You’ll need to be connected to vCloud and all of your vCenter instances:

connect-ciserver cloud.corp.local

connect-viserver vc.corp.local

Paste the following function into PowerCLI:


function reassociatevCDRPs {
  <#
  .SYNOPSIS
    Places vCloud Director virtual machines into the correct vCenter resource pools.  
  .DESCRIPTION
    Places vCloud Director virtual machines into the correct vCenter resource pools.  This can be helpful when the VMs are moved from their resource pool during a task such as manual vMotion. 
  .EXAMPLE
  reassociatevCDRPs -org all -promptOnMove $false
  .EXAMPLE
  reassociatevCDRPs -org admin -promptOnMove $true
  #>

  param(
    [Parameter(Mandatory=$true)] 
    [string] $org = "all",
    [Parameter(Mandatory=$true)] 
    [string] $promptOnMove = $false
  )

    $ovdcLookupTable = @{}
    $vmsToMove       = @{}

    # Build lookup tables
    $orgIds = @{}
    $orgNames = @{}
    search-cloud -querytype organization | % { $orgIds[$_.name] = $_.id; $orgNames[$_.id] = $_.name }

    $orgVdcIds       = @{}
    $orgVdcNames     = @{}
    search-cloud -querytype AdminOrgVdc | % { $orgVdcNames[$_.id] = $_.name; $orgvDCIds[$_.name] = $_.id }

    $vCDVMs = search-cloud -querytype adminvm

  $searchCommand = "search-cloud -querytype adminvm"

  if ($org -ne 'all') {
    # Throw an error if the organization is not found in the vCloud instance.  Otherwise add a filter to the search-cloud command to only work on the supplied organization.
    if (! $orgIds[$org]) { throw "Organization $org not found." }  
    $searchCommand += " -filter org==$($orgIds[$org])"
  }

  $vcdVMs = invoke-expression $searchCommand

    $vcdVMs | % { 
      $vcdVM = $_
      
      # Get the resouce pool name in the format of: orgVDC Name (UUID)
      $vcdRPName = "$($orgVdcNames[$vcdVM.Vdc]) ($($_.Vdc.split(':')[3]))" 

      $vcVM = get-vm -id "VirtualMachine-$($vcdVM.moref)"
      $vcRP = $vcVM.resourcepool 
      $vcRPName = $vcRP.name
      
      if ($vcdRPName -ne $vcRPName) {  # Test to see if vCD's resource pool matches vCenter's resource pool. 
        echo "$($vcdVM.name) is in the resource pool '$($vcRPName)' and should be the '$($vcdRPName)' resource pool."
        # Add to list of VMs that need to be moved.   
        $vmsToMove[$vcVM] = get-resourcepool $vcdRPName
      }

    }

    $vmsToMove.keys | % { 
      if ($promptOnMove -eq $true) {
        $response = read-host "Move $($_.name) to the correct resource pool ($($vmsToMove[$_])) ( [y]es, [n]o, [a]ll )?"
        if ($response -eq 'n') { # If the user selects not to move the VM, try the next VM in the list.
          return 
        } 
        elseif ($response -eq 'a') {
          $promptOnMove = $false
        } 
      }
      $resourcePool = $vmsToMove[$_] 
      echo "Moving $vm into resource pool $resourcePool"
      move-vm $_ -Destination $resourcePool | out-null
      #echo "result: $($?)"
    }

}

To correct all VMs with no prompting:

reassociatevCDRPs -org all -promptOnMove $false

To move VMs to a specific org and to prompt before each move:

reassociatevCDRPs -org “org name” -promptOnMove $true

This script won’t move shadow VMs, but those are easy… for each cluster just select them all and drag them into the cluster’s System vDC (uuid) resouce pool.  vShield Edges also won’t be moved.  Those will likely need to be moved into the System vDC resource pool as well or if they are a fenced vApp, they will go into an org vDC resource pool.  Or you can just de-deploy them and they should go to the correct resource pool.