Retrieving vCloud Director VM Chain Length with PowerCLI

In one of our vCloud Director environments I noticed that some datastores were reaching capacity even though there weren’t that many VMs on the datastores.  The VMs have a single 32 GB disk with no snapshots visible in the snapshot manager or API.  However, the vCenter storage reports showed that these VMs were using 300-400 GB of space.  I decide to browse the filesystem where the VMs reside and to my horror I saw that these VMs had hundreds of snapshots.  For some reason the backup software hasn’t been able to successfully commit snapshots after it backs up the VM.

Since the VMs didn’t report any snapshots through the vSphere API, I was wondering how I’d be able to find all the problem VMs and then I remembered that vCloud Director shows the chain length:

2015-11-19_13-36-17.jpg
I decided to whip up a quick little script using PowerCLI to find the problem VMs and some other info including the owner since I’d most likely have to reach out to them to resolve the issue.  Here is the script:

get-org lab | get-civm | ? { $($_.ExtensionData.VCloudExtension.any.VirtualDisksMaxChainLength -as [int]) -gt 32 } | `
select name, `
vapp, `
org, `
@{N='Owner id';E={$_.vapp.owner}}, `
@{N='Owner Full Name';E={$_.vapp.owner.fullname}}, `
@{N='Chain Length';E={$_.ExtensionData.VCloudExtension.any.VirtualDisksMaxChainLength} } `
| ft -auto

Here are the results.  I’ve changed the VM and owner names:

Name    VApp               Org   Owner id   Owner Full Name Chain Length
----    ----               ---   --------   --------------- ------------
vm067   vApp_EG_402639_1   LAB   507541     Chris Greene    152
vm069   vApp_EG_7840_1     LAB   507541     Chris Greene    148
vm071   vApp_EG_344672_1   LAB   507541     Chris Greene    148
vm109   vApp_EG_514536_1   LAB   507541     Chris Greene    153
vm111   vApp_EG_359042_1   LAB   507541     Chris Greene    144
vm124   vApp_EG_7602_1     LAB   507541     Chris Greene    148
vm171   vApp_EG_397450_1   LAB   507541     Chris Greene    150
vm175   vApp_EG_83002_2    LAB   507541     Chris Greene    150
vm179   vApp_EG_513827_2   LAB   507541     Chris Greene    149
vm181   vApp_EG_361126_1   LAB   507541     Chris Greene    153
vm183   vApp_EG_340167_1   LAB   507541     Chris Greene    150
vm188   vApp_EG_502694_1   LAB   507541     Chris Greene    152
vm197   vApp_EG_467027_1   LAB   507541     Chris Greene    150

In vCenter you’ll see the following events on VMs with this issue:

Warning message from esx01.vmware.local: This virtual machine has more than 100 redo logs in a single branch of its snapshot tree. Deleting some of the snapshots or consolidating the redo logs will improve performance. The maximum number of redo logs supported is 255.

warning

11/19/2015 5:50:29 PM

vm1

vpxuser

I didn’t see an alarm for this but you could user PowerCLI to search the event logs for these messages:

 get-vm 'broken-vm' | Get-VIEvent | ? { $_.FullFormattedMessage -like "*redo logs in a single branch of its snapshot tree*" } 

If you wanted to see how many delta vmdk files a VM has:

 get-vm 'broken-vm' | $vm.ExtensionData.LayoutEx.File | ? { $_.name -like "*delta*" } | measure | select -expandproperty count

I wanted to pass this along because the issue can be deceiving as the VMs appear to have no snapshots.  Maybe one of these methods will help you if you run into the issue.


Gathering real-time vSphere performance metrics in parallel using PowerCLI

Overview

In a previous post I described how to list all vCenter performance metrics and a project that I was working on.  To recap, I was asked to gather all real-time performance data from all virtual machines in vCenter and publish the results to an API.  As in the previous post on this topic, I’m going to use PowerCLI, but in upcoming posts I’ll show how to get the same data using Perl using the vSphere CLI (I haven’t kept up with the VMware Perl Toolkit in years and it looks like it was merged into the vSphere CLI) and with Ruby using VMware’s rbvmomi.  I’d like to integrate the latter two with docker (I’ve created a docker image that uses the vSphere CLI to gather the data but need to put some finishing touches on it).

Processes vs Threads

In Powershell there are at least two ways of parallelizing your scripts: jobs and runspaces.  Jobs create separate processes whereas runspaces use threads.  You can read more about processes vs threads here.  If you use jobs, you will see a powershell.exe processes for each job you create.  If you use threads, you’ll see a single powershell.exe.  I tried both approaches and found that, for my use case, using jobs was faster than using runspaces so I’m going to show the jobs based approach here.  It takes longer to create a process than it does to create a thread so it takes the jobs based approach a little longer to get going, but once they do they performed more quickly than runspaces.  Since the work that the jobs/threads performed took upwards of a minute and a half, the slower start up times of jobs was worth the trade off.  I imagine as the time it takes for your jobs/threads to perform their work decreases, the balance would start to favor using runspaces.

Getting real-time VM performance data

We will be using the Get-StatType command to retrieve what stat types are available for a VM.  In my case I’m only concerned about real-time data, which is stored on the vCenter server for an hour before it’s rolled up into the database in 5 minute intervals.  Here is an example of retrieving the available real-time stat types for a VM:

$vm = get-vm MyVM

Get-StatType -entity $vm -realtime

If a VM has been powered off for an hour (or if vCenter hasn’t been receiving performance data for the VM from the host for any reason), you’ll receive no results back when you run the Get-StatType command.  If you look at the VM’s real-time performance in vCenter’s performance charts, you’ll see “Performance data is currently not available for this entity.”  Since real-time data is collected every 20 seconds, if you power the VM on and wait 20 seconds, you’ll then be able to retrieve the available stat types as well as see data in the vCenter performance charts.   After doing this, here is the results of selecting the first five available stat types:

Get-StatType -entity $vm -realtime | sort | select -first 5

cpu.costop.summation
cpu.demand.average
cpu.entitlement.latest
cpu.idle.summation
cpu.latency.average

Now we need to retrieve the performance metrics of the stat types for the VM.  My requirements are pretty simple so there is quite a bit more to stats gathering than what I’m going to show.  All I need to get is the real-time data every hour.  Again, I’d suggest reading Lucd’s awesome PowerCLI & vSphere statistics series for more info.  Skimming through it makes me realize I should read it again.

Let’s get the first 5 data points for the ‘cpu.costop.summation’ stat type:

Get-Stat -entity $vm -realtime -stat ‘cpu.costop.summation’ | ? { $_.instance -eq “” } | select -first 5 | format-table -auto

MetricId             Timestamp            Value Unit        Instance
--------             ---------            ----- ----        --------
cpu.costop.summation 6/18/2015 8:17:20 PM     0 millisecond
cpu.costop.summation 6/18/2015 8:17:00 PM     0 millisecond
cpu.costop.summation 6/18/2015 8:16:40 PM     0 millisecond
cpu.costop.summation 6/18/2015 8:16:20 PM     0 millisecond
cpu.costop.summation 6/18/2015 8:16:00 PM     0 millisecond

This command should be pretty self-explanatory.  Please read Lucd’s explanation on the Instance column.  Let’s run the help command against the Get-Stat commandlet to see what options are available:

help Get-Stat

Get-Stat [-Entity] <VIObject[]> [-Common] [-Memory] [-Cpu] [-Disk] [-Network] [-Stat <String[]>] [-Start <DateTime>] [-Finish <DateTime>] [-MaxSamples <Int
32>] [-IntervalMins <Int32[]>] [-IntervalSecs <Int32[]>] [-Instance <String[]>] [-Realtime] [-Server <VIServer[]>] [<CommonParameters>]

Notice how the -stat parameter is formatted; [-Stat <String[]>].  The []s after String means that this parameter accepts an array of values.  This means we can define an array with the stat types we wish to retrieve and then pass it into the Get-Stat commandlet as such:

$statTypes = @(‘cpu.costop.summation’, ‘cpu.demand.average’)

Get-Stat -entity $vm -realtime -stat $statTypes | ? { $_.instance -eq “” }

I actually missed this my first go at the script.  It wasn’t until I implemented the script in Perl when I saw that you could pass in an array of values.  I thought surely you could do the same in PowerCLI so I checked the syntax and sure enough you can.  Before I was iterating through each available stat type and calling Get-Stat (100+ times for each VM). Doh.

Powershell Jobs

For a high-level overview of Powershell Jobs, check out Geek School: Learn How to Use Jobs in PowerShell.

Let’s checkout what job commandlets are available to us:

gcm *job* -module Microsoft.PowerShell.Core

CommandType Name        Version Source
----------- ----        ------- ------
Cmdlet      Debug-Job   3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Get-Job     3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Receive-Job 3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Remove-Job  3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Resume-Job  3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Start-Job   3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Stop-Job    3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Suspend-Job 3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Wait-Job    3.0.0.0 Microsoft.PowerShell.Core

We will be mostly working with Start-Job so let’s see what the help command has to say about it:

help Start-Job

NAME
Start-Job

SYNTAX
Start-Job [-ScriptBlock] <scriptblock> [[-InitializationScript] <scriptblock>] [-Name <string>] [-Credential <pscredential>] [-Authentication
<AuthenticationMechanism> {Default | Basic | Negotiate | NegotiateWithImplicitCredential | Credssp | Digest | Kerberos}] [-RunAs32] [-PSVersion <version>]
[-InputObject <psobject>] [-ArgumentList <Object[]>]  [<CommonParameters>]

Let’s cover the highlighted options:

  • -ScriptBlock: A chunk of code that our job will execute.
  • -InitializationScript: When your job runs the ScriptBlock, the code in the ScriptBlock isn’t going to have access to any of the variables, functions, etc that were defined in the main script unless you make them available to the newly created job.  Passing an initialization script to your ScriptBlock is a way of doing exactly this.  You can also pass arguments to the ScriptBlock.  I’m not sure when you should use one over the other.  So far I’ve been placing static info into the initialization script and dynamic info into arguments. In my case I’m going to make functions that I define in my main script available to the ScriptBlock by placing them into the initialization script.
  • -Name:  Name of the job.  I’d probably make this something unique like the instance UUID of the VM.  You could also call this the name of the VM, but if you’re working with multiple vCenters, you could run into duplicates.
  • -ArgumentList: A list of variables that we can pass into our ScriptBlock.

Here is what it looks like when I create a job in my script:


$job = Start-Job -name $vm.name -InitializationScript $jobFunctions -ScriptBlock $scriptBlock -ArgumentList $vm.name, $vcConnection, $statTypes, $outputDirectory

Since Start-Job returns the created job, I store it for later use so I can query the state of the job.

The Script

Since this script is for a pretty specific use-case, I doubt it will be of much use to anyone else as is.  My hope is that someone will be able learn something useful from the post itself and/or by looking through the script.  Since I had no idea what I was doing at first, I hope that this will at least get someone up and running more quickly than it took me.  I get the urge to go back and re-write it to be more general so someone could post the results to other endpoints such as building an HTML report, but I have other work to do and don’t want to spend time on something that may never be needed.  I’m also not going to go through the script line by line but I did try to comment it heavily.

There is a function named publishResults() that is wrapped in a variable named $jobFunctions.  Any function that is in the $jobFunctions variable will be available to the code in the script block.  publishResults() is what uploads the performance data stored in CSV files to our API.  You could probably re-write publishResults() to do something like create an HTML report.

The variable $scriptBlock is what contains all the code that each Powershell job will perform.  On line #106, I’ve commented out the publishResults() call since it wouldn’t work in your environment.  If you re-write publishResults() to do something useful, you’ll need to un-comment this line.

The start of the script begins on line #119.  I hope the comments will be enough to explain the rest.  I’ll paste the code below, but it’s probably easier to read here.  If I ever get around to it, I’ll upload my Perl & Ruby examples in this repo as well.

Here is what the script looks like when it’s running:

2015-08-06_10-51-31

The output folder will look similar to:

2015-08-06_11-01-23

Where each file is a VM’s vCenter UUID.

Each file will have contents similar to:

2015-08-06_11-01-44

In this example the VM’s name, VC UUID, power state, cpu.usagemhz.avg label, date, metric value and metric unit.

Here is the code, but like I mentioned earlier, it’s easier to read on Github.


$jobFunctions = {
function publishResults {
&lt;#
.SYNOPSIS
.DESCRIPTION
.EXAMPLE
.EXAMPLE
#&gt;

param(
[Parameter(Mandatory=$true)]
[string] $inFile
)

# Most of what is in this function probably won't be too useful to anyone.  You'd probably need to re-write it.

$method = 'POST'
$uri = [System.Uri] "http://2.2.3.1:6543/api/v1/data_sets?gridname=vm&clustername=vcenter"
$outputFile = 'C:\Users\chris\Documents\powercli\stats\publish-result.txt'

$request = Invoke-RestMethod -uri $uri -method $method -InFile $inFile

if ($request.'Job Status' -eq 'Success') {
add-content $outputFile "Successfully published results for $inFile"
}
else {
add-content $outputFile "Failed to published results for $inFile"
}
}
}

$scriptBlock = {
param($vmName, $vc, $statTypes, $outputDirectory)

# Even though I pass in the vCenter connection, I have to re-establish it.  I'd like to know of a way to prevent this.
$vc = Connect-VIServer $vc.name -Session $vc.sessionId

# Even if I pass in the VM object (and not just the name as is currently being done), I still have to retrieve the VM again.  I'd like to know of a way to prevent this.
try {
$vm = Get-VM $vmName -server $vc
}
catch {
$ErrorMessage = $_.Exception.Message
$FailedItem   = $_.Exception.ItemName
add-content "$($outputDirectory)\log.txt" "$errormessage, $faileditem"
}

$filteredStatTypes = @()

# Get all of the stat types for the time interval (realtime) specified.
# When querying for realtime stats, if no results are returned, this most likely means the VM has been powered off for an hour,
# the host the VM is on wasn't sending data to vCenter, the host the VM is on was disconnected, etc.
$availableStatTypes = get-stattype -entity $vm -realtime -Server $vc | sort

# There is no point continuing if there is no data for the VM so return to the calling context.
if ($availableStatTypes -eq $null) { return }

# We need to place all the stats we want to query into $filteredStatTypes.  If the user doesn't specify any stat types ($statTypes),
# then we just assign all available stats ($availableStatTypes) into $filteredStatTypes.
# If the user does specify stat types, we iterate through each one and make sure it's available in $availableStatTypes and add it
# to $filteredStatTypes.
if ($statTypes -eq $null) {
$filteredStatTypes = $availableStatTypes
}
else {
foreach ($statType in $statTypes) {
if ($statType -in $availableStatTypes) {
$filteredStatTypes += $statType
}
}
}

if ($filteredStatTypes -eq $null) { return }

# Collection info to be used in building the report.
$vmName       = $vm.name
$persistentId = $vm.PersistentId
$vmPowerState = $vm.powerstate
$cluster      = ($vm | get-cluster).name
$outputFile = $outputDirectory + $persistentId + '.txt'

# If the output file already exists, go ahead and delete it.
# if (Test-Path $outputFile) { Add-Content "$($outputDirectory)\dups.txt" "$($vm.name)" ; Remove-Item $outputFile }
if (Test-Path $outputFile) { Remove-Item $outputFile }

# During testing you may want limit the number of results you get back. Uncomment the following line and select the first X amount of stat types to limit the the results.
#$statTypes = $statTypes | select -first 5

$finish = Get-Date

# Since realtime stats only go back an hour before they are rolled up, we only need to get an hours worth of data.
$start = $finish.AddHours(-1)
$stats = Get-Stat -entity $vm -server $vc -realtime -stat $filteredStatTypes -start $start -finish $finish | ? { $_.instance -eq "" }

foreach ($stat in $stats) {
$temp = @()
# Build up a temp array that will contain each of the items to be used to build up a line in the report.
$temp += $vmName, $persistentId, $vmPowerState, $cluster, $stat.MetricId, $stat.Timestamp, $stat.Value, $stat.Unit
# Combine each of the items in the temp array to create a line of comma separated values.
$content = '"' + $($temp -join '","') + '"'
# Store each line into our output file.
Add-Content $outputFile $content
}

try {
#publishResults $outputFile
}
catch {
$ErrorMessage = $_.Exception.Message
$FailedItem   = $_.Exception.ItemName
add-content "$($outputDirectory)\log.txt" "$errormessage, $faileditem, $vmName, $persistentId"
}

# Disconnect this (and only this) instance of vCenter so we don't leave unused sessions lying around.
Disconnect-VIServer $vc -Confirm $false
}

# Record the time the script started.
$start = Get-Date

$vCenterName = 'vc5c.vmware.local'

$vcConnection = Connect-VIServer $vCenterName

# This is where we will store the CSV files with the performance data.
$outputDirectory = 'C:\Users\chris\Desktop\perfdumps\'

# Max amount of jobs (processes) we want running at any time.  You may need to tweak this depending on the resources of your machine.
$maxJobCount = 4

# sleep time in seconds between checking to see if it's okay to run another job.
$sleepTimer = 3

# Here we can define an array of the counters we want to retrieve.  If you have a large list of counters, it may be easier to store them in an external file. For example:
#$statTypes = @('mem.active.average', 'mem.granted.average')
# If you don't define this array, all performance counters will be pulled.

# Retrieve the VMs we want to retrieve stats from.
# When you're testing you may want to only grab a subset of all VMs.  Here are a few examples
# get all VMs in the vCenter(s) you're conneted to: $vms = get-vm
# get all VMs in a specific cluster:                $vms = get-cluster 'resource cluster' | get-vm
# get the first 10 VMs that are powered on:         $vms = get-vm | ? { $_.powerstate -eq 'PoweredOn' } | select -First 10

$vms = get-vm | ? { $_.powerstate -eq 'PoweredOn' } | select -First 10

# Create our job queue.
$jobQueue = New-Object System.Collections.ArrayList

# Main loop of the script.
# Loop through each VM and start a new job if we have less than $maxJobCount outstanding jobs.
# If the $maxJobCount has been reached, sleep 3 seconds and check again.
foreach ($vm in $vms) {
# Wait until job queue has a slot available.
while ($jobQueue.count -ge $maxJobCount) {
echo "jobQueue count is $($jobQueue.count): Waiting for jobs to finish before adding more."
foreach ($jobObject in $jobQueue.toArray()) {
if ($jobObject.job.state -eq 'Completed') {
echo "jobQueue count is $($jobQueue.count): Removing job: $($jobObject.vm.name)"
$jobQueue.remove($jobObject)
}
}
sleep $sleepTimer
}

echo "jobQueue count is $($jobQueue.count): Adding new job: $($vm.name)"
$job = Start-Job -name $vm.name -InitializationScript $jobFunctions -ScriptBlock $scriptBlock -ArgumentList $vm.name, $vcConnection, $statTypes, $outputDirectory
$jobObject     = "" | select vm, job
$jobObject.vm  = $vm
$jobObject.job = $job
$jobQueue.add($jobObject) | Out-Null
}

Get-Job | Wait-Job | Out-Null

#$regex = '([a-zA-Z0-9]+-){4}[a-zA-Z0-9]+.txt'
#gci $outputDirectory | ? { $_.name -match $regex } | % {
#  publishResults "$($outputDirectory)\$($_.name)"
#sleep 3
#}

# Record the time the script started.
$end = Get-Date

echo "Start: $($start), End: $($end)"


Listing all vCenter performance metrics with PowerCLI

Our big data team recently approached me and wanted to know if I could gather real time performance metrics on vCenter virtual machines and publish them to their API.  I asked which metrics they wanted they said, “all of them.”  I proceeded with gathering all of them but wanted to give them a list of all metrics so that they could pick and choose which ones they wanted.  I needed the name of the metric, the stat level it’s recorded at (in case they later wanted something other than real time metrics) and a description of the metric as they aren’t familiar with VMware.

I had never used PowerCLI to gather vCenter performance metrics so my first step was to see how to interact with vCenter and to see which performance metrics were available.  The method that I’m going to describe actually list all vCenter performance metrics and not just virtual machine metrics.  I ended up not using this in my final script to gather virtual machine performance data, but thought it may be useful to be able to look at spreadsheet with the performance metrics so I could easily sort, filter, etc.

I poked around on LucD’s page for information on how to get started.  I’d recommend that you’d check out the links below if you’re interested in using PowerCLI to gather vCenter performance metrics:

In an upcoming post I’ll describe the script I made to gather virtual machine performance metrics and publish them to an API.  For now, here is the script to generate a CSV file with all vCenter performance metrics.


# define where we will save our performance metrics.
$outputFile = "C:\Users\chris\Desktop\perfdumps\vCenterMetrics.csv"

# define a new Powershell credential and log into vCenter with the credentials
$creds = Get-Credential
$vCenter = Connect-VIServer vc6b.vmware.local -Credential $creds -SaveCredentials

# define our vCenter service instance and performance manager.
# https://www.vmware.com/support/developer/converter-sdk/conv43_apireference/vim.ServiceInstance.html
$serviceInstance = Get-View ServiceInstance -Server $vCenter
$perfMgr = Get-View $serviceInstance.Content.PerfManager -Server $vCenter

# get all available performance counters
$counters = $perfMgr.PerfCounter

# create an array where we will store each of our custom objects that will contain the information that we want.
$metrics = @()

foreach ($counter in $counters) {
# create a custom Powershell object and define attributes such as the performance metric's name, rollup type, stat level, summary, etc
$metric = New-Object System.Object
$metric | Add-Member -type NoteProperty -name GroupKey   -value $counter.GroupInfo.Key
$metric | Add-Member -type NoteProperty -name NameKey    -value $counter.NameInfo.Key
$metric | Add-Member -type NoteProperty -name Rolluptype -value $counter.RollupType
$metric | Add-Member -type NoteProperty -name Level      -value $counter.Level
$metric | Add-Member -type NoteProperty -name FullName   -value "$($counter.GroupInfo.Key).$($counter.NameInfo.Key).$($counter.RollupType)"
$metric | Add-Member -type NoteProperty -name Summary    -value $counter.NameInfo.Summary

# add the custom object to our array
$metrics += $metric
}

# each metric object will look simliar to the following.  We can use a select command to gather which attributes we want and export them to a CSV file.
#   GroupKey   : vsanDomObj
#   NameKey    : writeAvgLatency
#   Rolluptype : average
#   Level      : 4
#   FullName   : vsanDomObj.writeAvgLatency.average
#   Summary    : Average write latency in ms

$metrics | select fullname, level, summary | Export-Csv -NoTypeInformation $outputFile

You can view the results on my vCenter 6.0 appliance here.


Re-associating vCloud Director VMs with the correct resource pool

Our team was upgrading ESXi hosts in our vCloud Director environment and all of the VMs must have not have migrated when placing the hosts into maintenance mode so they manually migrated the VMs.  Since we have multiple org vDCs, we have multiple resource pools.  As you may know, when you try to migrate VMs that are in multiple resource pools, the migration wizard only allows you to place the VMs in a single resource pool.  I started seeing the following system alert on all of the vCloud Director VMs:

2015-01-15_20-52-07

I checked vCenter and sure enough all of the VMs were in the root resource pool (the cluster), and no longer in the correct resource pool.  If you migrate the VMs one at a time, the migration wizard automatically places the VM in the correct resource pool, but the team didn’t want to do this as it would take too much time.  If you do have to migrate the VMs manually, you can migrate VMs in bulk and have the migration wizard place the VMs in the correct resource pool if you use the following process:

  1. Select a resource pool.
  2. Select the Related Objects tab.
  3. Select Virtual Machines.
  4. Select all of the Virtual Machines.
  5. Select Migrate.

2015-01-15_21-03-23

Here you can see that the wizard automatically selected the correct resource pool:

2015-01-15_21-03-50

You still have to go through each resource pool and perform the migrations, but the VMs will retain the correct resource pool.  Of course, you could script it all as well.

So I was in the position of having all of the VMs in the wrong resource pool, which causes alarms in vCloud Director and could impact the performance of the virtual machines.  I could have went into vCloud Director and found each of the VMs individually, but this would have taken forever so I decided to use PowerCLI to re-associate the VMs with the correct resource pool.

You’ll need to be connected to vCloud and all of your vCenter instances:

connect-ciserver cloud.corp.local

connect-viserver vc.corp.local

Paste the following function into PowerCLI:


function reassociatevCDRPs {
  <#
  .SYNOPSIS
    Places vCloud Director virtual machines into the correct vCenter resource pools.  
  .DESCRIPTION
    Places vCloud Director virtual machines into the correct vCenter resource pools.  This can be helpful when the VMs are moved from their resource pool during a task such as manual vMotion. 
  .EXAMPLE
  reassociatevCDRPs -org all -promptOnMove $false
  .EXAMPLE
  reassociatevCDRPs -org admin -promptOnMove $true
  #>

  param(
    [Parameter(Mandatory=$true)] 
    [string] $org = "all",
    [Parameter(Mandatory=$true)] 
    [string] $promptOnMove = $false
  )

    $ovdcLookupTable = @{}
    $vmsToMove       = @{}

    # Build lookup tables
    $orgIds = @{}
    $orgNames = @{}
    search-cloud -querytype organization | % { $orgIds[$_.name] = $_.id; $orgNames[$_.id] = $_.name }

    $orgVdcIds       = @{}
    $orgVdcNames     = @{}
    search-cloud -querytype AdminOrgVdc | % { $orgVdcNames[$_.id] = $_.name; $orgvDCIds[$_.name] = $_.id }

    $vCDVMs = search-cloud -querytype adminvm

  $searchCommand = "search-cloud -querytype adminvm"

  if ($org -ne 'all') {
    # Throw an error if the organization is not found in the vCloud instance.  Otherwise add a filter to the search-cloud command to only work on the supplied organization.
    if (! $orgIds[$org]) { throw "Organization $org not found." }  
    $searchCommand += " -filter org==$($orgIds[$org])"
  }

  $vcdVMs = invoke-expression $searchCommand

    $vcdVMs | % { 
      $vcdVM = $_
      
      # Get the resouce pool name in the format of: orgVDC Name (UUID)
      $vcdRPName = "$($orgVdcNames[$vcdVM.Vdc]) ($($_.Vdc.split(':')[3]))" 

      $vcVM = get-vm -id "VirtualMachine-$($vcdVM.moref)"
      $vcRP = $vcVM.resourcepool 
      $vcRPName = $vcRP.name
      
      if ($vcdRPName -ne $vcRPName) {  # Test to see if vCD's resource pool matches vCenter's resource pool. 
        echo "$($vcdVM.name) is in the resource pool '$($vcRPName)' and should be the '$($vcdRPName)' resource pool."
        # Add to list of VMs that need to be moved.   
        $vmsToMove[$vcVM] = get-resourcepool $vcdRPName
      }

    }

    $vmsToMove.keys | % { 
      if ($promptOnMove -eq $true) {
        $response = read-host "Move $($_.name) to the correct resource pool ($($vmsToMove[$_])) ( [y]es, [n]o, [a]ll )?"
        if ($response -eq 'n') { # If the user selects not to move the VM, try the next VM in the list.
          return 
        } 
        elseif ($response -eq 'a') {
          $promptOnMove = $false
        } 
      }
      $resourcePool = $vmsToMove[$_] 
      echo "Moving $vm into resource pool $resourcePool"
      move-vm $_ -Destination $resourcePool | out-null
      #echo "result: $($?)"
    }

}

To correct all VMs with no prompting:

reassociatevCDRPs -org all -promptOnMove $false

To move VMs to a specific org and to prompt before each move:

reassociatevCDRPs -org “org name” -promptOnMove $true

This script won’t move shadow VMs, but those are easy… for each cluster just select them all and drag them into the cluster’s System vDC (uuid) resouce pool.  vShield Edges also won’t be moved.  Those will likely need to be moved into the System vDC resource pool as well or if they are a fenced vApp, they will go into an org vDC resource pool.  Or you can just de-deploy them and they should go to the correct resource pool.


Deploying vCAC/vRA Appliances with PowerCLI

Overview

I’ve been deploying vCAC/vRA quite a bit in my lab and I thought it was time to look into using PowerCLI to automate some of the pieces.  Most of what I’ve done was taken from PowerCLI 5.8 New Feature: Get-OvfConfiguration (Part 1 of 2).

The first example is more basic and won’t use any programming constructs such as loops.  There are separate sections for the vRA SSO and vRA Core appliances and most of the information is redundant.  On the $ovaConfig lines, the details such as common.varoot_password can be found by running the command

$ovaConfig.ToHashTable() | ft -auto

SSO Appliance


connect-viserver localhost

$ovaPath = 'z:\vcac\VMware-Identity-Appliance-2.1.0.0-2007605_OVF10.ova'
$ovaConfig = Get-OvfConfiguration $ovaPath

$ovaConfig.Common.vami.hostname.value                    = 'vcac61a-sso.vmware.local'
$ovaConfig.common.varoot_password.value                  = 'vmware123'
$ovaConfig.common.va_ssh_enabled.value                   = $true
$ovaConfig.IpAssignment.IpProtocol.Value                 = 'IPv4'
$ovaConfig.NetworkMapping.Network_1.Value                = Get-VDSwitch 'vDS1' | Get-VDPortgroup 'vlan3_mgmt'
$ovaConfig.vami.VMware_Identity_Appliance.ip0.value      = '192.168.3.88'
$ovaConfig.vami.VMware_Identity_Appliance.netmask0.value = '255.255.255.0'
$ovaConfig.vami.VMware_Identity_Appliance.gateway.value  = '192.168.3.1'
$ovaConfig.vami.VMware_Identity_Appliance.DNS.value      = '192.168.1.254'

$cluster = get-cluster 'compute2'
$clusterHosts = $cluster | get-vmhost
# Find a random host in the cluster
$vmHost = $clusterHosts[$(get-random -minimum 0 -maximum $clusterHosts.length)]
$datastore = $cluster | get-datastore 'nfs-ds412-hybrid0'

Import-VApp -name vcac61a-sso $ovaPath -OvfConfiguration $ovaConfig -VMHost $vmHost -datastore $datastore -DiskStorageFormat EagerZeroedThick | start-vm

Core Appliance

connect-viserver localhost
$ovaPath = 'z:\vcac\VMware-vCAC-Appliance-6.1.0.0-2077124_OVF10.ova'
$ovaConfig = Get-OvfConfiguration $ovaPath

$ovaConfig.Common.vami.hostname.value                = 'vcac61a.vmware.local'
$ovaConfig.common.varoot_password.value              = 'vmware123'
$ovaConfig.common.va_ssh_enabled.value               = $true
$ovaConfig.IpAssignment.IpProtocol.Value             = 'IPv4'
$ovaConfig.NetworkMapping.Network_1.Value            = Get-VDSwitch 'vDS1' | Get-VDPortgroup 'vlan3_mgmt'
$ovaConfig.vami.VMware_vCAC_Appliance.ip0.value      = '192.168.3.89'
$ovaConfig.vami.VMware_vCAC_Appliance.netmask0.value = '255.255.255.0'
$ovaConfig.vami.VMware_vCAC_Appliance.gateway.value  = '192.168.3.1'
$ovaConfig.vami.VMware_vCAC_Appliance.DNS.value      = '192.168.1.254'

$cluster = get-cluster 'compute2'
$clusterHosts = $cluster | get-vmhost&lt;/pre&gt;
# Find a random host in the cluster
$vmHost = $clusterHosts[$(get-random -minimum 0 -maximum $clusterHosts.length)]
$datastore = $cluster | get-datastore 'nfs-ds412-hybrid0'

Import-VApp -name vcac61a $ovaPath -OvfConfiguration $ovaConfig -VMHost $vmHost -datastore $datastore -DiskStorageFormat EagerZeroedThick | start-vm

Alternate Method

The second method is a little more complex and uses loops, hashes, etc.  I’m probably going to redo this at some point to allow me to specify all appliances or a subset of all appliances to deploy.

# Defaults
$vCenter       = 'localhost'
$password      = 'vmware123';
$sshEnabled    = $true;
$ipProtocol    = 'IPv4';
$vSwitchName   = 'vDS1';
$portgroup     = 'vlan3_mgmt';
$netmask       = '255.255.255.0';
$gateway       = '192.168.3.1';
$dns           = '192.168.1.254';
$powerOn       = $true;
$clusterName   = 'compute2';
$datastoreName = 'nfs-ds412-hybrid0';

connect-viserver $vCenter

$ovfInfo = @{
  VMware_Identity_Appliance = @{
    path       = 'z:\vcac\VMware-Identity-Appliance-2.1.0.0-2007605_OVF10.ova';
    hostname   = 'vcac61a-sso.vmware.local';
    ipAddress  = '192.168.3.88';
  };
  VMware_vCAC_Appliance = @{
    path       = 'z:\vcac\VMware-vCAC-Appliance-6.1.0.0-2077124_OVF10.ova';
    hostname   = 'vcac61a.vmware.local';
    ipAddress  = '192.168.3.89';
  };
}

$ovfInfo.keys | % {
  $ovfConfig = @{
    &quot;vami.hostname&quot;            = $ovfInfo[$_].hostname;
    &quot;varoot-password&quot;          = $password;
    &quot;va-ssh-enabled&quot;           = $sshEnabled;
    &quot;IpAssignment.IpProtocol&quot;  = $ipProtocol;
    &quot;NetworkMapping.Network 1&quot; = $portgroup
    &quot;vami.ip0.$_&quot;              = $ovfInfo[$_].ipAddress;
    &quot;vami.netmask0.$_&quot;         = $netmask;
    &quot;vami.gateway.$_&quot;          = $gateway;
    &quot;vami.DNS.$_&quot;              = $dns;
 };

 $cluster      = get-cluster $clusterName
 $datastore    = $cluster | get-datastore $datastoreName
 $clusterHosts = $cluster | get-vmhost
 # Find a random host in the cluster
 $vmHost       = $clusterHosts[$(get-random -minimum 0 -maximum $clusterHosts.length)]
 $vmName       = ($ovfInfo[$_].hostname).split('.')[0]
 $ovfPath      = $ovfInfo[$_].path

 $deployedVM = Import-VApp -name $vmName $ovfPath -OvfConfiguration $ovfConfig -VMHost $vmHost -datastore $datastore -DiskStorageFormat thin

 if ($deployedVM -and $powerOn) { $deployedVM | start-vm }
}

I’m not sure if it’s possible, but the next step would be to figure out how to configure settings such as SSO and certificates within the appliances.  The main goal of this exercise was to get more familiar with the new Get-OvfConfiguration commandlet.

Here is a version of the script that will work with vRA 6.2:

# Defaults
$vCenter       = 'localhost'
$password      = 'vmware123';
$sshEnabled    = $true;
$ipProtocol    = 'IPv4';
$vSwitchName   = 'vDS1';
$portgroup     = 'vlan3_mgmt';
$netmask       = '255.255.255.0';
$gateway       = '192.168.3.1';
$dns           = '192.168.1.254';
$powerOn       = $true;
$clusterName   = 'compute2';
$datastoreName = 'nfs-ds412-hybrid0';
 
connect-viserver $vCenter
 
$ovfInfo = @{
  VMware_Identity_Appliance = @{
    path       = 'z:\vra\VMware-Identity-Appliance-2.2.0.0-2300183_OVF10.ova';
    hostname   = 'vra62z-sso.vmware.local';
    ipAddress  = '192.168.3.100';
  };
  VMware_vRealize_Appliance = @{
    path       = 'z:\vra\VMware-vCAC-Appliance-6.2.0.0-2330392_OVF10.ova';
    hostname   = 'vra62z.vmware.local';
    ipAddress  = '192.168.3.101';
  };
}
 
$ovfInfo.keys | % {
  $ovfConfig = @{
    "vami.hostname"            = $ovfInfo[$_].hostname;
    "varoot-password"          = $password;
    "va-ssh-enabled"           = $sshEnabled;
    "IpAssignment.IpProtocol"  = $ipProtocol;
    "NetworkMapping.Network 1" = $portgroup
    "vami.ip0.$_"              = $ovfInfo[$_].ipAddress;
    "vami.netmask0.$_"         = $netmask;
    "vami.gateway.$_"          = $gateway;
    "vami.DNS.$_"              = $dns;
 };
 
 $cluster      = get-cluster $clusterName
 $datastore    = $cluster | get-datastore $datastoreName
 $clusterHosts = $cluster | get-vmhost
 # Find a random host in the cluster
 $vmHost       = $clusterHosts[$(get-random -minimum 0 -maximum $clusterHosts.length)]
 $vmName       = ($ovfInfo[$_].hostname).split('.')[0]
 $ovfPath      = $ovfInfo[$_].path
 
 $deployedVM = Import-VApp -name $vmName $ovfPath -OvfConfiguration $ovfConfig -VMHost $vmHost -datastore $datastore -DiskStorageFormat thin
 
 if ($deployedVM -and $powerOn) { $deployedVM | start-vm }
}

vCloud Director vApp Template Report with PowerCLI

I recently needed to get a list of all vApp templates and their datastores within vCloud Director.  I wasn’t able to easily get this information using the Get-* commandlets so I went with the Search-Cloud commandlet.


# Build a hash of arrays where the keys are the shadow VM's id and the values are arrays of the datastores where the shadow VMs live.
$shadowsVMs = @{}
search-cloud -querytype AdminShadowVM | % {
   if ( $shadowsVMs[$_.PrimaryVAppTemplate] -eq $null) {
     $shadowsVMs[$_.PrimaryVAppTemplate] = @($_.DatastoreName)
   }
   else {
     $shadowsVMs[$_.PrimaryVAppTemplate] += $_.DatastoreName
   }
}

# Build a hash where the keys are the org's id and the values are the org's name.  This is used below to display a vApp's org.
$orgNames = @{}
search-cloud -querytype organization | % { $orgNames[$_.id] = $_.name }

# Set the sort option
$sortField = 'Org'

# Use Search-Cloud to find all AdminVMs and filter to see if they are vApp Templates. This is the part I had a hard time with using the Get-* commandlets.
search-cloud -querytype adminvm -property Container, ContainerName, Name, CatalogName, Org, DatastoreName `
                                -filter "IsVappTemplate==True" | `
     select @{N='Org';                   E={ $orgNames[$_.org] }}, `
            @{N='Catalog';               E={ $_.CatalogName} }, `
            @{N='vApp Template';         E={ $_.ContainerName} }, `
            @{N='VM Name';               E={ $_.Name} }, `
            @{N='Datastore';             E={ $_.DatastoreName }}, `
            @{N='Shadow VMs';            E={ $shadowsVMs[$_.Container].count }}, `
            @{N='Shadow VMs Datastores'; E={ $shadowsVMs[$_.Container] -join ', '}} `
     | sort $sortField | ft -auto

If you’d rather send the output to a spreadsheet instead of the console, you can use export-csv on the last line:


# Build a hash of arrays where the keys are the shadow VM's id and the values are arrays of the datastores where the shadow VMs live.
$shadowsVMs = @{}
search-cloud -querytype AdminShadowVM | % {
   if ( $shadowsVMs[$_.PrimaryVAppTemplate] -eq $null) {
     $shadowsVMs[$_.PrimaryVAppTemplate] = @($_.DatastoreName)
   }
   else {
     $shadowsVMs[$_.PrimaryVAppTemplate] += $_.DatastoreName
   }
}

# Build a hash where the keys are the org's id and the values are the org's name.  This is used below to display a vApp's org.
$orgNames = @{}
search-cloud -querytype organization | % { $orgNames[$_.id] = $_.name }

# Set the sort option
$sortField = 'Org'

# Use Search-Cloud to find all AdminVMs and filter to see if they are vApp Templates. This is the part I had a hard time with using the Get-* commandlets.
search-cloud -querytype adminvm -property Container, ContainerName, Name, CatalogName, Org, DatastoreName `
                                -filter "IsVappTemplate==True" | `
     select @{N='Org';                   E={ $orgNames[$_.org] }}, `
            @{N='Catalog';               E={ $_.CatalogName} }, `
            @{N='vApp Template';         E={ $_.ContainerName} }, `
            @{N='VM Name';               E={ $_.Name} }, `
            @{N='Datastore';             E={ $_.DatastoreName }}, `
            @{N='Shadow VMs';            E={ $shadowsVMs[$_.Container].count }}, `
            @{N='Shadow VMs Datastores'; E={ $shadowsVMs[$_.Container] -join ', '}} `
     | export-csv templates.csv