Simple Powershell Script to Monitor vRealize Automation

Lately I’ve been learning vRealize Automation (vRA) and it has involved bringing the environment up and down frequently as well as breaking and fixing various services to see how the system would respond. I got tired of going into the VAMI (port 5480 of the vRA appliance) and selecting the Refresh button to get status on all of the various services so I created a little Powershell script that will display the status of the vRA services.

The simplest way to invoke the script is with:

get-vRAHealth vra71.vmware.local

Where vra71.vmware.local is my load balancer VIP for vRA. By default the script will continously refresh every 5 seconds.

You can disable the looping like so:

get-vRAHealth vra71.vmware.local -loop $false

And control the refresh interval:

get-vRAHealth vra71.vmware.local -refresh 10

Here is the output:

2016-11-07_22-10-21.png

The script can be found on GitHub and below:

function get-vRAHealth() {
  <#    .SYNOPSIS     Displays health status of vRA components   .DESCRIPTION      Displays health status of vRA components   .EXAMPLE     get-vRAHealth vra71.vmware.local   .EXAMPLE     get-vRAHealth https://vra71.vmware.local -loop $true   .EXAMPLE     get-vRAHealth https://vra71.vmware.local -loop $true $sleep 2   #&amp;amp;gt;

  param(
    [Parameter(Mandatory=$true,Position=0)]
    [string]$url,

    [Parameter(Mandatory=$false,Position=1)]
    [string]$loop=$true,

    [Parameter(Mandatory=$false,Position=2)]
    [Int32]$refresh=5
  ) 

  $uri = [System.Uri] $url

  if ($uri.Host -eq $null -and $uri.OriginalString) {
    $uri = [System.Uri] "https://$($uri.OriginalString)"
  }

  if ($uri.Scheme -eq 'http') {
    $uri = [System.Uri] "https://$($uri.Host)"
  }

  if ($uri.LocalPath -ne '/component-registry/services/status/current') {
    $uri = [System.Uri] "$($uri.AbsoluteUri)component-registry/services/status/current"
  }

  while ($true) {
    clear
    Write-Host "Checking $($uri.AbsoluteUri)"

    try {
      $content = Invoke-WebRequest $uri.AbsoluteUri

      if ($content.StatusCode -eq 200) {
        $json = $content.Content | ConvertFrom-Json
        $json.content | select serviceName, `
	                  @{N='Registered';E={ $_.serviceStatus.serviceInitializationStatus }}, `
	           	  @{N='Available';E={ if (!$_.notAvailable) {'True'} else {'False'}}}, `
	                       lastUpdated, `
		               statusEndPointUrl `
		      | ft -auto
        if ($loop -eq $false) { break }
      } else {
          Write-Host "Unable to access vRA Component Registry. Error: $content.StatusCode"
      }
    } catch {
       Write-Host "Unable to access vRA Component Registry. Error: $_.Exception.Message."
  }
  sleep $refresh
  }
}
Advertisements

Gathering real-time vSphere performance metrics in parallel using PowerCLI

Overview

In a previous post I described how to list all vCenter performance metrics and a project that I was working on.  To recap, I was asked to gather all real-time performance data from all virtual machines in vCenter and publish the results to an API.  As in the previous post on this topic, I’m going to use PowerCLI, but in upcoming posts I’ll show how to get the same data using Perl using the vSphere CLI (I haven’t kept up with the VMware Perl Toolkit in years and it looks like it was merged into the vSphere CLI) and with Ruby using VMware’s rbvmomi.  I’d like to integrate the latter two with docker (I’ve created a docker image that uses the vSphere CLI to gather the data but need to put some finishing touches on it).

Processes vs Threads

In Powershell there are at least two ways of parallelizing your scripts: jobs and runspaces.  Jobs create separate processes whereas runspaces use threads.  You can read more about processes vs threads here.  If you use jobs, you will see a powershell.exe processes for each job you create.  If you use threads, you’ll see a single powershell.exe.  I tried both approaches and found that, for my use case, using jobs was faster than using runspaces so I’m going to show the jobs based approach here.  It takes longer to create a process than it does to create a thread so it takes the jobs based approach a little longer to get going, but once they do they performed more quickly than runspaces.  Since the work that the jobs/threads performed took upwards of a minute and a half, the slower start up times of jobs was worth the trade off.  I imagine as the time it takes for your jobs/threads to perform their work decreases, the balance would start to favor using runspaces.

Getting real-time VM performance data

We will be using the Get-StatType command to retrieve what stat types are available for a VM.  In my case I’m only concerned about real-time data, which is stored on the vCenter server for an hour before it’s rolled up into the database in 5 minute intervals.  Here is an example of retrieving the available real-time stat types for a VM:

$vm = get-vm MyVM

Get-StatType -entity $vm -realtime

If a VM has been powered off for an hour (or if vCenter hasn’t been receiving performance data for the VM from the host for any reason), you’ll receive no results back when you run the Get-StatType command.  If you look at the VM’s real-time performance in vCenter’s performance charts, you’ll see “Performance data is currently not available for this entity.”  Since real-time data is collected every 20 seconds, if you power the VM on and wait 20 seconds, you’ll then be able to retrieve the available stat types as well as see data in the vCenter performance charts.   After doing this, here is the results of selecting the first five available stat types:

Get-StatType -entity $vm -realtime | sort | select -first 5

cpu.costop.summation
cpu.demand.average
cpu.entitlement.latest
cpu.idle.summation
cpu.latency.average

Now we need to retrieve the performance metrics of the stat types for the VM.  My requirements are pretty simple so there is quite a bit more to stats gathering than what I’m going to show.  All I need to get is the real-time data every hour.  Again, I’d suggest reading Lucd’s awesome PowerCLI & vSphere statistics series for more info.  Skimming through it makes me realize I should read it again.

Let’s get the first 5 data points for the ‘cpu.costop.summation’ stat type:

Get-Stat -entity $vm -realtime -stat ‘cpu.costop.summation’ | ? { $_.instance -eq “” } | select -first 5 | format-table -auto

MetricId             Timestamp            Value Unit        Instance
--------             ---------            ----- ----        --------
cpu.costop.summation 6/18/2015 8:17:20 PM     0 millisecond
cpu.costop.summation 6/18/2015 8:17:00 PM     0 millisecond
cpu.costop.summation 6/18/2015 8:16:40 PM     0 millisecond
cpu.costop.summation 6/18/2015 8:16:20 PM     0 millisecond
cpu.costop.summation 6/18/2015 8:16:00 PM     0 millisecond

This command should be pretty self-explanatory.  Please read Lucd’s explanation on the Instance column.  Let’s run the help command against the Get-Stat commandlet to see what options are available:

help Get-Stat

Get-Stat [-Entity] <VIObject[]> [-Common] [-Memory] [-Cpu] [-Disk] [-Network] [-Stat <String[]>] [-Start <DateTime>] [-Finish <DateTime>] [-MaxSamples <Int
32>] [-IntervalMins <Int32[]>] [-IntervalSecs <Int32[]>] [-Instance <String[]>] [-Realtime] [-Server <VIServer[]>] [<CommonParameters>]

Notice how the -stat parameter is formatted; [-Stat <String[]>].  The []s after String means that this parameter accepts an array of values.  This means we can define an array with the stat types we wish to retrieve and then pass it into the Get-Stat commandlet as such:

$statTypes = @(‘cpu.costop.summation’, ‘cpu.demand.average’)

Get-Stat -entity $vm -realtime -stat $statTypes | ? { $_.instance -eq “” }

I actually missed this my first go at the script.  It wasn’t until I implemented the script in Perl when I saw that you could pass in an array of values.  I thought surely you could do the same in PowerCLI so I checked the syntax and sure enough you can.  Before I was iterating through each available stat type and calling Get-Stat (100+ times for each VM). Doh.

Powershell Jobs

For a high-level overview of Powershell Jobs, check out Geek School: Learn How to Use Jobs in PowerShell.

Let’s checkout what job commandlets are available to us:

gcm *job* -module Microsoft.PowerShell.Core

CommandType Name        Version Source
----------- ----        ------- ------
Cmdlet      Debug-Job   3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Get-Job     3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Receive-Job 3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Remove-Job  3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Resume-Job  3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Start-Job   3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Stop-Job    3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Suspend-Job 3.0.0.0 Microsoft.PowerShell.Core
Cmdlet      Wait-Job    3.0.0.0 Microsoft.PowerShell.Core

We will be mostly working with Start-Job so let’s see what the help command has to say about it:

help Start-Job

NAME
Start-Job

SYNTAX
Start-Job [-ScriptBlock] <scriptblock> [[-InitializationScript] <scriptblock>] [-Name <string>] [-Credential <pscredential>] [-Authentication
<AuthenticationMechanism> {Default | Basic | Negotiate | NegotiateWithImplicitCredential | Credssp | Digest | Kerberos}] [-RunAs32] [-PSVersion <version>]
[-InputObject <psobject>] [-ArgumentList <Object[]>]  [<CommonParameters>]

Let’s cover the highlighted options:

  • -ScriptBlock: A chunk of code that our job will execute.
  • -InitializationScript: When your job runs the ScriptBlock, the code in the ScriptBlock isn’t going to have access to any of the variables, functions, etc that were defined in the main script unless you make them available to the newly created job.  Passing an initialization script to your ScriptBlock is a way of doing exactly this.  You can also pass arguments to the ScriptBlock.  I’m not sure when you should use one over the other.  So far I’ve been placing static info into the initialization script and dynamic info into arguments. In my case I’m going to make functions that I define in my main script available to the ScriptBlock by placing them into the initialization script.
  • -Name:  Name of the job.  I’d probably make this something unique like the instance UUID of the VM.  You could also call this the name of the VM, but if you’re working with multiple vCenters, you could run into duplicates.
  • -ArgumentList: A list of variables that we can pass into our ScriptBlock.

Here is what it looks like when I create a job in my script:


$job = Start-Job -name $vm.name -InitializationScript $jobFunctions -ScriptBlock $scriptBlock -ArgumentList $vm.name, $vcConnection, $statTypes, $outputDirectory

Since Start-Job returns the created job, I store it for later use so I can query the state of the job.

The Script

Since this script is for a pretty specific use-case, I doubt it will be of much use to anyone else as is.  My hope is that someone will be able learn something useful from the post itself and/or by looking through the script.  Since I had no idea what I was doing at first, I hope that this will at least get someone up and running more quickly than it took me.  I get the urge to go back and re-write it to be more general so someone could post the results to other endpoints such as building an HTML report, but I have other work to do and don’t want to spend time on something that may never be needed.  I’m also not going to go through the script line by line but I did try to comment it heavily.

There is a function named publishResults() that is wrapped in a variable named $jobFunctions.  Any function that is in the $jobFunctions variable will be available to the code in the script block.  publishResults() is what uploads the performance data stored in CSV files to our API.  You could probably re-write publishResults() to do something like create an HTML report.

The variable $scriptBlock is what contains all the code that each Powershell job will perform.  On line #106, I’ve commented out the publishResults() call since it wouldn’t work in your environment.  If you re-write publishResults() to do something useful, you’ll need to un-comment this line.

The start of the script begins on line #119.  I hope the comments will be enough to explain the rest.  I’ll paste the code below, but it’s probably easier to read here.  If I ever get around to it, I’ll upload my Perl & Ruby examples in this repo as well.

Here is what the script looks like when it’s running:

2015-08-06_10-51-31

The output folder will look similar to:

2015-08-06_11-01-23

Where each file is a VM’s vCenter UUID.

Each file will have contents similar to:

2015-08-06_11-01-44

In this example the VM’s name, VC UUID, power state, cpu.usagemhz.avg label, date, metric value and metric unit.

Here is the code, but like I mentioned earlier, it’s easier to read on Github.


$jobFunctions = {
function publishResults {
&lt;#
.SYNOPSIS
.DESCRIPTION
.EXAMPLE
.EXAMPLE
#&gt;

param(
[Parameter(Mandatory=$true)]
[string] $inFile
)

# Most of what is in this function probably won't be too useful to anyone.  You'd probably need to re-write it.

$method = 'POST'
$uri = [System.Uri] "http://2.2.3.1:6543/api/v1/data_sets?gridname=vm&clustername=vcenter"
$outputFile = 'C:\Users\chris\Documents\powercli\stats\publish-result.txt'

$request = Invoke-RestMethod -uri $uri -method $method -InFile $inFile

if ($request.'Job Status' -eq 'Success') {
add-content $outputFile "Successfully published results for $inFile"
}
else {
add-content $outputFile "Failed to published results for $inFile"
}
}
}

$scriptBlock = {
param($vmName, $vc, $statTypes, $outputDirectory)

# Even though I pass in the vCenter connection, I have to re-establish it.  I'd like to know of a way to prevent this.
$vc = Connect-VIServer $vc.name -Session $vc.sessionId

# Even if I pass in the VM object (and not just the name as is currently being done), I still have to retrieve the VM again.  I'd like to know of a way to prevent this.
try {
$vm = Get-VM $vmName -server $vc
}
catch {
$ErrorMessage = $_.Exception.Message
$FailedItem   = $_.Exception.ItemName
add-content "$($outputDirectory)\log.txt" "$errormessage, $faileditem"
}

$filteredStatTypes = @()

# Get all of the stat types for the time interval (realtime) specified.
# When querying for realtime stats, if no results are returned, this most likely means the VM has been powered off for an hour,
# the host the VM is on wasn't sending data to vCenter, the host the VM is on was disconnected, etc.
$availableStatTypes = get-stattype -entity $vm -realtime -Server $vc | sort

# There is no point continuing if there is no data for the VM so return to the calling context.
if ($availableStatTypes -eq $null) { return }

# We need to place all the stats we want to query into $filteredStatTypes.  If the user doesn't specify any stat types ($statTypes),
# then we just assign all available stats ($availableStatTypes) into $filteredStatTypes.
# If the user does specify stat types, we iterate through each one and make sure it's available in $availableStatTypes and add it
# to $filteredStatTypes.
if ($statTypes -eq $null) {
$filteredStatTypes = $availableStatTypes
}
else {
foreach ($statType in $statTypes) {
if ($statType -in $availableStatTypes) {
$filteredStatTypes += $statType
}
}
}

if ($filteredStatTypes -eq $null) { return }

# Collection info to be used in building the report.
$vmName       = $vm.name
$persistentId = $vm.PersistentId
$vmPowerState = $vm.powerstate
$cluster      = ($vm | get-cluster).name
$outputFile = $outputDirectory + $persistentId + '.txt'

# If the output file already exists, go ahead and delete it.
# if (Test-Path $outputFile) { Add-Content "$($outputDirectory)\dups.txt" "$($vm.name)" ; Remove-Item $outputFile }
if (Test-Path $outputFile) { Remove-Item $outputFile }

# During testing you may want limit the number of results you get back. Uncomment the following line and select the first X amount of stat types to limit the the results.
#$statTypes = $statTypes | select -first 5

$finish = Get-Date

# Since realtime stats only go back an hour before they are rolled up, we only need to get an hours worth of data.
$start = $finish.AddHours(-1)
$stats = Get-Stat -entity $vm -server $vc -realtime -stat $filteredStatTypes -start $start -finish $finish | ? { $_.instance -eq "" }

foreach ($stat in $stats) {
$temp = @()
# Build up a temp array that will contain each of the items to be used to build up a line in the report.
$temp += $vmName, $persistentId, $vmPowerState, $cluster, $stat.MetricId, $stat.Timestamp, $stat.Value, $stat.Unit
# Combine each of the items in the temp array to create a line of comma separated values.
$content = '"' + $($temp -join '","') + '"'
# Store each line into our output file.
Add-Content $outputFile $content
}

try {
#publishResults $outputFile
}
catch {
$ErrorMessage = $_.Exception.Message
$FailedItem   = $_.Exception.ItemName
add-content "$($outputDirectory)\log.txt" "$errormessage, $faileditem, $vmName, $persistentId"
}

# Disconnect this (and only this) instance of vCenter so we don't leave unused sessions lying around.
Disconnect-VIServer $vc -Confirm $false
}

# Record the time the script started.
$start = Get-Date

$vCenterName = 'vc5c.vmware.local'

$vcConnection = Connect-VIServer $vCenterName

# This is where we will store the CSV files with the performance data.
$outputDirectory = 'C:\Users\chris\Desktop\perfdumps\'

# Max amount of jobs (processes) we want running at any time.  You may need to tweak this depending on the resources of your machine.
$maxJobCount = 4

# sleep time in seconds between checking to see if it's okay to run another job.
$sleepTimer = 3

# Here we can define an array of the counters we want to retrieve.  If you have a large list of counters, it may be easier to store them in an external file. For example:
#$statTypes = @('mem.active.average', 'mem.granted.average')
# If you don't define this array, all performance counters will be pulled.

# Retrieve the VMs we want to retrieve stats from.
# When you're testing you may want to only grab a subset of all VMs.  Here are a few examples
# get all VMs in the vCenter(s) you're conneted to: $vms = get-vm
# get all VMs in a specific cluster:                $vms = get-cluster 'resource cluster' | get-vm
# get the first 10 VMs that are powered on:         $vms = get-vm | ? { $_.powerstate -eq 'PoweredOn' } | select -First 10

$vms = get-vm | ? { $_.powerstate -eq 'PoweredOn' } | select -First 10

# Create our job queue.
$jobQueue = New-Object System.Collections.ArrayList

# Main loop of the script.
# Loop through each VM and start a new job if we have less than $maxJobCount outstanding jobs.
# If the $maxJobCount has been reached, sleep 3 seconds and check again.
foreach ($vm in $vms) {
# Wait until job queue has a slot available.
while ($jobQueue.count -ge $maxJobCount) {
echo "jobQueue count is $($jobQueue.count): Waiting for jobs to finish before adding more."
foreach ($jobObject in $jobQueue.toArray()) {
if ($jobObject.job.state -eq 'Completed') {
echo "jobQueue count is $($jobQueue.count): Removing job: $($jobObject.vm.name)"
$jobQueue.remove($jobObject)
}
}
sleep $sleepTimer
}

echo "jobQueue count is $($jobQueue.count): Adding new job: $($vm.name)"
$job = Start-Job -name $vm.name -InitializationScript $jobFunctions -ScriptBlock $scriptBlock -ArgumentList $vm.name, $vcConnection, $statTypes, $outputDirectory
$jobObject     = "" | select vm, job
$jobObject.vm  = $vm
$jobObject.job = $job
$jobQueue.add($jobObject) | Out-Null
}

Get-Job | Wait-Job | Out-Null

#$regex = '([a-zA-Z0-9]+-){4}[a-zA-Z0-9]+.txt'
#gci $outputDirectory | ? { $_.name -match $regex } | % {
#  publishResults "$($outputDirectory)\$($_.name)"
#sleep 3
#}

# Record the time the script started.
$end = Get-Date

echo "Start: $($start), End: $($end)"


Listing all vCenter performance metrics with PowerCLI

Our big data team recently approached me and wanted to know if I could gather real time performance metrics on vCenter virtual machines and publish them to their API.  I asked which metrics they wanted they said, “all of them.”  I proceeded with gathering all of them but wanted to give them a list of all metrics so that they could pick and choose which ones they wanted.  I needed the name of the metric, the stat level it’s recorded at (in case they later wanted something other than real time metrics) and a description of the metric as they aren’t familiar with VMware.

I had never used PowerCLI to gather vCenter performance metrics so my first step was to see how to interact with vCenter and to see which performance metrics were available.  The method that I’m going to describe actually list all vCenter performance metrics and not just virtual machine metrics.  I ended up not using this in my final script to gather virtual machine performance data, but thought it may be useful to be able to look at spreadsheet with the performance metrics so I could easily sort, filter, etc.

I poked around on LucD’s page for information on how to get started.  I’d recommend that you’d check out the links below if you’re interested in using PowerCLI to gather vCenter performance metrics:

In an upcoming post I’ll describe the script I made to gather virtual machine performance metrics and publish them to an API.  For now, here is the script to generate a CSV file with all vCenter performance metrics.


# define where we will save our performance metrics.
$outputFile = "C:\Users\chris\Desktop\perfdumps\vCenterMetrics.csv"

# define a new Powershell credential and log into vCenter with the credentials
$creds = Get-Credential
$vCenter = Connect-VIServer vc6b.vmware.local -Credential $creds -SaveCredentials

# define our vCenter service instance and performance manager.
# https://www.vmware.com/support/developer/converter-sdk/conv43_apireference/vim.ServiceInstance.html
$serviceInstance = Get-View ServiceInstance -Server $vCenter
$perfMgr = Get-View $serviceInstance.Content.PerfManager -Server $vCenter

# get all available performance counters
$counters = $perfMgr.PerfCounter

# create an array where we will store each of our custom objects that will contain the information that we want.
$metrics = @()

foreach ($counter in $counters) {
# create a custom Powershell object and define attributes such as the performance metric's name, rollup type, stat level, summary, etc
$metric = New-Object System.Object
$metric | Add-Member -type NoteProperty -name GroupKey   -value $counter.GroupInfo.Key
$metric | Add-Member -type NoteProperty -name NameKey    -value $counter.NameInfo.Key
$metric | Add-Member -type NoteProperty -name Rolluptype -value $counter.RollupType
$metric | Add-Member -type NoteProperty -name Level      -value $counter.Level
$metric | Add-Member -type NoteProperty -name FullName   -value "$($counter.GroupInfo.Key).$($counter.NameInfo.Key).$($counter.RollupType)"
$metric | Add-Member -type NoteProperty -name Summary    -value $counter.NameInfo.Summary

# add the custom object to our array
$metrics += $metric
}

# each metric object will look simliar to the following.  We can use a select command to gather which attributes we want and export them to a CSV file.
#   GroupKey   : vsanDomObj
#   NameKey    : writeAvgLatency
#   Rolluptype : average
#   Level      : 4
#   FullName   : vsanDomObj.writeAvgLatency.average
#   Summary    : Average write latency in ms

$metrics | select fullname, level, summary | Export-Csv -NoTypeInformation $outputFile

You can view the results on my vCenter 6.0 appliance here.


Re-associating vCloud Director VMs with the correct resource pool

Our team was upgrading ESXi hosts in our vCloud Director environment and all of the VMs must have not have migrated when placing the hosts into maintenance mode so they manually migrated the VMs.  Since we have multiple org vDCs, we have multiple resource pools.  As you may know, when you try to migrate VMs that are in multiple resource pools, the migration wizard only allows you to place the VMs in a single resource pool.  I started seeing the following system alert on all of the vCloud Director VMs:

2015-01-15_20-52-07

I checked vCenter and sure enough all of the VMs were in the root resource pool (the cluster), and no longer in the correct resource pool.  If you migrate the VMs one at a time, the migration wizard automatically places the VM in the correct resource pool, but the team didn’t want to do this as it would take too much time.  If you do have to migrate the VMs manually, you can migrate VMs in bulk and have the migration wizard place the VMs in the correct resource pool if you use the following process:

  1. Select a resource pool.
  2. Select the Related Objects tab.
  3. Select Virtual Machines.
  4. Select all of the Virtual Machines.
  5. Select Migrate.

2015-01-15_21-03-23

Here you can see that the wizard automatically selected the correct resource pool:

2015-01-15_21-03-50

You still have to go through each resource pool and perform the migrations, but the VMs will retain the correct resource pool.  Of course, you could script it all as well.

So I was in the position of having all of the VMs in the wrong resource pool, which causes alarms in vCloud Director and could impact the performance of the virtual machines.  I could have went into vCloud Director and found each of the VMs individually, but this would have taken forever so I decided to use PowerCLI to re-associate the VMs with the correct resource pool.

You’ll need to be connected to vCloud and all of your vCenter instances:

connect-ciserver cloud.corp.local

connect-viserver vc.corp.local

Paste the following function into PowerCLI:


function reassociatevCDRPs {
  <#
  .SYNOPSIS
    Places vCloud Director virtual machines into the correct vCenter resource pools.  
  .DESCRIPTION
    Places vCloud Director virtual machines into the correct vCenter resource pools.  This can be helpful when the VMs are moved from their resource pool during a task such as manual vMotion. 
  .EXAMPLE
  reassociatevCDRPs -org all -promptOnMove $false
  .EXAMPLE
  reassociatevCDRPs -org admin -promptOnMove $true
  #>

  param(
    [Parameter(Mandatory=$true)] 
    [string] $org = "all",
    [Parameter(Mandatory=$true)] 
    [string] $promptOnMove = $false
  )

    $ovdcLookupTable = @{}
    $vmsToMove       = @{}

    # Build lookup tables
    $orgIds = @{}
    $orgNames = @{}
    search-cloud -querytype organization | % { $orgIds[$_.name] = $_.id; $orgNames[$_.id] = $_.name }

    $orgVdcIds       = @{}
    $orgVdcNames     = @{}
    search-cloud -querytype AdminOrgVdc | % { $orgVdcNames[$_.id] = $_.name; $orgvDCIds[$_.name] = $_.id }

    $vCDVMs = search-cloud -querytype adminvm

  $searchCommand = "search-cloud -querytype adminvm"

  if ($org -ne 'all') {
    # Throw an error if the organization is not found in the vCloud instance.  Otherwise add a filter to the search-cloud command to only work on the supplied organization.
    if (! $orgIds[$org]) { throw "Organization $org not found." }  
    $searchCommand += " -filter org==$($orgIds[$org])"
  }

  $vcdVMs = invoke-expression $searchCommand

    $vcdVMs | % { 
      $vcdVM = $_
      
      # Get the resouce pool name in the format of: orgVDC Name (UUID)
      $vcdRPName = "$($orgVdcNames[$vcdVM.Vdc]) ($($_.Vdc.split(':')[3]))" 

      $vcVM = get-vm -id "VirtualMachine-$($vcdVM.moref)"
      $vcRP = $vcVM.resourcepool 
      $vcRPName = $vcRP.name
      
      if ($vcdRPName -ne $vcRPName) {  # Test to see if vCD's resource pool matches vCenter's resource pool. 
        echo "$($vcdVM.name) is in the resource pool '$($vcRPName)' and should be the '$($vcdRPName)' resource pool."
        # Add to list of VMs that need to be moved.   
        $vmsToMove[$vcVM] = get-resourcepool $vcdRPName
      }

    }

    $vmsToMove.keys | % { 
      if ($promptOnMove -eq $true) {
        $response = read-host "Move $($_.name) to the correct resource pool ($($vmsToMove[$_])) ( [y]es, [n]o, [a]ll )?"
        if ($response -eq 'n') { # If the user selects not to move the VM, try the next VM in the list.
          return 
        } 
        elseif ($response -eq 'a') {
          $promptOnMove = $false
        } 
      }
      $resourcePool = $vmsToMove[$_] 
      echo "Moving $vm into resource pool $resourcePool"
      move-vm $_ -Destination $resourcePool | out-null
      #echo "result: $($?)"
    }

}

To correct all VMs with no prompting:

reassociatevCDRPs -org all -promptOnMove $false

To move VMs to a specific org and to prompt before each move:

reassociatevCDRPs -org “org name” -promptOnMove $true

This script won’t move shadow VMs, but those are easy… for each cluster just select them all and drag them into the cluster’s System vDC (uuid) resouce pool.  vShield Edges also won’t be moved.  Those will likely need to be moved into the System vDC resource pool as well or if they are a fenced vApp, they will go into an org vDC resource pool.  Or you can just de-deploy them and they should go to the correct resource pool.


Deploying vCAC/vRA Appliances with PowerCLI

Overview

I’ve been deploying vCAC/vRA quite a bit in my lab and I thought it was time to look into using PowerCLI to automate some of the pieces.  Most of what I’ve done was taken from PowerCLI 5.8 New Feature: Get-OvfConfiguration (Part 1 of 2).

The first example is more basic and won’t use any programming constructs such as loops.  There are separate sections for the vRA SSO and vRA Core appliances and most of the information is redundant.  On the $ovaConfig lines, the details such as common.varoot_password can be found by running the command

$ovaConfig.ToHashTable() | ft -auto

SSO Appliance


connect-viserver localhost

$ovaPath = 'z:\vcac\VMware-Identity-Appliance-2.1.0.0-2007605_OVF10.ova'
$ovaConfig = Get-OvfConfiguration $ovaPath

$ovaConfig.Common.vami.hostname.value                    = 'vcac61a-sso.vmware.local'
$ovaConfig.common.varoot_password.value                  = 'vmware123'
$ovaConfig.common.va_ssh_enabled.value                   = $true
$ovaConfig.IpAssignment.IpProtocol.Value                 = 'IPv4'
$ovaConfig.NetworkMapping.Network_1.Value                = Get-VDSwitch 'vDS1' | Get-VDPortgroup 'vlan3_mgmt'
$ovaConfig.vami.VMware_Identity_Appliance.ip0.value      = '192.168.3.88'
$ovaConfig.vami.VMware_Identity_Appliance.netmask0.value = '255.255.255.0'
$ovaConfig.vami.VMware_Identity_Appliance.gateway.value  = '192.168.3.1'
$ovaConfig.vami.VMware_Identity_Appliance.DNS.value      = '192.168.1.254'

$cluster = get-cluster 'compute2'
$clusterHosts = $cluster | get-vmhost
# Find a random host in the cluster
$vmHost = $clusterHosts[$(get-random -minimum 0 -maximum $clusterHosts.length)]
$datastore = $cluster | get-datastore 'nfs-ds412-hybrid0'

Import-VApp -name vcac61a-sso $ovaPath -OvfConfiguration $ovaConfig -VMHost $vmHost -datastore $datastore -DiskStorageFormat EagerZeroedThick | start-vm

Core Appliance

connect-viserver localhost
$ovaPath = 'z:\vcac\VMware-vCAC-Appliance-6.1.0.0-2077124_OVF10.ova'
$ovaConfig = Get-OvfConfiguration $ovaPath

$ovaConfig.Common.vami.hostname.value                = 'vcac61a.vmware.local'
$ovaConfig.common.varoot_password.value              = 'vmware123'
$ovaConfig.common.va_ssh_enabled.value               = $true
$ovaConfig.IpAssignment.IpProtocol.Value             = 'IPv4'
$ovaConfig.NetworkMapping.Network_1.Value            = Get-VDSwitch 'vDS1' | Get-VDPortgroup 'vlan3_mgmt'
$ovaConfig.vami.VMware_vCAC_Appliance.ip0.value      = '192.168.3.89'
$ovaConfig.vami.VMware_vCAC_Appliance.netmask0.value = '255.255.255.0'
$ovaConfig.vami.VMware_vCAC_Appliance.gateway.value  = '192.168.3.1'
$ovaConfig.vami.VMware_vCAC_Appliance.DNS.value      = '192.168.1.254'

$cluster = get-cluster 'compute2'
$clusterHosts = $cluster | get-vmhost&lt;/pre&gt;
# Find a random host in the cluster
$vmHost = $clusterHosts[$(get-random -minimum 0 -maximum $clusterHosts.length)]
$datastore = $cluster | get-datastore 'nfs-ds412-hybrid0'

Import-VApp -name vcac61a $ovaPath -OvfConfiguration $ovaConfig -VMHost $vmHost -datastore $datastore -DiskStorageFormat EagerZeroedThick | start-vm

Alternate Method

The second method is a little more complex and uses loops, hashes, etc.  I’m probably going to redo this at some point to allow me to specify all appliances or a subset of all appliances to deploy.

# Defaults
$vCenter       = 'localhost'
$password      = 'vmware123';
$sshEnabled    = $true;
$ipProtocol    = 'IPv4';
$vSwitchName   = 'vDS1';
$portgroup     = 'vlan3_mgmt';
$netmask       = '255.255.255.0';
$gateway       = '192.168.3.1';
$dns           = '192.168.1.254';
$powerOn       = $true;
$clusterName   = 'compute2';
$datastoreName = 'nfs-ds412-hybrid0';

connect-viserver $vCenter

$ovfInfo = @{
  VMware_Identity_Appliance = @{
    path       = 'z:\vcac\VMware-Identity-Appliance-2.1.0.0-2007605_OVF10.ova';
    hostname   = 'vcac61a-sso.vmware.local';
    ipAddress  = '192.168.3.88';
  };
  VMware_vCAC_Appliance = @{
    path       = 'z:\vcac\VMware-vCAC-Appliance-6.1.0.0-2077124_OVF10.ova';
    hostname   = 'vcac61a.vmware.local';
    ipAddress  = '192.168.3.89';
  };
}

$ovfInfo.keys | % {
  $ovfConfig = @{
    &quot;vami.hostname&quot;            = $ovfInfo[$_].hostname;
    &quot;varoot-password&quot;          = $password;
    &quot;va-ssh-enabled&quot;           = $sshEnabled;
    &quot;IpAssignment.IpProtocol&quot;  = $ipProtocol;
    &quot;NetworkMapping.Network 1&quot; = $portgroup
    &quot;vami.ip0.$_&quot;              = $ovfInfo[$_].ipAddress;
    &quot;vami.netmask0.$_&quot;         = $netmask;
    &quot;vami.gateway.$_&quot;          = $gateway;
    &quot;vami.DNS.$_&quot;              = $dns;
 };

 $cluster      = get-cluster $clusterName
 $datastore    = $cluster | get-datastore $datastoreName
 $clusterHosts = $cluster | get-vmhost
 # Find a random host in the cluster
 $vmHost       = $clusterHosts[$(get-random -minimum 0 -maximum $clusterHosts.length)]
 $vmName       = ($ovfInfo[$_].hostname).split('.')[0]
 $ovfPath      = $ovfInfo[$_].path

 $deployedVM = Import-VApp -name $vmName $ovfPath -OvfConfiguration $ovfConfig -VMHost $vmHost -datastore $datastore -DiskStorageFormat thin

 if ($deployedVM -and $powerOn) { $deployedVM | start-vm }
}

I’m not sure if it’s possible, but the next step would be to figure out how to configure settings such as SSO and certificates within the appliances.  The main goal of this exercise was to get more familiar with the new Get-OvfConfiguration commandlet.

Here is a version of the script that will work with vRA 6.2:

# Defaults
$vCenter       = 'localhost'
$password      = 'vmware123';
$sshEnabled    = $true;
$ipProtocol    = 'IPv4';
$vSwitchName   = 'vDS1';
$portgroup     = 'vlan3_mgmt';
$netmask       = '255.255.255.0';
$gateway       = '192.168.3.1';
$dns           = '192.168.1.254';
$powerOn       = $true;
$clusterName   = 'compute2';
$datastoreName = 'nfs-ds412-hybrid0';
 
connect-viserver $vCenter
 
$ovfInfo = @{
  VMware_Identity_Appliance = @{
    path       = 'z:\vra\VMware-Identity-Appliance-2.2.0.0-2300183_OVF10.ova';
    hostname   = 'vra62z-sso.vmware.local';
    ipAddress  = '192.168.3.100';
  };
  VMware_vRealize_Appliance = @{
    path       = 'z:\vra\VMware-vCAC-Appliance-6.2.0.0-2330392_OVF10.ova';
    hostname   = 'vra62z.vmware.local';
    ipAddress  = '192.168.3.101';
  };
}
 
$ovfInfo.keys | % {
  $ovfConfig = @{
    "vami.hostname"            = $ovfInfo[$_].hostname;
    "varoot-password"          = $password;
    "va-ssh-enabled"           = $sshEnabled;
    "IpAssignment.IpProtocol"  = $ipProtocol;
    "NetworkMapping.Network 1" = $portgroup
    "vami.ip0.$_"              = $ovfInfo[$_].ipAddress;
    "vami.netmask0.$_"         = $netmask;
    "vami.gateway.$_"          = $gateway;
    "vami.DNS.$_"              = $dns;
 };
 
 $cluster      = get-cluster $clusterName
 $datastore    = $cluster | get-datastore $datastoreName
 $clusterHosts = $cluster | get-vmhost
 # Find a random host in the cluster
 $vmHost       = $clusterHosts[$(get-random -minimum 0 -maximum $clusterHosts.length)]
 $vmName       = ($ovfInfo[$_].hostname).split('.')[0]
 $ovfPath      = $ovfInfo[$_].path
 
 $deployedVM = Import-VApp -name $vmName $ovfPath -OvfConfiguration $ovfConfig -VMHost $vmHost -datastore $datastore -DiskStorageFormat thin
 
 if ($deployedVM -and $powerOn) { $deployedVM | start-vm }
}

Using Powershell to create Veeam Backup jobs of vCloud vApps

When creating Veeam vCloud backup jobs you have a lot of options.  You can specify the entire vCloud environment, an organization, an org vDC, etc.  Usually we create backup jobs that point to a vCloud organization.  This is nice because the next time the Veeam backup job runs, it will pick up any new vApps that have been created in vCloud since the last backup job  run.  Unfortunately, we have one organization that always has failures (all other backup jobs are fine), and it seems like each time the backup job is ran, it’s different vApps that fails.  If I create a backup job with a single vApp from this same org, it always succeeds.  As a test I wanted to create a single backup job for each vApp in the org, but obviously doing this by hand would be a pain so I decided to look at Veeam’s Powershell providers.  Basically the idea is to see if there is any difference between running one Veeam job with all vApps in the org or running many Veeam jobs at once with each job containing a single vApp.

This was a quick script that I tried to create as fast as possible.  I did no research and read no documentation (beyond Powershell’s built-in documentation) so it’s probably not optimal.  It’s as bare-bones as possible as there is no error handling, etc.

# Set environment variables
$vcloudServerName = "cloud.vmware.local"
$backupRepoName = "repo.vmware.local"
$proxyName = "VMware Backup Proxy"
$orgVdcName = "Test Org vDC*"

# Set Veeam “Backup Infrastructure” items
$vcdServer = get-vbrserver -name $vcloudServerName
$backupRepo = Get-VBRBackupRepository -name $backupRepoName
$proxy = get-vbrviproxy -name $proxyName

# Get all vCloud vApps. Doesn’t seem like you can provide more than one filter so need to filter out the org/org vDC vApps with another command.
$vcdVapps = Find-VBRvCloudEntity -Server $vcdServer -vApp

# Get all of the vApps in the org vDC
$vcdVapps = $vcdVapps | ? { $_.Type -eq 'vApp' -and $_.Path -match $orgVdcName }

# Create a Veeam job for each DCO vApp, and store each job into $vbrJobs
$vbrJobs = $vcdVapps | % { Add-VBRvCloudJob -name "Test Org vDC - $($_.name)" -description "Created automatically by Powershell - $([environment]::username)" -entity $_ -BackupRepository $backupRepo }

# Set the proxy for each job.  This is only if you don't want automatic proxy selection to take place. Veeam always selects a proxy halfway across the world so we need to manually select a local proxy.
$vbrJobs | % { set-vbrjobproxy -job $_ -proxy $proxy }

# Create a new job schedule
$jobScheduleOptions = New-VBRJobScheduleOptions

# Set the daily run time to run at 11 PM.
$jobScheduleOptions.OptionsDaily.time = "$(get-date -format d) 11:00 PM"

# Apply the schedule to all the jobs
$vbrJobs | % { Set-VBRJobScheduleOptions $_ -options $jobScheduleOptions }

# Enable the schedule
$vbrJobs | % { Enable-VBRJobSchedule -job $_ }

# Set the jobs to disabled (This was just done so I could verify that the jobs looked good before than ran)
$vbrJobs | % { Disable-VBRJob -job $_ }

# You can enable the jobs with
$vbrJobs | % { Enable-VBRJob -job $_ }

Using PowerCLI and ovftool to move VMs between vCenters

In this post I’ll show how to use PowerCLI and ovftool to move a VM between two vCenter servers. You can use multiple methods to export a VM from vCenter to a temporary location and then import it into a vCenter from said location, but with this method there is no temporary location used.The network traffic used to transfer the VM goes through the client where the script is ran so you probably want to run it from a location that makes sense. The traffic will go from Source vCenter > where the script is ran > Destination vCenter.The script below is just a listing of commands, you’ll probably want to wrap some of it in functions and add some error handling.

# Configuration
$ovftool = "ovftool.exe"  # If the ovftool.exe is not in your path, you need to specify the full path here.
$sourceVM = 'ovftest'
$sourceVIServer = 'vCenter1.vmware.local'
$targetVIServer = 'vCenter2.vmware.local'
$targetDatacenter = 'Super Datacenter'
$sourceNetwork = 'VM Network'  # This is the portgroup that the VM is currently on.
$targetNetwork = 'VLAN188'  # This is the portgroup that you want the VM placed onto.
$targetCluster = 'Super Cluster'
$targetDatastore = 'VM_Template_Transfer'

# Connect to the source and destination vCenters.
Connect-VIServer $sourceVIServer
Connect-VIServer $targetVIServer

# Assign the vCenters in a lookup table to make things easy to access.
$VIServers = @{
  $DefaultVIServers[0].name = $DefaultVIServers[0];
  $DefaultVIServers[1].name = $DefaultVIServers[1]
}

# Get the moref of the source VM
$sourceVMMoref = (get-vm $sourceVM -Server $VIServers[$sourceVIServer]).extensiondata.moref.value

# Get a session ticket for the source and destination vCenter servers.  This is what allows us to specify a vCenter server as the source and destination with the ovftool.
echo "sourceVIServer = $($VIServers.$sourceVIServer)"
$sourceSession = Get-View -server $VIServers.$sourceVIServer -Id sessionmanager
$sourceTicket = $sourceSession .AcquireCloneTicket()

echo "targetVIServer = $($VIServers.$targetVIServer)"
$targetSession = Get-View -server $VIServers.$targetVIServer -Id sessionmanager
$targetTicket = $targetSession .AcquireCloneTicket()

# Build the parameters that will be used with the ovftool.
$sourceTicket = "--I:sourceSessionTicket=$($sourceTicket)"
$targetTicket = "--I:targetSessionTicket=$($targetTicket)"
$datastore = "--datastore=$($targetDatastore)"
$network = "--net:$($sourceNetwork)=$($targetNetwork)"
$source = "vi://$($sourceVIServer)?moref=vim.VirtualMachine:$($sourceVMMoref)"
$destination = "vi://$($targetVIServer)/$($targetDatacenter)/host/$($targetCluster)/"

# Display the command that will be ran.
echo $datastore $network $sourceTicket $targetTicket $source $destination

# Use PowerCLI to run the ovftool.  PowerCLI makes it easy to run commands with multiple parameters.  Sometimes this can be tricky to do with Windows.
& $ovftool $datastore $network $sourceTicket $targetTicket $source $destination