To protect VMs with Zerto they need to be placed into Virtual Protection Groups (VPGs) which are consistency groupings of VMs that are typically configured on a per application basis. A VM can only exist in 1 VPG at once, you can only failover the entire VPG and you can define the boot order of VMs inside each VPG.
A common request I receive is to specify a boot order between VPGs (a recovery plan) so that you can bring VPGs online in a specified order with time delays and pre/post failover scripts. A perfect use case is bringing Application 1 (I.E a Finance DB) online before Application 2 (I.E a CRM). You could work around this by placing all the VMs that form both applications in the same VPG, but this then removes the fidelity of failing over an individual application in the cases of logical failures. Another use case is running a site wide script that should only be initiated when failing over everything.
Based on this requirement I decided to create the Recovery Plan script. I could spend the next month breaking down each section of the script however I have decided to get some useful scripts out in the wild then explain key elements retrospectively. You can download it from:
The script uses a combination of PowerShell and a CSV to failover each VPG specified in the order listed to the most recent point in time. It utilises the Zerto Cmdlets, many IF statements and the Zerto API. The username and password for logging into the ZVM must be utilized with the PowerShell users.txt configured to match. Each section of the script is explained with comments so you can clearly see how it works. Ensure you have met the requirements listed in:
http://virtuallysober.com/2014/03/11/scripting-with-zerto-basics/
The PowerShell script requires the first 3 sections to be edited which includes the Zerto settings, a location for the CSV and location for the log file. Sorry to state the obvious but it is definitely recommend to configure the script to use your Recovery site Zerto Virtual Manager and store it there! You then need to configure the CSV with the VPGs in the order in which you want them to be recovered:
To explain the settings available in the CSV:
- VPGName – The name of the VPG to failover! – This is both case and space sensitive so triple check that the name of each VPG exactly matches as configured in Zerto. The script will check the VPGName exactly matches before attempting any operation and log the result if not found.
- Action – Whether you want to perform a Failover or Failover Test operation. I recommend starting with Failover Test first. – Parameters; FAILOVER, TEST
- CommitPolicy – When using a Failover operation this sets the default commit policy for operation allowing you to rollback out of the point in time selected. This script will always failover to the most recent point in time so it is therefore recommended to use a manual commit policy. – Parameters; none, commit, rollback
- CommitTime – The time before applying the Commit operation if commit or rollback is selected. – Parameters; 0+ – number of seconds
- ShutdownPolicy – The action to perform on source VMs (if contactable) before performing the failover action on the VPG. I recommend forceshutdown which covers VMs without VMware tools but will try a graceful shutdown first. – Parameters; None, shutdown, forceshutdown
- RunScriptsinTest – Whether you want to run any specified scripts for a TEST operation as well as for FAILOVER. – Parameters; TRUE, FALSE
- PreFailoverScript – A PowerShell script to execute before starting the operation. – Parameters; c:\example\example.ps1 or .\example.ps1 for the same directory as the PowerShell
- PostFailoverScriptDelay – Time delay before executing the PostFailoverScript. Zerto won’t wait for the VPG action to complete before running the post failover script so this is configurable. – Parameters; 0+ – number of seconds
- PostFailoverScript – A PowerShell script to execute after starting the operation and the delay specified. – Parameters; c:\example\example.ps1 or .\example.ps1 for the same directory as the PowerShell
Warning: both pre and post failover scripts will run irrespective of the outcome of the Zerto operation so ensure you have performed a failover test before executing a recovery plan.
Once you have configured the PowerShell default variables, entered your VPGs into the CSV (hopefully with the Action set to TEST to begin with) then you are ready to rock! Every action performed by the script is logged so you can easily troubleshoot any issues. The exact same script can be run for both failover and failback as it just initiates the failover or test, it doesn’t ask where from and to as this is already configured in Zerto.
Any feedback, questions or improvements feel free to comment. Happy scripting,
Joshua
Hi Joshua,
I have modified your script to get around an issue where it would not ignore the certificate when running the Invoke-WebRequest cmdlet. The entire modified script is below:
# Welcome to the Zerto Recovery Plan Failover Script
# Legal Disclaimer:
# This script is written by Joshua Stenhouse is not supported under any Zerto support program or service.
# All scripts are provided AS IS without warranty of any kind.
# The author and Zerto further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose.
# The entire risk arising out of the use or performance of the sample scripts and documentation remains with you.
# In no event shall Zerto, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if the author or Zerto has been advised of the possibility of such damages.
#
# 1. Change this – Edit the Zerto defaults for your Recovery Site here. The ports should normally be left as default unless this has been changed when installing Zerto.
$ZVMIPAddress = “192.168.x.x”
$ZVMPort = “9080”
$ZVMAPIPort = “9669”
$ZVMUser = “xxxxxx”
$ZVMPassword = “xxxxxxxxx”
# 2. CSV Location – enter the name of the recovery plan location
# The CSV columns are VPGName, Action (Failover,Test – if test used then the commit policy, commit time and shutdown setting do nothing) TimeBeforeNextVPGFailover (seconds), CommitPolicy (none,commit,rollback),CommitTime (seconds), ShutdownPolicy (None, shutdown, forceshutdown), PreFailoverScript (powershell in same directory), PostFailoverScript (same as pre but runs after the TimeBeforeNextVPGFailover period).
# The VPGs are recovered in the order that they are listed in the CSV
$csvdirectoryandname = “C:\Zerto\Scripts\ZertoRecoveryPlan.csv”
# 3. Log file settings where to store it (on the target ZVM running the script) and the share name to allow access to the logs
$logDirectory = “C:\Zerto\Scripts\RecoveryPlan”
# No need to edit ANYTHING BELOW THIS LINE
# Setting log file naming convention
$now = Get-Date
$logFile = $logDirectory + “\ZERTO-RecoveryPlanLog-” + $now.ToString(“yyyy-MM-dd”) + “@” + $now.ToString(“HH-mm-ss”) + “.log”
# Logging Session Variables
$startofscript = Get-Date
“$startofscript – Starting Zerto Recovery Plan Failover Script” | out-file $logfile
“$now – ZVMIPAddress = $ZVMIPAddress – ZVMPort = $ZVMPort – ZVMAPIPort = $ZVMAPIPort – ZVMUser = $ZVMUser – ZVMPassword = $ZVMPassword” | out-file $logfile -append
# Adding VMware and Zerto Powershell Commands
function LoadSnapin{
param($PSSnapinName)
if (!(Get-PSSnapin | where {$_.Name -eq $PSSnapinName}))
{Add-pssnapin -name $PSSnapinName}}
LoadSnapin -PSSnapinName “Zerto.PS.Commands”
# Building authentication and URLs for API interaction and authentication
$vpgListApiUrl = “https://” + $ZVMIPAddress + “:”+$ZVMAPIPort+”/v1/vpgs”
$xZertoSessionURI = “https://” + $ZVMIPAddress + “:”+$ZVMAPIPort+”/v1/session/add”
$authInfo = (“{0}:{1}” -f $ZVMUser,$ZVMPassword)
$authInfo = [System.Text.Encoding]::UTF8.GetBytes($authInfo)
$authInfo = [System.Convert]::ToBase64String($authInfo)
$headers = @{Authorization=(“Basic {0}” -f $authInfo)}
add-type @”
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
“@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
$xZertoSessionResponse = Invoke-WebRequest -Uri $xZertoSessionURI -Headers $headers -Method POST
$xZertoSession = $xZertoSessionResponse.headers.get_item(“x-zerto-session”)
$zertoSessionHeader = @{“x-zerto-session”=$xZertoSession}
# Importing the Recovery Plan CSV
$csv = import-csv $csvdirectoryandname
# Logging VPG Check
$now = Get-Date
“$now – Getting list of VPGs from ZVM: $ZVMIPAddress ” | out-file $logfile -append
# Getting VPG list from ZVM for Checking if VPG exists
$vpglist = get-protectiongroups -zvmip $ZVMIPAddress -zvmport $ZVMPort -username $ZVMUser -password $ZVMPassword -site all
# Logging output from VPG list check
$now = Get-Date
“$now – VPGs found:” | out-file $logfile -append
foreach ($_ in $vpglist)
{“$_” | out-file $logfile -append}
# Starting Recovery Plan using the CSV imported
foreach ($vpg in $csv)
{
# Setting the current VPG Name variable from the CSV
$currentvpgselected = $vpg.VPGName
# Logging the START of Actions being performed for this VPG in the CSV
$starttime = get-date
“$starttime – Checking VPG: $currentvpgselected exists” | out-file $logfile -append
# Checking VPG Exists first, also checking the case is correct as the failover cmd issued to Zerto is case sensitive.
if ($vpglist -ccontains $currentvpgselected)
{$VPGEXists = “TRUE”}
else
{$VPGEXists = “FALSE”}
# If the VPG exists then continue with running the script for this VPG. Else it will log the result and will move onto the next VPG.
if ($VPGEXists -eq “TRUE”)
{
# Setting the most recent checkpoint for the VPG as this is required for both FAILOVER and TEST
$cp_list = get-checkpoints -virtualprotectiongroup $currentvpgselected -zvmip $ZVMIPAddress -zvmport $ZVMPort -username $ZVMUser -password $ZVMPassword -confirm:$false
$last_cp = $cp_list[$cp_list.Count-1]
$latesttimeobjectforFailover = $last_cp | select-object Identifier | select -expandproperty Identifier
$latesttimeobjectforTest = $last_cp | select-object timestamp | select -expandproperty timestamp
# Checking if Operation is for FAILOVER or a TEST then running the TEST Actions if the VPG is set to TEST in the ACTION column
if ($vpg.Action -eq “TEST”)
{
# Logging TEST Action
$now = Get-Date
“$now – Starting Failover TEST Actions for VPG: $currentvpgselected” | out-file $logfile -append
# Running the Pre failover TEST script if specified
if ($vpg.PreFailoverScript -ne “” -and $vpg.RunScriptsinTest -eq “TRUE”)
{
# Logging Scripting Action
$currentPreFailoverScript = $vpg.PreFailoverScript
$now = Get-Date
“$now – Running PRE-Failover script: $currentPreFailoverScript” | out-file $logfile -append
# Running PRE failover script
invoke-expression $currentPreFailoverScript
}
# Logging TEST Action
$now = Get-Date
“$now – Performing Failover TEST for VPG: $currentvpgselected to Checkpoint: $latesttimeobjectforTest Using the below cmd:” | out-file $logfile -append
“$now – failovertest-start -virtualprotectiongroup $currentvpgselected -checkpointdatetime $latesttimeobjectforTest -zvmip $ZVMIPAddress -zvmport $ZVMPort -username $ZVMUser -password $ZVMPassword -confirm:$false” | out-file $logfile -append
# Running a TEST as this is set as the Action in the CSV
failovertest-start -virtualprotectiongroup $currentvpgselected -checkpointdatetime $latesttimeobjectforTest -zvmip $ZVMIPAddress -zvmport $ZVMPort -username $ZVMUser -password $ZVMPassword -confirm:$false
# Running POST failover TEST script if specified
if ($vpg.PostFailoverScript -ne “” -and $vpg.RunScriptsinTest -eq “TRUE”)
{
# Setting time to before running Post failover TEST script
$currentPostFailoverScriptDelay = $vpg.PostFailoverScriptDelay
# Logging time to sleep before starting Post failover Test script
$now = Get-Date
“$now – Waiting: $currentPostFailoverScriptDelay Seconds before running POST Failover Test Script” | out-file $logfile -append
# Applying currentPostFailoverScriptDelay value from CSV
sleep $currentPostFailoverScriptDelay
# Logging Scripting Action
$currentPostFailoverScript = $vpg.PostFailoverScript
$now = Get-Date
“$now – Running POST-Failover script: $currentPostFailoverScript” | out-file $logfile -append
# Running POST failover script
invoke-expression $currentPostFailoverScript
}
# Setting time to sleep before failing over next VPG
$currentNextVPGFailoverDelay = $vpg.NextVPGFailoverDelay
# Logging time to sleep before starting TEST of next VPG
$now = Get-Date
“$now – Waiting: $currentNextVPGFailoverDelay Seconds before starting TEST of next VPG” | out-file $logfile -append
# Applying currentNextVPGFailoverDelay value from CSV
sleep $currentNextVPGFailoverDelay
# End of the block for TEST Actions. Will now repeat for the next VPG in the CSV until finished.
}
# Checking if Operation is for FAILOVER or a TEST then running the FAILOVER Actions if the VPG is set to FAILOVER in the ACTION column
if ($vpg.Action -eq “FAILOVER”)
{
# Logging FAILOVER Action
$now = Get-Date
“$now – Starting FAILOVER Actions for VPG: $currentvpgselected” | out-file $logfile -append
# Running a FAILOVER as this is set as the Action in the CSV
# Getting the ID of the VPG then setting the variable using the API. Only needed for a Failover, not Test
$vpglistfromAPI = Invoke-RestMethod -Uri $vpgListApiUrl -TimeoutSec 100 -Headers $zertoSessionHeader
foreach ($vpgsAPI in $vpglistfromAPI | where {$_.Name -eq $currentvpgselected})
{
$vpgid = $vpgsAPI.VpgIdentifier
$vpgidselected = $vpgid | select Id -expandproperty Id
}
# Building URL for failover action
$commitpolicyselected = $vpg.CommitPolicy
$committimeselected = $vpg.CommitTime
$Shutdownpolicyselected = $vpg.ShutdownPolicy
$currentfailoverURL = “https://” + $ZVMIPAddress + “:” + $ZVMAPIPort + “/v1/vpgs/” + $vpgidselected + “/failover?checkpoint=” + $latesttimeobjectforFailover + “&commitPolicy=” + $commitpolicyselected + “&commitValue=” + $committimeselected + “&ShutdownPolicy=” + $Shutdownpolicyselected
# Running the Pre failover script if specified
if ($vpg.PreFailoverScript -ne “”)
{
# Logging Scripting Action
$currentPreFailoverScript = $vpg.PreFailoverScript
$now = Get-Date
“$now – Running PRE-Failover script: $currentPreFailoverScript” | out-file $logfile -append
# Running PRE failover script
invoke-expression $currentPreFailoverScript
}
# Logging FAILOVER Action
$now = Get-Date
“$now – Performing FAILOVER for VPG: $currentvpgselected to Checkpoint: $latesttimeobjectforFailover Using the below cmd:” | out-file $logfile -append
“$now – invoke-webrequest -uri $currentfailoverURL -headers $zertoSessionHeader -method POST” | out-file $logfile -append
# Initiating failover for VPG
invoke-webrequest -uri $currentfailoverURL -headers $zertoSessionHeader -method POST
# Running Post failover script if specified
if ($vpg.PostFailoverScript -ne “”)
{
# Setting time to before running Post failover script
$currentPostFailoverScriptDelay = $vpg.PostFailoverScriptDelay
# Logging time to sleep before starting Post failover script
$now = Get-Date
“$now – Waiting: $currentPostFailoverScriptDelay Seconds before running POST Failover Script” | out-file $logfile -append
# Applying currentPostFailoverScriptDelay value from CSV
sleep $currentPostFailoverScriptDelay
# Logging Scripting Action
$currentPostFailoverScript = $vpg.PostFailoverScript
$now = Get-Date
“$now – Running POST-Failover script: $currentPostFailoverScript” | out-file $logfile -append
# Running POST failover script
invoke-expression $currentPostFailoverScript
# Setting time to sleep before failing over next VPG
$currentNextVPGFailoverDelay = $vpg.NextVPGFailoverDelay
# Logging time to sleep before failing over next VPG
$now = Get-Date
“$now – Waiting: $currentNextVPGFailoverDelay Seconds before FAILOVER over next VPG” | out-file $logfile -append
# Applying currentNextVPGFailoverDelay value from CSV
sleep $currentNextVPGFailoverDelay
}
# End of the block for Failover Actions.
}
# End of the block for when the VPG exists. Will now repeat for the next VPG in the CSV until finished.
}
# End of for each VPG Actions. Will now repeat for the next VPG in the CSV until finished.
if ($VPGEXists -eq “FALSE”)
{
# Logging VPG Not FOUND result
$now = Get-Date
“$now – $currentvpgselected Not FOUND in VPG List. Check VPG exists, spaces and case sensitivity between CSV and Zerto.” | out-file $logfile -append
}
# Logging the END of Actions being performed for this VPG in the CSV
$endtime = Get-Date
“$endtime – Finished actions for VPG: $currentvpgselected” | out-file $logfile -append
# Calculating time it took to execute the script for this VPG
$timedifferencebeforerounding = new-timespan -start $starttime -end $endtime | select-object totalseconds -expandproperty totalseconds
$timedifference = “{0:N2}” -f $timedifferencebeforerounding
“$endtime – Taken: $timedifference Seconds to Execute Actions for VPG: $currentvpgselected” | out-file $logfile -append
“$endtime – Continuing to next VPG in CSV” | out-file $logfile -append
“$endtime” | out-file $logfile -append
}
# End of script time taken
$endofscript = Get-Date
“$endofscript – Ended Zerto Recovery Plan Failover Script” | out-file $logfile -append
# Calculating time it took to execute the script for this VPG
$scripttimedifferencebeforerounding = new-timespan -start $startofscript -end $endofscript | select-object totalseconds -expandproperty totalseconds
$scripttimedifference = “{0:N2}” -f $scripttimedifferencebeforerounding
“$endtime – Taken: $scripttimedifference Seconds to Execute the Recovery Plan” | out-file $logfile -append
# Exiting script
exit
This is an awesome article, thank you. I was wondering if you have any pre failover script examples and post failover examples.
Nice script, but how odd that a “best of breed” DR product costing “best of breed” prices doesn’t have such a simple facility to order vpg failover! I would suggest that a well timed run book facility is at the top of my DR product want-list. Simply gobsmacked that Zerto doesn’t have this!