Veeam Backup & Replication - automatic recovery testing

Note: The description in the article is based on Veeam Backup & Replication 11a, licensed using the Veeam Universal License (VUL), which is equivalent to the Enterprise Plus edition.

Recovery Verification

Documentation: SureBackup for VMware, SureBackup for Hyper-V

Let's look at the SureBackup technology, which allows you to test backed-up VMs. For VMware, there is also SureReplica.

SureBackup uses the following components:

• Virtual Lab
• Application Group
• SureBackup Job

The configuration is located in the Veeam Backup & Replication Console under Backup Infrastructure - SureBackup.

Brief Verification Process

• Starts the VM from the backup using Instant Recovery (directly from the backup repository) in an isolated environment
• The VM runs in the Virtual Lab with a special internal network to avoid conflicts with production
• Performs defined tests to verify its functionality
• After completion, unpublishes the VM (turns it off) and creates a report

SureBackup has various limitations. It's important to note that we can only verify VM backups (not agent-based backups) and only on a lab of the same platform (VMware or Hyper-V). Within a single verification job, we cannot combine VMs of different platforms.

Virtual Lab

• Isolated virtual environment (Sandbox) where we can start the verified VMs
• Runs on VMware or Hyper-V infrastructure
• Represents certain objects and resources on the chosen ESXi or Hyper-V host
• Doesn't consume resources on its own, but the started VMs consume resources on the ESXi/Hyper-V host where the Virtual Lab is deployed
• VMs start directly from the Backup Repository using Instant Recovery (on VMware it uses the vPower NFS Service)
• VMs run in read-only mode, and changes are written to the redo log in the Virtual Lab, which is deleted after the session
• Network-isolated from the production environment: on VMware, a Virtual Switch and Port Group are created, on Hyper-V, it's a Private Network
• The network configuration of the VMs is the same as in production (VMs have the same IP addresses and network interfaces), but the networks are isolated
• Communication is possible only through the optional Proxy Appliance, which is connected to both networks, using routing and NAT

The Virtual Lab can be used not only for SureBackup, but also for Universal Application Item Recovery (U-AIR), On-Demand Sandbox (for internal testing, training, and troubleshooting), or staged recovery.

Proxy Appliance

• Deployed optionally if you want to allow network communication to the tested VM (and network-based testing)
• Enables secure communication between the production environment and the isolated network of the Virtual Lab
• It's an auxiliary VM based on Linux, which Veeam deploys on the ESXi/Hyper-V host where the Virtual Lab is created
• Must have an IP address assigned from the production network (has a virtual network adapter to it), and it should be in the same network (VLAN) where the Backup Server is located (for routing communication)
• Within the isolated networks in the Virtual Lab, it uses network adapters (vNICs), one for each isolated network (to communicate with them)

Note: The term network is important. Typically, the production network consists of multiple subnets or VLANs that are routed between each other. However, Veeam seems to use the term network to refer to a single subnet or VLAN. It uses the term Production network for the original network where the VM is, and Isolated network for the same/mapped network in the lab.

IP Masquerading - Network Address Translation (NAT)

• The Proxy Appliance uses address translation (NAT) to communicate with the isolated networks (to the tested VMs)
• The VMs in the lab have their original IP address from the production network range, but are connected to the isolated Lab network
• The Proxy is connected to the production network (VLAN) and has an assigned IP address
• It also has a special range configured, which is not used in production, for example, for the production network 10.0.0.0/24, it uses 10.255.0.0/24
• The Proxy is also connected to the isolated Lab network and has an IP address assigned here, typically the original network gateway address, for example, 10.0.0.1
• The Proxy translates (NATs) communication for target addresses 10.255.0.0/24 to addresses 10.0.0.0/24 in the Lab network, for example, 10.255.0.10 is translated to 10.0.0.10
• When we want to communicate with the tested VMs, we use their IP address and change the second octet in our case, but we must ensure that this communication (the target subnet is unknown in our network) reaches the Proxy Appliance, which will handle the further routing, this can be done using a local routing table on the computers in the same subnet (network) as the Proxy
• On the Backup Server, the routing table is automatically set up when the virtual lab starts
• We can also use Static IP Mapping if we create an Advanced Single-host Virtual Lab, then we can reserve an additional IP address in the production network (in the same subnet as the Proxy) that will be permanently mapped to a specific address in the Lab network on the Proxy Appliance (to which the IP address will be assigned)

The entry in the Route Table on the Backup server may look like this:

Network Destination  Netmask       Gateway     Interface    Metric
10.255.0.0           255.255.255.0 10.0.0.100  10.0.0.200   21

Note: Veeam does not verify whether the Proxy and Backup server are in the same network. So it will allow us to create a configuration that won't work (the Backup server won't be able to communicate with the lab network).

Virtual Lab Network Configuration (Networking Mode)

When creating a Virtual Lab, we choose the networking (Networking). If the basic option is sufficient, everything will be set up automatically. If we need advanced, we need to configure several additional parameters and make some adjustments.

• Basic Single-host Virtual Lab
  • Can be used if everything is connected to the same network (VLAN), i.e., the verified VM, Backup Server, and Proxy Appliance
  • A single isolated virtual network is created, mapped to the production network, created based on the parameters of the network where the Backup Server is located
  • On ESXi, a Resource Pool, VM folder, Standard vSwitch are created, on Hyper-V, a Private virtual switch is created, and the created switches are only used by VMs started in the lab
  • Veeam automatically configures all settings, including the Proxy Appliance
• Advanced Single-host Virtual Lab
  • If we want to verify VMs from multiple networks, the VM has a dependency on a VM from another network, or the verified VM is in a network different from the Backup Server
  • Multiple mapped virtual networks are created, corresponding to the production networks to which the verified VMs are connected
  • On ESXi, a Resource Pool, VM folder, Standard vSwitch are created, on Hyper-V, a Private virtual switch is created, and the created switches are only used by VMs started in the lab
  • Veeam configures the basic network settings in the virtual lab, but we need to check and manually fine-tune them
• Advanced Multi-host Virtual Lab - a special case, can be used for VM backups or VM replicas on different ESXi hosts (VMware only)

Creating a Virtual Lab

• Backup Infrastructure - SureBackup - Virtual Labs

We can create a Virtual Lab on VMware or Hyper-V, or we can connect to an existing lab.

• Name - the name of the Virtual Lab (e.g., VMwareLab, Sandbox-01), optionally add a description
• Host - for VMware, we select the ESXi server where the lab will be created, a folder (not possible for a standalone ESXi) and a Resource Pool (only possible for a standalone ESXi or a cluster with DRS enabled, which requires at least the vSphere Enterprise license) are created for each lab
• Destination - for Hyper-V, we similarly select the host where the lab will be created, a dedicated folder is created for each lab, where the Proxy Appliance files are copied, and it also serves as the Mount Point for the started VMs
• Datastore - for VMware, we can select a datastore for the redo logs (where changes in the running VM are written, and are deleted after the job completes), by default it is stored on the vPower NFS server
• Proxy - if we want to allow communication to the isolated lab, we must configure the Proxy Appliance, it has certain default parameters that we can change, it's important to select the production network (the same VLAN where the Backup Server is) and set the IP address in it, by default, VMs from the lab do not have access to the internet, we can set up the use of an http(s) Proxy (we then need to configure the proxy inside the VMs)

• Networking - we select the network mode: Basic Single-host, Advanced Single-host (we configure additional steps), and Advanced Multi-host (VMware only)

If we choose the Advanced Single-host mode, further configuration is required:

• Isolated Networks - we set the mapping of production networks (selected from the virtualization) to the created isolated networks in the lab (we enter a name or select an existing one, add the VLAN ID), we can create a maximum of 9 networks

• Network Settings - for each created isolated network, we must enter the network parameters, a vNIC is created on the Proxy Appliance, we enter the IP address in this network (gateway) and the subnet mask, which corresponds to the production VLAN, the Masquerade network is set, which we can change, i.e., the NATed addresses, we can enable a DHCP server, we can globally enable communication (routing) between the specified isolated networks, the Proxy will act as a router so that the verified VMs can communicate with each other

• Static Mapping - we can create a mapping of a specific production IP address (from the same network where the Proxy is) to an IP address in the isolated network

Application Group

The tested VMs often need certain services to run, such as DNS, domain controller, database, etc. We can create an Application Group where we specify one or more VMs on which the tested VM depends. We set the start order and role of each VM. The tested VM starts only when all the VMs in the application group are running.

According to the Veeam description, the tested VMs are typically specified in the Linked Jobs. But the VMs defined in the Application Group are also verified. We can define tests to be performed for their verification, just like for the VMs in the Linked Jobs. The VMs in the group run for the entire duration of the SureBackup job. We can configure them to run even after the job completes.

Unfortunately, all the VMs must be of the same platform, either VMware or Hyper-V. Both the VMs in the Application Group and those in the Linked Jobs.

Creating an Application Group

• Backup Infrastructure - SureBackup - Application Groups

Creating an Application Group is straightforward. We specify whether we are creating a VMware or Hyper-V group. We add the VMs that make up the group. For Hyper-V, we select VMs from the existing Hyper-V backups. For VMware, we can select VMs from existing VMware backups, replicas, or Storage Snapshots. The VMs are listed in the order they will start.

For each VM in the group, we can set the recovery verification options and tests. The same can be set for VMs in the Linked Jobs in the SureBackup Job.

Verification Options

For each VM in the group, we can set its role (Role) or manually configure the parameters.

Roles

The predefined roles are DNS server, DC, Global Catalog, Mail Server, SQL Server, Web Server, Veeam Backup for Microsoft Office 365. Choosing a role automatically sets the startup options and test scripts. We can modify these values or choose not to select a role and set them manually.

The roles are described using XML files in the folder %ProgramFiles%\Veeam\Backup and Replication\Backup\SbRoles. We can create our own custom roles.

The SureBackup Job performs various verification tests - Backup Recovery Verification Tests, which depend on the role or manual configuration. These include Heartbeat, ping, and application tests.

Startup Options

• Memory - how much percent of the original VM's memory should be allocated
• Maximum allowed boot time - the VM must boot up within this time, otherwise the verification fails
• Application initialization timeout - how long to wait for applications to start in the VM
• VM heartbeat is present - performs a heartbeat test using VMware Tools / Hyper-V Integration Services
• VM responds to ping on any network interface - tests using ping, network communication must be available (and not blocked by a firewall in the VM)

Test Script

We can use predefined test scripts or provide parameters for our own.

Credentials

By default, the verification scripts run under the account that the Veeam Backup Service is running as. If needed, we can select a different account.

SureBackup Job

The SureBackup Job is a task for verifying the recovery. It combines the settings and policies for verification, such as the Application Group, Virtual Lab, and VM backup. We can run it manually or automatically. The created SureBackup jobs are typically found among the other jobs in Home - Jobs.

Job Workflow

• Starts the Virtual Lab
• Starts the VMs from the Application Group in the specified order and verifies them, they remain running for the duration of the test or we can configure them to run even after the test completes (if we want to perform manual operations)
• Starts the verified VM (from the Linked Job) and verifies it according to the configuration (one at a time or a certain number simultaneously)
• After completion, the VMs are turned off and unpublished, the Virtual Lab is stopped

Detailed description of the steps: SureBackup Job Processing.

Creating a SureBackup Job

When creating a SureBackup Job, we choose the platform: VMware or Hyper-V.

• Name - the name of the job (e.g., SureBackup-Exchange), optionally add a description
• Virtual Lab - we select the virtual lab to be used for the tests, information about the host and datastore is displayed
• Application Group - optionally, we select an application group, we have the option to Keep the application group running after the job completes
• Linked Jobs - optionally, we select one or more Backup Job whose VMs we want to verify, we specify how many VMs can be processed simultaneously, and just like for the Application Group, we can set the verification options, either together for the entire Backup Job or for individual VMs (using the Advanced button)
• Settings - additional settings allow us to perform integrity check of the virtual disk (CRC), if we have an antivirus installed and configured, we can perform malware check for Windows VMs, we can send the test result as an SNMP trap or email (it's important that if in the General Options under E-mail Settings we haven't enabled Notify on Success, we won't receive the message about the successful test)

• Schedule - we can schedule the job to run regularly (the second option is to run it manually) on certain days of the week or month, or chain it with another job and have it run after the completion of the other job

Practical Usage

According to the Veeam description, we typically use the Application Group to include the supporting VMs, and the Linked Job where the actual tested VMs are. But the testing, according to the specified parameters, is performed on all the VMs. So for verifying the functionality of a backed-up VM, it doesn't matter whether it's in the Application Group or the Linked Jobs. In the SureBackup Job, at least one of these must be configured.

The disadvantage of the Linked Jobs is that we can't select specific VMs from the Backup Job, but we have to specify the entire job, and all the VMs in it are verified. A general disadvantage is that when we have a heterogeneous environment with both VMware and Hyper-V, we can't combine them in a single lab. So if we have infrastructure services (on which the verified VMs depend) on a different platform than the verified VMs, we can't perform a comprehensive test.

In practice, we can place the verified VMs in the Application Group and not use any Linked Job. This is also necessary for situations where we want to use the Virtual Lab as a Sandbox for certain manual tests. We must use the Keep the application group running after the job completes setting in the Application Group step to keep the VMs running. The SureBackup Job will start the environment, perform the tests, and then leave it running. When we no longer need the VMs in the application group, we can manually stop the job - Stop Session.

Detailed logs about the progress of the test job can be found on the Veeam Server in the folder %PROGRAMDATA%\Veeam\Backup\.