Veeam Backup & Replication - application backup

Note: The description in the article is based on Veeam Backup & Replication 11a, licensed using Veeam Universal License (VUL), which is equivalent to Enterprise Plus.

Application Backup - Application-aware backup

Veeam Backup & Replication offers native support for backup and recovery for several selected applications. This means it ensures data security and transactional consistency during backup, usually using Application-aware processing. It also supports granular recovery at the application item level using Veeam Explorers.

For backing up applications that are not natively supported, we must verify the backup requirements in the application documentation. Some applications don't require special operations to ensure transactional consistency. For other applications, we can use pre-freeze and post-thaw scripts that run inside the VM. These scripts freeze transactions or create a consistent snapshot at the application level (or the application may be terminated, causing downtime for clients).

For VM backup, we can usually use native Veeam Backup & Replication backup through virtualization. For backing up physical servers, and a few specific cases of applications in VMs, we must use Veeam Agent.

Image-level backup

Veeam Backup & Replication (whether natively or using Veeam agent) works fundamentally differently in one aspect when backing up applications compared to other backup applications I've used.

Veeam Backup & Replication always backs up the entire VM or computer/volume (using an agent, if we want to use Application-aware processing, we can't back up only files and folders), so it's an image-level backup. We can't select only a specific Mailbox Database on an Exchange server or a database on an SQL or Oracle database server. This can be quite inconvenient if we have production and (for example) test databases on one server that we don't want to back up (they can take up tens of terabytes).

Domain Controller (DC) and Active Directory Domain Services (AD DS)

Application-consistent (Application Aware) backup of physical (using agent) and virtual (natively) domain controllers is supported. It's recommended to back up the controller that has the most FSMO roles (nothing prevents backing up all DCs). For restoring application items, it doesn't matter which domain controller we back up.

For proper functioning, Application-Aware Processing must be enabled in the backup job. We need Enterprise Administrator or Domain Administrator permissions.

Microsoft recommends operating several DCs, ideally in different locations, that replicate the Active Directory Database (ntds.dit) and contain a complete copy of the directory. When one server is lost, it's often better to rebuild this server than to restore from backup.

Backup is useful for restoring deleted items (but for many situations we can use Active Directory Recycle Bin), when database corruption occurs or in case of a ransomware attack.

Restoring DC or AD items

During recovery, a Non-Authoritative Restore is typically performed, where the restored DC updates the database with the latest changes from others. Veeam Backup & Replication restores VM files, starts in Directory Services Restore Mode (DSRM), applies settings, and restarts in normal mode.

For Authoritative Restore, where the restored DC authoritatively updates other DCs with its own data, we must perform manual steps (How to recover a Domain Controller: Best practices for AD protection).

If we want to restore a deleted AD object, including the original objectSID and objectGUID, a Tombstone object or an object in the AD Recycle Bin must exist. If the Tombstone Lifetime or Deleted Object Lifetime has already passed (the object was physically deleted), then upon restoration, it will receive new objectSID and objectGUID attributes. More in Veeam Explorers User Guide - Considerations and Limitations.

Microsoft Exchange Server

Application-consistent (Application Aware) backup of physical (using agent) and virtual (natively) Exchange servers is supported. This includes various configurations of Database Availability Group (DAG).

For proper functioning, Application-Aware Processing must be enabled in the backup job. By default, Microsoft VSS is used. If the Exchange servers are virtual, we can use native VM backup.

Detailed information is hard to find with Veeam, there are only some old articles, such as How to: Back up Exchange Database Availability Groups (DAGs) with Veeam Backup & Replication. Similarly, we can find an older description of backup using Agent How to Backup a Windows Failover Cluster with Veeam Agent for Microsoft Windows.

Exchange DAG

DAG ensures high availability of mailbox databases by replicating to other servers, with one copy being active and others inactive. It uses Windows Failover Cluster technology. Within a DAG, we can back up only one node (Exchange server) or more/all.

Usually, it's important that transaction logs are shortened (deleted) after backup (Transaction Log Truncation). For DAG, it's enough to back up one node and transaction logs will be deleted on all nodes.

Recommendations

Veeam provides a few recommendations. Create one VM backup job and add all DAG nodes to it. If we need to ensure that backup doesn't affect Exchange service performance, operate one node only with passive databases and back up only this one. During backup time on the Exchange server, don't use other maintenance tasks, etc.

In practice, I encountered a problem where occasionally during backup of one DAG node, a warning appeared:

VSS: Backup job failed. Cannot notify writers about the 'BACKUP FINISH' event. A VSS critical writer has failed.
 Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: 
[{c9d66233-dd01-4acd-bce2-b83bc4dade6c}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]

In discussions, I found that there's a problem if VSS operations occur simultaneously on multiple nodes. The solution is to split Exchange servers into different tasks. It really helped.

Increasing cluster time limits (DAG)

When a VM Snapshot is created or deleted (during backup), the VM is briefly frozen. This can be longer than the default times for Exchange DAG, resulting in a Failover to the backup node. Veeam recommends increasing the timeout for the cluster Tips for DAG Exchange Backup and Replication in vSphere.

[PS] C:\> Get-cluster | fl *subnet*

CrossSubnetDelay          : 1000
CrossSubnetThreshold      : 20
PlumbAllCrossSubnetRoutes : 0
SameSubnetDelay           : 1000
SameSubnetThreshold       : 10

(get-cluster).SameSubnetThreshold = 20
(get-cluster).SameSubnetDelay = 2000   
(get-cluster).CrossSubnetThreshold = 40
(get-cluster).CrossSubnetDelay = 4000

Microsoft SQL Server

Veeam Backup & Replication allows native backup of virtual SQL Server, whether it's a Standalone SQL Server or Always On Availability Group. For proper functioning (natively or with agent), Application-Aware Processing must be enabled in the backup job.

An interesting document on SQL server backup - SQL Server Backup and Restore in a Veeam environment.

When to use Veeam Agent

With Veeam Agent, we can additionally back up virtual SQL Server Failover Cluster (we add a cluster account to a job of type Failover cluster, not individual cluster members) and everything on a physical SQL Server. Agent must also be used if we use RDM (Raw Device Mapping) on a virtual SQL server, disks (LUNs) directly on the disk array.

Veeam Agent has certain limitations listed in Veeam Agent Management Guide - Failover Cluster Support.

VM with SQL AlwaysOn Availability Groups

Veeam published a material Benefits of using SQL AlwaysOn Availability Groups for Virtual Environment & Veeam, where it compares SQL Failover Cluster and SQL AlwaysOn Availability Groups in a virtual environment. The conclusion recommends using SQL AlwaysOn Availability Groups and backing up (natively) at the VM image level. Veeam Backup & Replication analyzes database information and performs a VSS Snapshot with corresponding flags.

Transaction Logs

By default, after a successful backup, transaction logs are processed (Transaction Log Truncation). We can also use transaction log backup (Transaction Log Backup) - Microsoft SQL Server Log Backup, and achieve a shorter RPO (minimum 5 minutes). Logs can only be backed up if the database was properly backed up. For SQL AlwaysOn Availability Groups, logs are backed up from one node, which is selected by certain rules (all nodes must be in the same backup job).

When transaction logs are truncated, the freed space is available for reuse. But the size of the log file does not decrease. We can use the Transaction Log Shrink operation, which Veeam recommends performing regularly. More information about these operations can be found at Microsoft Shrink a file, SQL Server Transaction Log Backup, Truncate and Shrink Operations.

Recommendations

For correct recovery options, it is necessary that the Recovery Model for databases is either Full or Bulk-logged. More at Required Backup Job Settings or at Microsoft Recovery Models (SQL Server).

Veeam in its Best Practice Guide (Microsoft SQL Server) states that it's good to check VSS writers and providers (but in my opinion, more details are missing there) using the commands:

vssadmin list providers
vssadmin list writers

Also, check the performance state of the server, because backup puts a load on I/O operations. For SQL AlwaysOn Availability Groups, we can increase cluster times, just as indicated for Exchange DAG. When backing up transaction logs, a temporary folder <drive with most free space>:\Veeam\Backup\SqlLogBackup is created (we can change this using registry).

Problem with backing up many databases - VSS writer failed

We encountered a problem on an SQL Server with a large number of databases (over 300). Suddenly, the backup started returning an error. The following error appeared in the log:

Failed to create snapshot: Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}].
 Instance ID: [{b2a57bac-1ee9-4bc0-a861-c470f55ed5b9}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT].
 Error code: [0x800423f4].]

The VSS writers listing on the SQL Server returned an error

vssadmin list writers
...
Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {fbf31294-b5ff-49c7-baba-cede9eea3513}
   State: [8] Failed
   Last error: Non-retryable error

There were many errors in the Application Event Log. This problem is described in the article Veeam SQL Server backup – A VSS critical writer has failed. Writer name: [SqlServerWriter]/ Cannot open backup device.

It advises to manually set a larger value for Max Worker Threads, which is described at Microsoft Configure the max worker threads Server Configuration Option. Setting the value to 960 solved the problem and backups are now running. The question is whether such an increase will affect server performance.

Oracle Database

Veeam Backup & Replication offers three different methods for backing up Oracle Database. Natively virtual server at the VM image level, physical or virtual server using Veeam Agent, and best of all, using Oracle Recovery Manager (RMAN) with integration via the certified Veeam Plug-in for Oracle RMAN.

An interesting document on Oracle backup - How to back up and restore an Oracle database - best practices.

Support

Supported Oracle versions on Windows or Linux can be found in the documentation Supported Applications. Other documentation lists certain conditions Considerations and Limitations and prerequisites Required Backup Job Settings. Oracle Database Express Edition (XE) is supported only on Windows. Oracle Real Application Clusters (RAC) is supported only using the Veeam Plug-in for Oracle RMAN.

Image-level Backup

The entire physical or virtual server with Oracle Database can be backed up using image-level backup with Veeam Backup & Replication or Veeam Agent. For proper functioning, Application-Aware Processing must be enabled in the backup job.

Only databases in ARCHIVELOG mode are backed up online. The database is switched to backup mode (ALTER DATABASE BEGIN BACKUP), a Snapshot is created, it's returned to the initialized state and backed up from the Snapshot. If it's in NOARCHIVELOG mode, it's shut down during backup. When we restore a VM with a database in ARCHIVELOG mode, we must switch the database out of backup mode after restoration (ALTER DATABASE END BACKUP).

Archived Redo Logs

After a successful backup, we can process archived redo logs (Archived Log Processing). We can choose to delete logs older than a given number of hours (not recommended less than 24 hours) or exceeding a size. We can also use archived log backup (Archived Log Backup) - Oracle Log Backup (VMware Oracle Archived Log Settings, Agent Oracle Processing Settings). More interesting information can be found in the documentation for Veeam Agent for Linux Oracle Backup.

Veeam Plug-in for Oracle RMAN

In many cases, we find that we need to use the Veeam Plug-in for Oracle RMAN. For backup, Oracle RMAN is then used, where backup is configured and managed, creating transactionally consistent database backups. The Veeam Plug-in provides integration with Veeam Backup & Replication and stores backups in the Veeam Backup Repository. The Veeam Server has access to the created backups.

The advantage is that RMAN is a native Oracle tool recommended for backing up and restoring Oracle databases. It supports all configurations and options of Oracle Database. It backs up selected databases (not the entire server or operating system).

The disadvantage is that it must be managed on the database server side using RMAN scripts. This is also where backup scheduling, backup retention time, etc. are planned. We lose centralized backup management. Backup is also more often configured by the database team, which has the necessary knowledge.

Veeam recommends backing up Oracle Home, using Veeam Backup & Replication or Veeam Agent, when backing up databases using Oracle RMAN. For backing up Oracle RAC, it is recommended to install the plug-in on all nodes so that backup can run from any of them. Backups using the plug-in are transferred directly to the backup repository. If the repository doesn't have a free slot (many tasks are being processed in parallel), the backup will fail. Backups create many files and a lot of metadata, which is stored in the Veeam configuration database.

Veeam Plug-in Usage

At the beginning, it's good to familiarize yourself with the requirements and characteristics of the Oracle environment - Oracle Environment Planning. And the Veeam environment - Veeam Environment Planning.

After setting up the plug-in and connecting to the Backup Server, a Backup Job is automatically created. Its name is automatically generated from the server/cluster name and storage. On the Backup Server, we cannot modify or start the job, but we can view statistics and disable the job (then it can't be started from Oracle RMAN).

The backup creates VAB files (compressed copy of Oracle database), VASM (metadata containing information about the backup) and VACM (backup job metadata).

We need to install and configure the plug-in. During configuration, we enter the address of the Veeam Backup & Replication Server (and port, default 10006), credentials, i.e., the account we use to authenticate against the server and access the storage, and select the Backup Repository for storing backups (encrypted storage is not supported).

For the account we use in the plug-in, it's important what permissions it must have (the documentation is not very good in this regard). We authenticate with this account on the Veeam Backup Server using Windows Authentication, so it can be a local Windows account on this server. Membership in the Users group is sufficient. The account must also have write permission to the selected Backup Repository (Console - Backup Repositories - right-click on the repo and select Access permissions). More info at Veeam Backup Server User.

Last year, the documentation Configuring Plug-in on Linux or Unix contained false information (and similar information is still in Configuring Plug-in on Windows):

The account must have the local Administrator permissions on the Veeam Backup & Replication server.

All backup properties are controlled by RMAN. This means that we also set backup retention rules there - Retention of RMAN Backups and Archived Logs. Alternatively, we can use Backup Copy Job and manage these backups with Veeam. And the actual backup and its execution - Database Protection.

From Oracle RMAN, we can perform all supported types of database recovery. We can also perform recovery from Veeam Backup & Replication using Veeam Explorer for Oracle.

Moving backup files (created using Veeam Plug-in) to another storage

Veeam has a description in the article Importing Backup Files, where I'm missing two steps to make the operation functional.

• move the backup files (the entire folder) to the new Repository
• rescan the new storage, Backup Infrastructure - Backup Repositories - select Rescan on the storage
• scanning will find one new backup, it will appear in Home - Backups - Disk (Imported)
• rescan the original storage, Backup Infrastructure - Backup Repositories - select Rescan on the storage
• delete the Backup Job (it will be created anew with a new name after running the command with map-backup), Home - Jobs, select Delete on the job
• continue on the Oracle Server, where we use OracleRMANConfigTool, first select the new storage, then mapping is performed

OracleRMANConfigTool --set-repositories
OracleRMANConfigTool --map-backup

• modify the RMAN script, change the Repository ID, which we find in veeam_config.xml (maybe the tool should do this itself when selecting the storage, but it failed for me)

Until I performed Rescan of the original storage, the command with map-backup returned an error:

Repository XXX: failed to map: There is an existing backup with the same original ID linked to the job

Until I changed the Repository ID in the RMAN script, it was still trying to store backups on the original storage and the backup ended with an error:

Session failed: Info call failed during backup session  

MySQL and PostgreSQL

Veeam Agent for Linux allows creating transactionally consistent backups of machines with MySQL database and PostgreSQL database. More information in Veeam Agent for Linux - Backup of Database Systems.

When backing up VMs, we can use pre and post thaw scripts. Recommendations for MySQL Backup.

Windows File Server Failover Clusters

For backup, we must use Veeam Agent. A detailed description is in the article How to backup a Windows 2019 file server cluster.