Veeam Backup & Replication - data transfer and backup or restore speed

Note: The description in the article is based on Veeam Backup & Replication 11a, licensed with Veeam Universal License (VUL), similar to Enterprise Plus.

Network Path for Data Transfer

During backup or restore, a large amount of data is transferred. Therefore, besides the necessary hardware performance for processing, it is crucial to consider the networks through which the data is transferred and their bandwidth. This is influenced by the backup topology, which offers several options. Another factor affecting speed is the type of data (size) being transferred. Practically, whether the data is reduced (primarily deduplicated and compressed) or raw.

Generally, data transfer during backup (during restore it is the opposite) occurs as follows:

• The backup source (server) has data stored (on local disks or) on a storage system accessible via SAN network
• Backup Proxy (Data Mover) or Veeam Agent or Veeam Plugin reads data from the source, either running directly on the source or transferring over the LAN network
• Backup Proxy or Veeam Agent performs data reduction (deduplication and compression)
• Backup Proxy or Veeam Agent or Veeam Plugin transfers the (reduced) data to the Backup Repository, sometimes the Proxy runs on the same server as the Repository, in other cases it transfers over the LAN network
• Backup Repository may have storage connected from a storage system via SAN network

There are other options not covered by the above scheme, such as using Direct Storage Access (Backup from Storage Snapshot).

Location of Backup Proxy

A significant difference is where the Backup Proxy runs. There are many variants, but let's look at the basic one where all roles are on the Backup Server. For VMware vSphere, it runs on the Backup Server along with the Backup Repository. In the Network transport mode, data is transferred from the source over the network, and data reduction occurs before storage. For storage, data does not need to be transferred over the network to the Backup Repository, but is stored directly on the storage (via SAN).

For Microsoft Hyper-V (and similarly for Veeam Agent or Veeam Plugin installed on the backed-up server), we can use On-Host Proxy, which runs on the Hyper-V server where the backed-up VM is located. Thus, data is not transferred over the network but read directly from the server's storage (via SAN). Data reduction is performed, and the reduced data is sent over the network to the Backup Repository. There, it is stored on the storage (via SAN).

Using iSCSI SAN for Data Transfer

Veeam documentation (e.g., Backup Architecture - On-Site Backup) often states that data between Backup Proxy and Backup Repository is transferred over LAN. I never considered it could be different. However, I noticed that some backups were too fast for a 1 Gbps network. Monitoring showed almost no traffic on the Backup server's LAN adapters during backup.

Eventually, I concluded (by studying traffic and Veeam logs on servers) that Veeam components (Source Data Mover) transferring data to the Backup Repository (Target Data Mover) select the network path to use. They receive instructions and a list of all IP addresses for the Backup Repository from the Backup Server. They test their availability and likely also quality/speed and choose the best one.

If we use iSCSI SAN network for connecting disks from the system, Veeam likely treats it the same as LAN. This is partly logical because both networks use Ethernet and TCP/IP. Therefore, they can communicate identically. When we normally communicate between servers, we use IP addresses from the LAN, so the traffic goes that way. But Veeam also uses information from iSCSI adapters.

We can have a source server connected to one or more LAN (VLAN) at 1 Gbps and also to iSCSI SAN at 10 Gbps. If the backup server (Backup Repository) is connected to the same LAN and SAN networks, the faster SAN network is chosen for data transfer. This is practically applied when backing up Microsoft Hyper-V, Veeam Plug-in for Oracle RMAN, and likely also Veeam Agent.

In the documentation, I found only a part related to this topic: Network Traffic Management. It provides information on how to limit this behavior if needed: Specifying Preferred Networks. The descriptions of how backup works are very brief:

Veeam Data Movers on the backup proxy and backup repository establish a connection with each other for data transfer.

Veeam Agent service that runs on the protected computer and Veeam DataMover that runs on the backup repository establish a connection with each other for data transfer.

Backup and Restore Performance

The basic question is what data to consider as backup speed or data recovery speed. In practice, we care about how long it takes to back up or restore a certain amount of data. A clear metric is the average speed in MB per second. This can be the speed of network transfer or processing speed.

Job Run Statistics

Veeam Backup & Replication displays statistics for jobs, providing various information - Viewing Real-Time Statistics. However, it's essential to understand them well (and not trust everything blindly).

• Processed - total size of the VM (size of all disks) or occupied disk space during agent backup
• Read - volume of data read by the source Data Mover before deduplication and compression; for VM, the occupied disk space; for incremental backup, only changed blocks (CBT); for agent, CBT or filtered data for backup
• Transferred - volume of data transferred from the source Data Mover to the target Data Mover, i.e., after deduplication and compression; for Hyper-V and agent, data transferred over the LAN network
• Duration - job run time
• Processing rate - average data processing speed, calculated as the ratio of read data (Read) to job duration (Duration); in practice, the reported speed is often much higher than this calculation, sometimes even higher than the disk read speed (in VM details)
• Throughput - for the last job run, a color graph shows the read and transfer speed over time (hover over a specific time to see details)

Performance Bottlenecks

Veeam Backup & Replication tries to identify bottlenecks in the data transfer process during backup - Performance Bottlenecks. It displays the percentage utilization of various resources (components), helping optimize resource usage and data flow efficiency. In the statistics of the completed job, we see Bottleneck, indicating the most utilized processing step. Hovering over the data shows percentages for all phases, also listed in the job log. The log also provides these values for individual objects (VM, computer).

Resource utilization (percentages) shows the time components are busy during the job. Efficient data flow assumes all parts work approximately the same time. If a component is inefficient, it works 100% of the time while others are idle, waiting for data transfer.

Data channel points (processing stages) are:

• Source - component reading the source disk, responsible for obtaining data from the source storage
• Proxy - Backup Proxy component, responsible for data processing
• Network - network queue write component, responsible for obtaining processed data from Backup Proxy and sending it over the network to Backup Repository
• Target - component writing to the target disk (backup storage)

Note: If using WAN Acceleration, Source and Target WAN accelerators are added.

I found an interesting PDF document: Extremely fast processing to deliver the MAXIMUM performance.

Practical Examples of Backup and Restore Speed

Here are examples of various backup types and real achieved speeds in an environment with a 1 Gbps LAN and 10 Gbps iSCSI SAN. I aim to provide realistic values, not always the Processing rate. Something that matches the ratio of read data to job duration. Always for Active Full Backup.

Veeam Agent

During backup and restore using Veeam Agent, communication occurs over the LAN network (the server with the Agent did not have iSCSI SAN connected). The agent performs deduplication and compression, so less data is transferred over the network to storage.

During backup and restore on a 1 Gbps network, I achieve speeds of 100 to 130 MBps. Restoring 6 TB took 13:15:23, with an average speed of 132 MBps.

VMware Network Mode

During backup and restore of VMware in network transport mode [nbd/nbdssl], the VMkernel network interface is used. If we have a 1 Gbps network, we can achieve quite poor throughput. In many discussions, people report speeds of 20 to 50 MBps, but some responses suggest it should work at full network speed. Old documentation (Network Mode) directly states that throttling mechanisms are implemented for the VMware Management Interface, achieving 40% throughput. I have not found confirmation of this information.

During backup and restore on a 1 Gbps network, I achieve speeds of 50 to 60 MBps. Restoring 25.3 GB (30 GB) took 9:51, with an average speed of 52 MBps.

VMware Direct Storage Access

If we use the Direct SAN transport mode [san], data is read/written via the SAN network, which typically has higher speed than LAN. For backup, we can use Backup from Storage Snapshot.

Backup of type Backup from Storage Snapshot on a 10 Gbps network achieves speeds up to 400 MBps (but often less, 200 to 300 MBps). However, restore in tests was slower. Restore of 25.3 GB (30 GB) took 5:47, with an average speed of 89 MBps.

VMware Virtual Appliance

Various materials recommend using Hot-Add Proxy for VMware restore, i.e., Virtual Appliance transport mode [hotadd]. It will read/write data from the source via SAN (direct access to Datastore), but then the question is what network we have for communication with the Repository. The Proxy performs deduplication and compression, so less data is transferred over the network to storage. The VMkernel network interface is not used for transfer, so its potential throttling does not apply (it should always be faster than Network Mode).

I have not tested this mode yet.

Microsoft Hyper-V

During backup and restore of Hyper-V, we can use On-Host Proxy, which runs on the Hyper-V server where the backed-up VM is located. It performs deduplication and compression, so less data is transferred over the network to storage. Data transfer occurs over the LAN network, but as mentioned, it can also use iSCSI SAN (which is my case).

Backup on a 10 Gbps network achieves speeds up to 550 MBps (in one case even 720 MBps, with significant data reduction, so less than half is transferred over the network). Restore of 34.5 GB took 1:52, with an average speed of 315 MBps.