Veeam Backup & Replication - Managed Hardened Repository

Note: The description in this article is based on Veeam Backup & Replication 12.2, licensed using Veeam Universal License (VUL), which is equivalent to Enterprise Plus.

In the first part Veeam Backup & Replication - Immutable Repositories and Secure Backups we described the principle of immutable backups and the options we can use in Veeam Backup & Replication. Today, we'll focus practically on Veeam Hardened Repository (specifically Managed Hardened Repository), whose principle was also described in the previous part.

Veeam Hardened Repository

• Hardened Repository

As a Hardened Repository, we can add a Linux server that we should try to secure as much as possible (Hardening). It is recommended (I'm not sure if it shouldn't be a crucial condition) to use the XFS file system on the server. This configuration enables protection of backed-up data using Immutable Backups.

System Requirements

Veeam lists various requirements for Hardened Repository, but some are rather recommendations. If we want to just test this storage, we don't need to follow everything. However, if we want to use it in production for secure data storage, it's necessary to secure the server very well. See the reflection on Immutability reliability at the end of the first part.

It is recommended to use a physical server with local disks (DAS) or remotely attached block storage (SAN, i.e., using iSCSI or Fibre Channel Protocol). You cannot use an attached NFS or SMB/CIFS volume. For testing, we can also use a virtual machine (VM).

Note: It's quite interesting how Veeam uses the term Direct Attached Storage in many places (including GUI). Yet often it can be a server that uses disk space on SAN block storage. The official documentation requirements for Hardened Repository correctly states remotely attached block storage. In various online guides, you'll even find descriptions of using NAS devices connected via iSCSI. This is nonsense, of course, but many devices support both NAS and SAN. I once described the terminology in the article Storage technologies and SAN networks or connecting servers to a disk array.

The operating system must be 64-bit Linux, it should be one of the supported distributions. Veeam lists distributions with advanced XFS integration. These include certain versions of AlmaLinux, Debian, Red Hat (RHEL), Rocky Linux, SUSE (SLES) and Ubuntu. The server requires Bash Shell and uses SSH for deployment. For storing backed-up data, we create a separate folder (partition) where we set defined permissions.

The hardware should be compatible with the given Linux distribution. Basic requirements for server and OS are listed in system requirements for backup repositories. Additional requirements in Requirements and Limitations. A special document describes sudo permissions Granular sudo Permissions Required for Hardened Repository.

Linux Server Installation and Configuration

Veeam documentation describes server preparation for Ubuntu and Red Hat.

• Preparing Ubuntu Linux Server as Hardened Repository
• Preparing Red Hat Enterprise Linux Server as Hardened Repository

There is an official video with instructions on YouTube.

• Veeam Data Platform: How to Set Up a Veeam Hardened Repository

More detailed description on the Veeam blog.

• Installing Ubuntu Linux for Veeam Hardened Repository

We can find many other guides on the internet, here are a few examples.

• Veeam Ransomware Protection with Red Hat Enterprise Linux as the Immutable Repository
• Veeam: Linux Hardened Repository using iSCSI
• Build an immutable backup repository for Veeam Backup & Replication
• Veeam Hardened Linux Repository - Part I
• Veeam Hardened Repository

Managed Hardened Repository

On 10/29/2024, the project status moved from Community Preview to experimentally supported.

• [RELEASE] Managed Hardened Repository ISO by Veeam
• Managed Hardened Repository Preview Now Available!
• [PREVIEW] Managed Hardened Repository ISO by Veeam

Veeam is working on preparing a bootable ISO (Veeam Hardened Repository ISO) that will install Hardened Repository in a simple way without requiring any Linux system knowledge.

As I was finishing the article, a new section appeared in the official documentation.

• Using Veeam Hardened Repository ISO

Main Features

• currently it's a Community Preview Experimentally Supported
• it's based on Rocky Linux
• simplified base OS installer, allows setting only a few parameters
• Pre-Hardened Base OS
• after installation, we only have access to the Hardened Repository Configurator Tool
• updates of the operating system and Backup Repository components are provided directly by Veeam (HTTPS communication to repository.veeam.com required, uses GPC keys that must be renewed)

System Requirements (in addition to previously mentioned)

• Veeam Backup & Replication 12.2 or newer
• physical server (Red Hat compatibility list) or virtual machine (officially not supported)
• at least 2 disks, each with a minimum size of 100 GB (otherwise installation cannot proceed)
• UEFI Secure Boot enabled
• wired network connection with minimum speed of 1 Gbps
• supports only local disks (DAS) with HW RAID controller, does not support SAN

Installing Server as Managed Hardened Repository

From Download: Veeam Hardened Repository ISO Preview Customer Portal or trial downloads (Additional Downloads - Extensions and Other - Veeam Hardened Repository ISO) we download the installation ISO along with PDF documentation and two videos showing installation and configuration (it's probably not here anymore). At the time of writing, it was 0.1.15.PREVIEW (does not support upgrading to a higher version), version with experimental support 0.1.17 (should go further to upgrade).

In the following description, we're creating a test repository as a VM on VMware (including examples of various parameters).

• connect to vSphere Client
• New Virtual Machine - create VM HardRepo, select Guest OS Rocky Linux, 2 vCPU, 4GB vRAM, 2x vDisk 100 GB Thin
• start the VM and VMware Remote Console (VMRC)
• menu Removable Devices - CD/DVD drive 1 - Connect to Disk Image File (iso) - attach installation iso
• boot and start Install Hardened Repository (deletes all data)

• if preparation (Python script) runs correctly, the GUI installation wizard starts with options to set keyboard, time, and network (Network & Host Name)
• start installation with Begin Installation button, it takes a few minutes, restart the server

• log in with default username vhradmin and password vhradmin
• we must enter a new complex password with various restrictions (including maximum 3 characters of the same class in a row)

• accept license terms
• we get to Veeam Hardened Repository Configurator, where we have only a few options available

Adding Hardened Repository

Now we need to add a new Backup Repository of type Hardened Repository. The installed Linux server must be added to the backup infrastructure as a managed server. We then create a Repository from a specific folder on the server. We can add the Linux server beforehand using New Linux Server, but we can also launch this wizard within New Backup Repository.

Enabling SSH

For deployment, we must first enable SSH on the Linux server. After completion, we'll disable it again (it turns off automatically after a certain time).

• Veeam Hardened Repository Configurator
• Start SSH
• password for user veeamsvc is displayed, we'll use this account to add the repository to Veeam Backup & Replication

New Hardened Repository

• Veeam Backup & Replication Console
• Backup Infrastructure - Backup Repositories - Add Repository
• select Direct Attached Storage - Linux (Hardened Repository)

• Name - unique name and description of the storage
• Server - select existing Repository server or add new Add New
  • Name - enter either full DNS name (FQDN) or server IP address
  • SSH Connection - we must use single-use credentials that aren't stored in Veeam configuration, use the previously displayed veeamsvc account

• • Veeam will try to connect to the server, you may need to approve server trust (SSH fingerprints), installed components are detected
  • clicking the Apply button will start installing components and configuring the server

• Server - back in the wizard we have the newly added server selected, click Populate and select folder (mounted second volume/disk, typically /mnt/veeam-repository01)
• Repository - Populate button loads disk space, check Use fast cloning on XFS volumes, enter number of days for immutability period, we can modify other settings including special parameters under Advanced

• Mount Server - select mount server and its parameters for file and application item recovery
• Review - click Apply and installation and configuration of all required components will proceed

Disabling SSH

• Veeam Hardened Repository Configurator
• Stop SSH

Backup Job

We create a backup job in the standard way, for example according to Veeam Backup & Replication - Backup Job. As Storage - Backup repository we select our Hardened Repository. The immutability of backups will be at least as long as the Immutability period specified on the repository.

Attempting to Delete Files

When we try to delete a backup in Veeam Backup & Replication, the deletion won't occur and we'll receive information that Immutability is set until a certain date. Veeam also checks the .veeam.N.lock file and won't allow deletion even if the Immutability attribute was removed.

If we access the files directly, we also won't succeed in deleting the backup files. The exception is the VBM file, which isn't protected and can be deleted.

Hardened Repository Server

Veeam services on the server

• Veeam Data Mover - Transport Service (veeamtransport)
• Veeam Immutability Service (veeamimmureposvc)
• Veeam Installer Service for Linux (veeamdeploymentsvc)

Note: We described more in the last part.

Users on the server

• vhradmin - for server management
• veeamsvc - uses Veeam Backup & Replication to deploy and manage the Hardened Repository, a non-root account with selected root-equivalent permissions

Access to the server

We can use an account vhradmin when logging in via the console, but we only get to Veeam Hardened Repository Configurator.

When we enable SSH, we can log in with an account veeamsvc. We can then use various Linux commands, but only with limited rights. Probably according to Granular sudo Permissions Required for Hardened Repository.

cd /mnt/veeam-repository01/backups/Test-Immutability
lsattr -a
getfattr * -n user.immutable.until
cat .veeam.N.lock

Commands won't work for us

rm *.vbk
chattr -i *.vbk
sudo chattr -i *.vbk