Note: The description in the article is based on Veeam Backup & Replication 12.2, licensed using Veeam Universal License (VUL), which is similar to Enterprise Plus.
In this introductory part, we'll look at various ways to achieve Immutability with Veeam Backup & Replication. Mainly from a theoretical or general perspective. In the following parts, we will focus on practical tests of various options (they also contain a more detailed description of some principles).
Backup Protection and Related Terms
Backups serve to protect important data against accidental deletion or corruption, failures, attacks, malware (Ransomware), or other malicious activities. Source data is protected by keeping copies in backups that can be restored.
Backups contain the company's core data. Today's attackers therefore focus on backups. They don't need to search for sensitive data across the entire network but can find it in one place. It's therefore crucial to have secured access to backups, encrypted backup content (to prevent attackers from obtaining data), and prevent the possibility of data modification (if servers get encrypted, to have reliable backup available).
Note: It's also important to be certain that the backup is reliable and data can be restored. Therefore, we should perform regular recovery tests.
The 3-2-1-1-0 Rule
The 3-2-1 rule has been used for backups for 20 years. It means keeping at least 3 copies of data (production, backup, backup copy), using 2 different backup media (copies on different devices), and keeping 1 backup offsite.
Veeam recently extended this rule to 3-2-1-1-0. The added 1 and 0 mean keeping 1 copy of data Offline (powered off), air-gapped (physically isolated/disconnected from the network) or immutable, and when checking recovery using SureBackup achieve 0 errors.
Hardening
The term Hardening in computer security means securing a system by reducing the attack surface. This includes reducing system functions (unnecessary services), removing unnecessary software and users, changing default passwords, access control (permissions and strong authentication), and securing network communication.
Veeam Backup Repository vs. Storage System
Since we're dealing with data storage, we should specify two basic terms.
Storage System means a storage system (solution for storing data) or storage, a specific example is a storage array (though this term is used less frequently). It's a device or software that serves for reliable and secure data storage and management. It provides storage space for data, has no knowledge related to backups. It provides a scalable and organized way to store data so it can be easily accessed, searched, and processed.
Backup Repository means backup storage. It's a logical entity within Veeam Backup & Replication that represents the target location for backups. It can be configured on various types of physical or cloud storage. It adds a backup-specific management and organization layer above the storage space itself. Contains metadata and information about backup structure. Can have specific settings like compression, deduplication, or encryption.
Various storage types can serve as a Backup Repository.
Direct Attached Storage(DAS) - virtual or physical servers using local disks (DAS) or SAN - Windows server, Linux server, Hardened RepositoryNetwork Attached Storage(NAS) - shared network folders (file server or NAS) - SMB/CIFS share, NFS shareObject Storage- cloud service or local object storageDeduplicating Storage Appliance- supported physical device
Object Storage
Note: More information can be found in the new article What is Object Storage?.
Object storage is a system for storing and managing data that treats information as objects instead of traditional files or blocks. Each object contains a unique ID and metadata describing its content and properties. It can store different types of data efficiently and securely. It's suitable for large volumes of unstructured data. It can have various implementations and interfaces (API). May not be compatible with other systems.
Object storages are the main type of storage that supports Immutability. Their subset is S3 compatible storage.
Amazon S3 (Simple Storage Service)
Probably the most widespread object storage is the cloud service Amazon Simple Storage Service (Amazon S3) from Amazon Web Services (AWS). It has good scalability, data availability, durability, security, and performance. Provides version control, lifecycle, global availability, and various APIs (including REST API).
S3 Compatible Storage
Due to the widespread adoption of Amazon S3, its API has become the de facto standard for Object Storage API. Therefore, many storage and software manufacturers implement API compatible with Amazon S3. These storages are referred to as S3 compatible (S3 Compatible Storage) or S3 Object Storage. This brings a unified interface for storage operations and compatibility between different systems. Existing S3 tools and libraries can be used, which is an extensive ecosystem.
Veeam Backup & Replication supports selected types of object storage.
- the largest group uses S3 API
- general S3 Compatible
- specific Amazon S3, Google Cloud Storage (doesn't support Immutability), IBM Cloud Object Storage, Wasabi Cloud Storage
- Microsoft Azure Storage
- Veeam Data Cloud Vault
Immutability
Immutability is a property (data state) where stored data cannot be modified or deleted after creation. Reading is not restricted. With Immutable Backups, we ensure security, integrity, and compliance of backed-up data. A similar term that has been used for a very long time is WORM (Write Once, Read Many). Description at Veeam What is WORM (Write Once, Read Many)?
For backups, we use temporary immutability and set an immutability period. That is, the time during which data cannot be modified. After this period expires, data can be deleted (often automatically with the end of retention) and is no longer protected.
For backup jobs, we set a Retention Policy that determines how long backups are kept. It's recommended to set the Immutability Period shorter than or equal to the Retention Policy.
Setting Immutability affects (prevents) various Veeam functions and can complicate storage management. Depending on the storage type, but generally it's not possible to modify (consolidate) protected files, and thus use features like Reverse Incremental. We cannot move backups between storages, including in Scale-Out Backup Repositories moves between extents (Rebalancing), to Capacity Tier, etc.
Immutable Storage
Technologically, we can implement immutability either at the level of storage array (storage) or cloud storage (which is basically the same) or through software using a special file system. Storage arrays often also allow creating immutable volume snapshots (read-only snapshots).
Veeam supports the following backup repositories with immutability function:
Veeam Hardened RepositoryObject Storage Repository- primarily S3 compatible with S3 Object Lock function, uses Veeam deduplication and compression- local storage from manufacturers ObjectFirst, Scality, Pure Storage, NetApp, HPE and others
- cloud services Amazon (S3 bucket), Microsoft (Azure Container), Wasabi and other providers
Deduplicating Storage Appliances- storage with own deduplication and compression, HPE StoreOnce uses Dual Authorization, others (Dell Data Domain, ExaGrid, Fujitsu CS800, Quantum DXi, Infinidat InfiniGuard) use time locks or Secure Snapshot
Veeam supports immutability at all levels of Performance Tier, Capacity Tier, and Archive Tier. They call it a start-to-finish immutability approach. They have a nice illustration for this (I just think a lock is missing for IBM Object Storage).
Immutability and Backup Repository
In Veeam Backup & Replication, Immutability is set at the whole Repository level. When adding storage as a Repository to the backup infrastructure, for object storage or deduplication appliance we check Make recent backups immutable for (Hardened Repository has this function automatically enabled) and enter the number of days. For object storage, 10 or 30 days are added within Block Generation.
Immutability is managed by Veeam, so on the storage we only activate necessary functions but don't enable immutability settings (we must not enable default retention, Lifecycle Rules or Tiering). On S3 storage, we must activate Object Lock and versioning when creating a bucket.
Documentation states that for Object Storage we can set 7 to 90 days in the VBR Console (GUI). For longer periods, we can use PowerShell cmdlets Set-VBRAmazonS3CompatibleRepository and Set-VBRAzureBlobRepository. For Hardened Repository, we can set the entire supported interval of 7 to 9999 days of immutability.
Immutability and Backup Chain
Veeam Backup & Replication sets immutability on the entire Backup Chain with all its restore points. Which Backup Mode we can use depends on the type of Repository. According to this, the corresponding Backup Chain is created.
For S3 object storage, we can only use Forever Forward Incremental. For Hardened Repository, it can only be Forward Incremental (with active or synthetic Full Backup enabled, otherwise it would be Forever Forward Incremental). We can't use Reverse Incremental anywhere. When immutability is set on backup files, they cannot be merged or deleted before it expires.
Retention Policy sets the number of restore points or days we want to keep. After exceeding this, the oldest Restore Point is removed. In the case of Forward Incremental, we need the entire Backup Chain to restore data. Therefore, some Restore Points are kept longer. Only when the entire Backup Chain falls outside the Retention Policy, it is removed. The Immutability Period affects this by protecting the entire Backup Chain, and the period extends to all backup files in the active chain.
Immutability is set on files with backed up data (VBK, VIB, etc.). It is not set on the backup metadata file (VBM).
Immutability and Object Storage
Object storage doesn't work with files, but with objects. Veeam splits backup files into blocks of specific size (default 1 MB) and uploads them as objects (they are also compressed). Immutability is set on each object.
During incremental backup, only new or changed data is uploaded as new objects. For existing data, only the immutability date is updated. To save I/O operations, it adds several days to the immutability expiration date. This is called Block Generation.
Veeam Hardened Repository
- Hardened Repository
- Linux Hardened Repositories: Achievable Immutability for All
- Selecting Hardware and Setting Up Environment for Veeam Hardened Repository
- Installing Ubuntu Linux for Veeam Hardened Repository
- WORM Storage with Veeam Hardened Repository
Linux Hardened Repository is a special type of backup storage in Veeam Backup & Replication. It's a way to achieve Immutability without having to buy new storage (array) or use a cloud service. It supports a general Linux server, so you can choose hardware and Linux distribution according to your preferences.
This storage appeared in Veeam Backup & Replication 11 and was significantly improved in version 12. It uses Veeam deduplication and compression. The server cannot contain other Veeam roles (except for VMware Backup Proxy in Network mode). In version 11, it couldn't be used for enterprise application backups, transaction logs, and more. In version 12, we can use it to store almost all types of backups.
Hardware for Hardened Repository
For Hardened Repository, we can use a standard physical server with local disks or space on an existing storage array connected via SAN. Veeam recommends using internal disks because it eliminates the possibility of an attacker compromising the storage array. The server must be sufficiently powerful, redundant, and secure.
For security, it's recommended to disable remote management (IPMI - Intelligent Platform Management Interface), such as HPE iLO or Dell DRAC, and access via SSH. Correct time (reliable NTP) and its security are also important.
We can also use a virtual server (VM), although a physical server offers significantly higher security (see the reflection in the last chapter).
Operating System for Hardened Repository
We install a specific Linux distribution (Ubuntu, Red Hat, SUSE etc.) on the server, so basic knowledge of that OS is needed. Veeam's documentation provides instructions for installing and configuring Ubuntu Linux Server or Red Hat Enterprise Linux Server. The actual use of the storage is simple (same as for other Repositories) and is done from Veeam Backup & Replication.
It uses the XFS file system, which supports Block Cloning - reflink (Fast Clone) and Immutability. For Repository deployment, single use credentials are used, which are not stored on the Veeam server. Root privileges are not needed for operation (compromising the Veeam Server won't endanger the data). It's recommended to disable SSH, which is needed for deployment and updates of Veeam Data Mover.
How Immutability Works
Two Veeam services run on the server:
- Veeam Data Mover - Transport Service (
veeamtransport) - runs as a non-root user, communicates with the backup server, uses port 6162 - Veeam Immutability Service (
veeamimmureposvc) - runs with root privileges, but is local and has no network access, checks immutability attributes every 20 minutes, calculates immutability period for files and sets or removes the Immutable attribute, also performs time drift detection
Note: Deployment and updates are done using Veeam Installer Service for Linux (veeamdeploymentsvc).
To ensure immutability, XFS file system properties are used. The following is set on backup files:
- Immutability attribute (flag)
i, we can list it usinglsattr -a - extended attribute
xattrwith immutability time period, we can list it usinggetfattr * -n user.immutable.until
For each backup job (documentation states for each backup file), a file .veeam.N.lock is also created, which contains information about the immutability period of individual files. The retention information is thus stored twice and Veeam uses information from the .veeam.N.lock file.
Immutability Period can be 7 to 9999 days. The Immutability flag is set on the file when the current backup session is completed. If the Immutability Period is longer than Job retention, backups won't be deleted before Immutability expires. If GFS is used, for full backups with GFS, immutability is set as the longer of both periods (often according to GFS).
Managed Hardened Repository
Veeam is trying to make using Hardened Repository even simpler and is working on Managed Hardened Repository. It's a prepared ISO based on Rocky Linux, currently in Community Preview. It can be used even by someone without Linux OS knowledge. The system is optimized for Hardened Repository and offers limited functions. Updates are handled by Veeam.
Official announcement Managed Hardened Repository Preview Now Available! and forum post [PREVIEW] Managed Hardened Repository ISO by Veeam.
My Thoughts on Immutability Reliability
Immutability should protect data from modification or deletion. Today, we mainly try to protect against attackers, but this protection shouldn't allow even the system administrator to delete data.
First, of course, we must try to secure the Veeam Backup & Replication Server and the entire backup infrastructure. If an attacker compromises it, they have the ability to write (and read) everywhere Veeam has access. The next step is to ensure that an attacker cannot delete or encrypt data on the Repository.
Hardened Repository
When using Hardened Repository, the XFS file system allows setting an immutable attribute. Files with this setting cannot be deleted, modified, overwritten, moved, or renamed by anyone. However, a root user can remove this attribute and then manipulate the file. Therefore, it's important to carefully secure the root account or not use it.
There are other ways to delete protected data. If the Hardened Repository is a VM, the virtualization administrator can delete its disks. For volumes on a standard storage array, the storage administrator can do this. Local disks on the server can be deleted using management tools (from the manufacturer and others).
There's always a risk that an attacker gains administrative access and deletes data. These possibilities must be minimized. Another question is the vulnerabilities of various software.
Object Storage
The second category is object storage and various special appliances. If Immutability is set for some data, even the system administrator shouldn't be able to change this. For such storage, we have an administrator account, but it's limited and we don't have super administrator access. The question of potential bugs and vulnerabilities remains. Also, whether there's any possibility to perform a factory reset.
Cloud Storage Services
It's similar with cloud services. I've heard warnings that if we set Immutability in Azure using a locked policy with time retention, it's not possible to cancel (delete data) before time expires. I believe Microsoft must have the ability to delete or cancel data. The question is whether they'll allow us to terminate such a subscription. If, for example, a company went bankrupt and stopped paying, Microsoft probably wouldn't be interested in keeping their data. Additionally, I came across a mention that MS can only delete data when canceling a subscription due to non-payment.
There are no comments yet.