Backup & recovery

Backup site failure: What the second copy needs

Backup site failure tests whether the second copy works alone. The server, keys, catalog and capacity it needs to be usable.

5 min read
Two dark server cabinets standing side by side with a faint cyan glow between them

Backup site failure is the scenario a second copy exists for: the room or building that holds the primary backup repository is gone, along with the backup server and often the production systems beside it. Having a second copy somewhere else is the first requirement. Being able to restore from it with nothing from the first site is a separate one, and it is where many two-site designs turn out to be incomplete.

This post lists what the second copy needs so that it can be used alone, on the day the primary site cannot help.

A copy is not yet a recovery capability

Backup data at a second site is a set of files or objects in the backup software's format. To turn them back into running systems, something has to read that format, know which restore points exist, hold the keys to decrypt them and have somewhere to restore to. In normal operation all of that lives at the primary site. If it is lost with the site, the second copy is intact and unreadable until those pieces are rebuilt.

What the second copy depends on

DependencyWhere it usually livesWhat is needed at or beyond the second site
Backup server and its configurationPrimary siteA standby server, or a tested rebuild from a configuration backup
Catalog of restore pointsBackup server databaseAbility to re-import or rescan the repository
Encryption keys and passwordsBackup server, password managerA recovery copy stored away from both the server and the domain
Repository credentialsBackup server configurationAccess keys recorded securely off the primary site
Directory, DNS and time servicesPrimary siteLocal instances, or a plan to operate without them
Compute and storage to restore ontoPrimary siteReserved capacity, a contract for hardware or a cloud landing zone
Network access for usersPrimary siteRouting, remote access and addressing for restored systems

The backup server and its catalog

The backup server knows what has been backed up and where each restore point is stored. Most products can export their configuration on a schedule and can rebuild their catalog by scanning a repository. Both steps need to be proven at the second site. The configuration backup should be stored on the second copy's repository or another location that survives the primary site, and the time to install a server, import the configuration and rescan the repository should be measured. Trusting the backup console after an incident covers the related case where the server exists but cannot be relied on.

Keys and credentials

Encrypted backups cannot be read without the key or the password that protects it. If the only record is on the lost backup server or in a password manager that ran at the primary site, the second copy is unusable. The same applies to the access keys for the repository. A recovery copy of each should be held somewhere independent of both sites and of the production directory. Who holds the recovery copy of the backup encryption keys sets out the options.

Independence from the first site

The second site should not rely on services the first site provided. If its administrative logins go through a domain controller at the primary site, or its name resolution and time source are there, losing the primary site can lock administrators out of the surviving repository. Local accounts for the repository and backup server, and local DNS and time services, remove that dependency. Keeping backup systems off the production domain explains the reasoning in more depth.

Freshness and immutability

The second copy is only useful up to its newest restore point. If copies run nightly and the failure happens in the afternoon, most of a day's changes are missing. That gap should be a deliberate choice that the business has agreed to, and it should be monitored so that a stalled copy job is noticed within a day.

A site failure is not always a physical event. If the primary site is lost to ransomware, the attacker may have had access to the backup server and its stored credentials for the second repository. Immutability enforced by the second repository itself keeps those restore points intact whatever the backup server is instructed to do.

Somewhere to restore to

A repository holds data, not workloads. Restoring several hundred virtual machines requires hosts, storage, network and licenses. Options range from a standby cluster at the second site, to hardware held under a recovery contract, to restoring the most important systems into a cloud environment. Whichever is chosen, the capacity should be matched against a prioritized list of systems so that the first day's restores are known in advance.

Where ARTESCA fits

ARTESCA is object storage software used as a backup target, deployed on servers the customer runs. As the repository at a second site, it holds restore points behind an S3 endpoint with its own administrative accounts, independent of the primary site's backup server and directory. Immutability is provided by S3 Object Lock on each object, and in compliance mode no account, including ARTESCA administrators, can remove a lock before it expires.

ARTESCA can span several servers, so the second copy does not rest on a single machine, and it grows within the same deployment from roughly 50 TB to 8.5 PB. It is used as a target for Veeam and other backup software.

The test that proves it

The only reliable check is an exercise in which the primary site is treated as unreachable. Starting from the second site with the documented recovery materials, the team builds a backup server, connects to the repository, imports the restore points and restores a small set of representative systems. The first run usually finds a missing password or an undocumented dependency. That finding, made on a quiet day, is the point of the exercise.

Try ARTESCA free

Immutable object storage that scales from 20TB to petabytes. Deploy a working cluster in under an hour.

Start a free test drive