Backup & recovery

Hardened repository and S3 copy: How they work together

A hardened repository and S3 copy protect against different failures. How the two layers combine, and what to set for the second copy.

5 min read
A band of cyan light running down the aisle between two rows of dark server cabinets

A hardened repository and S3 copy are often presented as alternatives, as if a backup team has to pick one immutability model and live with it. In practice many estates run both. The hardened Linux repository takes the first backup, close to production, and a copy of each restore point goes to S3-compatible object storage with Object Lock. The two layers protect against different failures, and the design works well when each is given the job it suits.

This post does not compare the two models. Hardened repository versus S3 Object Lock does that. The subject here is how the two behave when they are combined, and what has to be set correctly for the second layer to be worth having.

What each layer is responsible for

The hardened repository is usually the performance copy. It sits on local disks, it is fast to write to and fast to restore from, and it holds the most recent restore points. Its immutability comes from file system attributes that the backup software sets for a defined period.

The S3 copy is the resilience copy. It lives on separate hardware, ideally in a separate room or site, and each object carries its own retention lock enforced by the storage layer. Its job is to be present when the first repository is not, whether the cause is a failed controller, a lost site or an attacker who reached the server's console.

How the two layers differ

PropertyHardened repositoryS3 copy with Object Lock
Where immutability is enforcedFile system on one serverStorage layer, per object
Typical roleFirst copy, daily restoresSecond copy, site loss and cyber recovery
Failure domainOne server and its disksDepends on the object store's layout
Access pathBackup software's data moverS3 API with access keys
Retention usually heldDays to a few weeksWeeks to years
GrowthAdd disks or a second serverAdd capacity within the same store

Getting the copy there

There are two common ways to move data from the first layer to the second. A backup copy job reads restore points from the hardened repository and writes them to the object store on its own schedule, with its own retention. A scale-out repository tiers data instead, copying each restore point to the capacity tier as soon as the backup job finishes.

The difference matters for the gap between layers. A copy job that runs nightly leaves up to a day of restore points on one layer only. Copying on creation narrows that gap to the time the transfer takes. Backup copy job lag explains how to measure the gap and what makes it grow. Whichever method is used, the number worth watching is the age of the newest restore point on the second layer, not whether the job reported success.

Aligning the two immutability periods

Each layer has its own immutability setting, and they do not need to match. A common arrangement keeps a short period on the hardened repository, enough to cover a weekend and an investigation, and a longer one on the object store. The longer period belongs on the layer that costs less per terabyte and has room to grow.

What needs checking is that the second layer's lock actually covers the retention the business expects. The lock applied to an object is a product of the job's retention, the immutability period on the repository and any block generation setting the software uses. How Veeam immutability and retention settings interact walks through that arithmetic.

Keeping the layers independent

The value of a second layer disappears if both share a weakness. Three points are worth confirming. The S3 access keys used for the copy should not be stored on the hardened repository or reachable with the same administrator account. The object store should sit on a different network segment from the first repository, with only the S3 endpoint exposed to the backup servers. And the two layers should not rely on the same directory service for administrative logins.

Independence also applies to hardware. A second copy on a volume in the same chassis, or on a virtual machine hosted by the cluster being protected, shares a failure domain with what it is meant to back up.

Restoring from each layer

Day to day, restores come from the hardened repository because it is closer and holds the newest points. The second layer is used when the first is unavailable or untrusted. That restore path gets far less practice, so it should be tested deliberately: a restore run directly from the object store, with the first repository treated as offline. The test confirms that the keys work, that the network path carries the load and that the restore speed is known in advance.

Where ARTESCA fits

ARTESCA is object storage software used as a backup target, deployed on servers the customer runs. In this design it is the second layer: an S3 endpoint with Object Lock, where in compliance mode no account, including ARTESCA administrators, can remove a lock before it expires. It is used with Veeam and other backup software as a copy target or capacity tier.

ARTESCA can span several servers, so the second copy does not depend on one machine, and capacity grows within the same deployment from roughly 50 TB to 8.5 PB. Some teams also write backups to it directly and keep the hardened repository only for the workloads that need the fastest local restores.

A short check for a two-layer design

For a team that already runs both layers, four questions show whether the design is doing its job. How old is the newest restore point on the object store right now? What lock period is actually set on an object written last night? Where are the S3 keys stored, and who can read them? And when was a restore last run from the second layer alone? If each has a clear answer, the two layers are working together. If any is unknown, that is the place to start.

Try ARTESCA free

Immutable object storage that scales from 20TB to petabytes. Deploy a working cluster in under an hour.

Start a free test drive