Backup job failures usually arrive as a short message in the morning report: a timeout, an I/O error, a repository that was unavailable. The message says what the backup software observed, which is not always where the fault lies. A job can fail because of the source, the network, the backup server or the storage, and the storage is blamed more often than it is responsible. Tracing a failure to its cause is quicker with a fixed order of checks than with a guess.
This post gives a method for working out whether a failed job is a storage problem, and what to look at on the repository when it is.
Every backup follows the same route. Data is read from the source through a snapshot or agent, processed by a proxy or data mover, sent across the network and written to the repository. Metadata is updated on the backup server at the end. A failure at any stage stops the job, and the error text reflects the stage that noticed the problem, which may be downstream of the stage that caused it.
| Symptom in the job log | Likely area | First check |
|---|---|---|
| Snapshot creation failed | Source or hypervisor | Datastore free space, guest tools, existing snapshots |
| Job slow, then timed out | Any stage | Bottleneck statistics in the job report |
| Connection reset or refused | Network or repository service | Reachability of the repository endpoint from the proxy |
| Access denied or authentication failed | Credentials | Key or password validity, clock difference between systems |
| Insufficient space | Repository capacity | Free capacity, data waiting for locks to expire |
| Service unavailable or slow down responses | Repository under load | Concurrent tasks, repository health, background operations |
| Checksum or integrity error | Storage media or network | Disk and node health, network error counters |
The first question is scope. If a single job failed and others writing to the same repository succeeded during the same hour, the repository was working, and the cause is more likely on the source side or specific to that job. If every job to one repository failed while jobs to another succeeded, the repository or the path to it is the place to look. If everything failed, the backup server or a shared dependency such as DNS, the directory or the system clock comes first.
The second question is timing. Failures that cluster at the same time each night point to contention: too many jobs starting together, or a maintenance task on the repository that overlaps the backup window. Backup window overruns covers how overlapping schedules produce this pattern.
When the evidence points at storage, four things are worth examining. Capacity comes first, including capacity that appears used because expired data is still under an immutability lock. Health comes second: a failed drive or a node that is offline may leave the system running in a degraded state that slows writes. Load is third, since a rebuild, a scrub or a rebalance running during the backup window competes with incoming data. Credentials and certificates are fourth. An access key that was rotated, a certificate that expired overnight or a clock that has drifted by several minutes will cause every request to be rejected. Backup storage certificates explains the certificate case in detail.
Object storage reports problems through HTTP status codes, and the backup software usually records them. A 403 response means the request was understood and refused, which points to keys, permissions, a bucket policy or an attempt to change a locked object. A 404 means an object or bucket the software expected is not there. A 503, sometimes labelled slow down, means the store is asking the client to reduce its request rate. Timeouts with no status code indicate that the request never completed, which is more often a network or load issue than a logic error.
A 503 during peak load is a signal to reduce concurrency, not to retry harder. Tuning job concurrency describes how to match the number of parallel tasks to what the repository can accept.
The hardest failures are the ones that clear on retry. A job that fails one night in ten and succeeds when rerun tends to be ignored. Intermittent failures are worth logging in a simple table with date, job, error and what else was running. After a few weeks a pattern usually appears: the same proxy, the same hour, the same weekly task on the repository. Without the log, each occurrence looks like bad luck.
Some errors are the storage doing its job. An attempt to delete or overwrite a restore point that is still under Object Lock is refused, and the backup software may report that as a failure. This typically happens when retention on the job is shorter than the immutability period on the repository, so the software tries to remove data that the lock still protects. The fix is to align the two settings. How immutability and retention settings interact walks through the relationship.
ARTESCA is object storage software used as a backup target, deployed on servers the customer runs. It reports capacity, health and alerts through its management interface, which gives the storage side of an investigation a single place to check. It can span several servers, so a failed drive or server reduces redundancy without taking the repository offline.
Immutability is provided by S3 Object Lock on each object, and in compliance mode no account, including ARTESCA administrators, can remove a lock before it expires. ARTESCA grows within the same deployment from roughly 50 TB to 8.5 PB and is used as a target for Veeam and other backup software.
A short written sequence saves time on every incident: scope, timing, the job's own bottleneck statistics, then repository capacity, health, load and credentials. Following the same order each time means the morning after a failure starts with evidence. It also produces a record that shows, over months, where the estate's real weak points are.