Backup & recovery

Why the backup window stops fitting overnight

Backup window overruns usually trace to one growing phase, not slow storage. How to find it, and four ways to make the night fit.

5 min read
Dark data center aisle with server racks and glowing cyan light paths tracing data movement between them

When the backup window stops fitting overnight, the cause is usually one phase of the job that has grown, most often the source read, a change tracking map that was reset, or snapshot commits, not a repository that became slow. Total job duration hides which phase it was, so the first fix is measuring phases rather than buying faster storage.

The pattern is familiar. A job that used to finish by 4 a.m. now finishes at 6:30, and nobody changed anything. A month later it runs into business hours and the virtualization team is asked why the hosts are busy at 9 a.m. A backup window rarely breaks in a single night. It erodes a few minutes at a time until one added workload or one longer chain pushes the finish past the point anyone can ignore.

Six phases inside a backup job

Nine reported hours tell nobody which were work and which were waiting. A job is a sequence of distinct operations, each of which stops scaling for its own reason.

  1. Source read. A proxy reads blocks from production storage, competing with whatever else runs then. A datastore that is comfortable by day becomes the constraint at 2 a.m. if a database maintenance plan, a storage rebalance or a replication task was scheduled into the same hours.
  2. Change block tracking. Incrementals depend on the hypervisor or agent reporting which blocks changed. When that tracking is reset, the next job has no map and reads everything. The job type still says incremental while the duration looks like a full. This is the most common cause of a window that blows out one night and recovers the next.
  3. Snapshot commit. Creating a snapshot is fast. Committing it is not, because writes that accumulated during the backup are merged into the base disk, on the datastore users hit.
  4. Target write. The phase everyone assumes is the problem.
  5. Catalog and metadata. Significant on jobs with very many files.
  6. Synthetic full construction. The target assembles a new full from restore points it already holds, as a synthetic full backup.

Why a faster repository rarely rescues the backup window

A faster repository helps one phase. If target write is thirty percent of the night, halving it saves fifteen percent of the window: real, but often smaller than the variance between two ordinary Tuesdays.

The upgrade also underdelivers because backup software seldom drives a target at its ceiling. Task counts, proxy numbers, gateway placement and stream settings cap throughput independently of the media, so a capable repository idles between tasks. Concurrency settings cost nothing to change and deserve a look before hardware does. And when the read side is the ceiling, no target shortens the night.

Reading phase durations instead of the total

Most backup platforms already record per-task statistics: duration, processing rate, data read against data transferred, and a bottleneck attribution across source, proxy, network and target. In Veeam these are the performance bottleneck statistics in the job session detail, one click from the green summary everyone looks at. Exporting them weekly for the three or four jobs that carry the business turns "why was it slow" into "which phase grew", which has an answer.

PhaseWhat a long duration usually meansWhat to check
Source readProduction storage is the constraint, or something competes for itOther activity in the same hours, proxy placement, datastore queues
Change block trackingTracking was reset, so an incremental read everythingData read against data transferred, recent migrations, host events
Snapshot commitThe snapshot lived too long and the delta grew largePer machine duration, snapshot age at removal, datastore space
Target writeA genuine target limit, or too few tasks to reach itTask slot counts, gateway placement, network path, bottleneck figures
Catalog and metadataFile indexing or restore point bookkeeping is heavy hereFile counts per machine, indexing settings, database location
Synthetic fullThe target is assembling a full on its own scheduleWhich days run synthetic operations, chain type, target read speed

One signature is unambiguous. When the data read approaches the full size of the protected workload on a job that should be incremental, change tracking has been reset. Broadcom notes that a power failure or hard shutdown can reset changed block tracking, after which an incremental can back up the whole disk.

Four ways to make the night fit

Once the phases are measured and the night still does not fit, four moves remain, and each trades something different.

  • Stagger. Jobs that all start at 22:00 contend for the same proxies, datastores and target. Spreading start times so heavy jobs do not overlap often recovers an hour with no purchase.
  • Split. Smaller jobs improve parallelism and shorten each snapshot's life, which cuts commit cost. They also multiply the chains to manage and the things that can fail on their own.
  • Change the chain type. Forward incremental with periodic synthetic fulls moves effort onto the target and away from production. Forever forward incremental removes the periodic full backup but adds a merge after every run. Each gives a different restore profile, so the choice belongs with whoever owns the recovery time expectations.
  • Accept a longer window. Legitimate if the risk is named. A job finishing at 8 a.m. leaves the recovery point older by the length of the overrun, overlaps production load, and delays copies to a second site, so copy job lag becomes part of the same decision.

Writing the window down

A ten minute monthly routine covers most of this: pull session statistics for the jobs that matter, record total and per phase duration, data read, data transferred and the reported bottleneck, and keep them in one sheet. Trends show in three months and are obvious in six. Alongside it, record when each job is expected to finish, the time after which a late finish becomes a problem, and who is told when it passes. If the team accepted a longer window rather than split a job, record that too, beside the recovery point objectives it affects, so it is not rediscovered as a surprise.

Try ARTESCA free

Immutable object storage that scales from 20TB to petabytes. Deploy a working cluster in under an hour.

Start a free test drive