What is deduplication?
Deduplication is a data reduction technique that stores each repeated block of data only once and replaces every other occurrence with a pointer to that stored copy. In backup it shrinks the space taken by repeated full backups and similar machines, and the ratio achieved depends mostly on the data being protected.
How does deduplication find repeated data?
Backup software splits the data stream into chunks and gives each chunk a short fingerprint calculated from its content. An index records every fingerprint already stored. A new fingerprint means the chunk is written and added to the index. A known fingerprint means only a pointer to the existing chunk is written. Chunk boundaries can be fixed or variable in size. Variable boundaries find more repeats when data is inserted into the middle of a file, because the insertion does not shift every chunk after it.
What sets the deduplication ratio?
The ratio compares the amount of data protected with the space it occupies. Four factors set it:
- Number of retained copies. Weekly full backups kept for months hold mostly the same data, so each extra copy adds little.
- Similarity across systems. Virtual machines built from the same image share most of their blocks.
- Change rate. Databases and logs that change constantly produce new chunks on every run.
- Data already reduced. Compressed media, encrypted files and previously deduplicated data contain almost no repeats.
A datasheet ratio such as 20 to 1 is measured on a data set with a great deal of repetition. It is accurate for that test and says little about another environment. The figure that applies to an organization is the one measured by running the product on a sample of its own data.
What does deduplication cost at restore time?
Reading deduplicated data back means following pointers to chunks scattered across the repository and reassembling the original files, a step called rehydration. Chunks written in order on the first backup end up spread across many later ones, so reads become more random. As a result, restore throughput is often lower than backup throughput, and the gap widens as the repository ages. A restore time measured on a small, fresh test volume overstates what a large recovery will achieve.
Why do encryption and locks reduce the savings?
Encrypted data has no repeating patterns, so data encrypted before it is reduced barely shrinks. Backup software therefore reduces data first and encrypts it afterward. Retention locks have a second effect. A chunk shared by many restore points cannot be freed until the last restore point that uses it expires, so on locked storage the space actually used stays above what the ratio suggests.
Frequently asked questions
Is deduplication the same as compression?
No. Deduplication removes repeated chunks across files and backups. Compression shrinks each chunk's content by encoding it more compactly. Backup products often use both.
Does deduplication happen at the source or the target?
Either. Source-side deduplication checks fingerprints before sending, so repeated data never crosses the network. Target-side deduplication reduces what arrives, at the storage.
Can deduplicated backups be verified?
Yes, but integrity matters more because one chunk may support many restore points. A damaged shared chunk affects all of them, which is why health checks and test restores matter.