Why Backups Fail

Why Backups Fail

Backups fail because most systems measure the wrong event: they prove that data was copied, not that the service can be rebuilt from that copy. A backup is only useful if it can be restored into a working environment under the constraints you will face during an outage.

The mechanism is simple. A backup job may read files, write them to storage, checksum them, and report success. That process can still miss the pieces that make an application run. File permissions, ownership, mount points, application users, SELinux or AppArmor labels, environment variables, secrets, and database state can all be absent or wrong. A live database may have committed transactions that a plain file copy cannot capture unless the database is placed in backup mode or dumped with a transaction-consistent snapshot. Encryption can also turn a readable archive into an unusable one if the key, certificate, or recovery passphrase is not stored separately and tested.

A worked example shows the gap. Suppose a web application stores 20 GB of uploaded files and a 5 GB PostgreSQL database. A nightly job copies the files to object storage at 02:00 and reports 25 GB transferred. At 02:10, users submit 300 orders, and the database writes them. At 03:00, the disk fails. The file restore succeeds, but the database restore fails because the copied files are not a consistent database snapshot. If the team had run a scheduled restore into a separate environment, they would have discovered the problem while the original system was still healthy. The test would have revealed missing permissions, the wrong database user, an absent encryption key, or a version mismatch before customers noticed.

The common misunderstanding is that “backup success” means “recovery success.” In practice, success is a property of the whole recovery path. The backup media, the restore tool, the target operating system, the application version, the network path, the credentials, and the people performing the recovery all matter. A backup that has never been restored is an assumption about all of those components, not evidence.

There are also limits. Full restore testing is not equally practical for every system. A small stateless application can often be rebuilt quickly from configuration and object storage. A large data warehouse, an air-gapped archive, or a service with petabytes of data may need sampled restores, metadata checks, application-level consistency tests, or staged recovery drills instead of a complete boot-and-run test. The idea does not apply when the data is truly disposable, when the system can be rebuilt from authoritative upstream sources faster than it can be restored, or when compliance requires immutability and access controls that make routine live testing unsafe.

The rule is to schedule proof. Decide which services matter, define what “working” means, and run a restore that meets that definition. The backup job is only the first step.