The backup nobody checks until it's too late

Every team I've worked with had the same setup. Automated database backups running on a schedule. Files going to S3 or some NAS. Everyone assumes it works because the script hasn't thrown errors.

Then one day you actually need to restore something.

That's when you find out the dumps have been empty for two months because someone changed the DB password and forgot to update the backup script. Or the S3 bucket hit its lifecycle policy and deleted everything older than 30 days.

I started running a monthly restore test on a throwaway instance. Takes maybe 15 minutes to set up, and it's the only way to know your backups are real. Not "the script ran successfully" real — actually "I can boot a working copy of the database from this file" real.

It's boring work. Nobody wants to do it. But the alternative is finding out your backups are useless at the worst possible moment.

If you schedule it, treat it like any other job — monitor it, alert on failure, and don't skip it because "it worked last month."