DBDeployer All articles
Case Studies

Stop Winging It: The Database Deployment Checklist That Actually Holds Up Under Pressure

DBDeployer
Stop Winging It: The Database Deployment Checklist That Actually Holds Up Under Pressure

Let's be honest about something: your team probably has a database deployment checklist somewhere. Maybe it lives in Confluence. Maybe it's a pinned Slack message from 2021. Maybe it's a Google Doc that three people have edit access to and nobody's touched in eight months.

And when a deployment is actually happening — especially a stressful one, with stakeholders watching and a release window closing — that checklist is the last thing anyone looks at.

The problem usually isn't laziness. It's that most database deployment checklists are written to cover every possible scenario, which means they end up being either so generic they're useless or so long they're paralyzing. "Verify backup exists" is not an actionable checklist item when you're ten minutes from deploying a multi-table migration to a production database with 200 million rows.

What follows is a different approach: a tighter, more opinionated framework built around the checks that actually prevent failures, organized by when they need to happen, and designed to be adapted to your stack without becoming a bureaucratic nightmare.

The Myth of the "One-Size" Checklist

Before getting into the specifics, it's worth calling out the assumption that kills most deployment checklists before they're ever useful: the idea that a single checklist can serve every database change equally well.

Adding a nullable column to a small reference table is not the same operation as dropping a column from a high-traffic transactional table. Both are schema changes. The risk profile, the rollback complexity, and the validations you need are completely different. A checklist that treats them the same is going to be either overkill for the first or dangerously thin for the second.

A practical checklist framework has tiers. Small, low-risk changes move quickly with lightweight validation. Larger, high-impact changes get more gates. The criteria for what falls into which tier should be explicit and agreed upon by the team in advance — not decided by whoever happens to be on call.

Pre-Deployment: The Checks That Actually Matter

1. Confirm the migration is idempotent or has a clean guard. Can this migration run twice without breaking things? If your deployment pipeline has any chance of retrying on failure, a non-idempotent migration is a trap. At minimum, every migration should check for the existence of the object it's modifying before acting on it.

2. Verify the migration against a production-sized dataset. This one gets skipped constantly, and it's one of the most expensive skips you can make. An ALTER TABLE that adds an index runs in milliseconds on your dev database with 10,000 rows. On a production table with 50 million rows, that same operation can lock the table for minutes. Test with production-scale data, or at minimum, model the expected execution time based on row counts and table structure.

3. Check that your current migration version matches expectations. Before you deploy, confirm that the target environment is at the exact migration version your deployment expects. If there's a mismatch — even one version off — stop and investigate. This is the schema drift check that catches problems before they become incidents.

4. Review for destructive operations with no recovery path. DROP TABLE, DROP COLUMN, and TRUNCATE are not inherently evil, but they require explicit sign-off and a verified backup taken within the last deployment window. "The nightly backup runs at 3 a.m." is not a recovery path if you're deploying at 4 p.m. and the operation goes wrong.

5. Validate that application code and schema changes are deployed in the right order. This one is a logic problem that bites teams constantly. If your application code references a new column, that column has to exist before the code goes live. If you're removing a column, the code that references it has to be removed first. Map out the dependency order before deployment day, not during it.

6. Confirm rollback SQL is written, tested, and accessible. Not "we can figure it out if we need to." Actual, tested rollback SQL that someone has run against a copy of the database and confirmed works. This should be a hard gate for any migration that involves destructive changes or structural modifications to high-traffic tables.

Post-Deployment: Don't Walk Away Early

The deployment running to completion is not the same thing as the deployment succeeding. Post-deployment validation is where a lot of teams drop the ball because everyone's relieved the migration finished and eager to move on.

1. Verify row counts and data integrity on affected tables. After a migration, spot-check the tables that were touched. If you migrated data, confirm the counts look right. If you added constraints, verify they didn't silently reject data that should have been retained. A quick sanity query goes a long way here.

2. Run your smoke tests against the actual production environment. Not staging. Production. The specific user flows that touch the tables you just modified should be exercised immediately after deployment. Automated smoke tests that run as part of the deployment pipeline are ideal; a manual walkthrough of critical paths is the minimum.

3. Check application error rates for the first 10 minutes. Set a timer. Watch your error tracking tool — Datadog, Sentry, New Relic, whatever you use — for the first 10 minutes post-deployment. A spike in database-related errors in that window is your signal to consider rollback before the blast radius grows.

4. Confirm query performance on affected tables hasn't degraded. If your deployment added, removed, or modified indexes, check your slow query log or query performance dashboard immediately after. An index change that seemed safe in testing can have unexpected effects on the production query planner. Catching a performance regression in the first few minutes is dramatically cheaper than catching it in a 3 a.m. alert.

Real Rollback: What It Takes to Actually Use One

Here's the uncomfortable truth about rollback plans: most teams have them in theory and can't actually execute them under pressure. Either the rollback SQL wasn't written in advance, or it was written but never tested, or it was tested but nobody's sure if it accounts for data that was written after the migration ran.

A rollback plan is only real if it answers three questions:

For destructive migrations — anything that drops or modifies existing data — the honest answer to question three is often "we lose some data." That's sometimes acceptable. It needs to be a conscious decision made in advance, not a surprise at 11 p.m.

Making This Stick Without Adding Bureaucracy

The goal isn't a longer process. It's a more deliberate one. The checks above shouldn't add hours to a deployment — most of them, if you've done the pre-work, take minutes. The ones that do take time (production-scale testing, writing and testing rollback SQL) are front-loaded into the planning phase, where they belong.

Build the tier criteria into your PR template. Make the pre-deployment checks a required step in your deployment pipeline, not a voluntary item on a doc nobody reads. And treat every skipped check as a deliberate risk decision that someone signs off on — not a default.

The deployments that wreck weekends are almost never unforeseeable. They're the ones where the checklist existed but the team convinced themselves it didn't apply this time. It applies. Run the checks.

All Articles

Related Articles

Your Database Pipeline Is Bleeding Time: Here's Where to Find the Leaks

Your Database Pipeline Is Bleeding Time: Here's Where to Find the Leaks

When the Database Goes Dark: 5 Deployment Failures That Wrecked Production (And What Actually Caused Them)

When the Database Goes Dark: 5 Deployment Failures That Wrecked Production (And What Actually Caused Them)

The Silent Killer of Database Deployments: How Schema Drift Sneaks Into Production

The Silent Killer of Database Deployments: How Schema Drift Sneaks Into Production