"We've Got This" — And Other Things Teams Say Right Before a Database Incident

There's a particular kind of confidence that shows up in pre-deploy conversations. It sounds like "we've done this before," "the runbook is solid," and "staging looked clean." It feels like competence. It often isn't.

False deployment confidence is one of the most underexamined failure modes in database operations. It's not about teams being reckless or inexperienced — in fact, it tends to show up more in experienced teams, because experience generates familiarity, and familiarity generates the assumption that you know what you're walking into. Sometimes you do. Sometimes production has been quietly diverging from your mental model for three months and nobody noticed.

Why Confidence Is Easy to Manufacture

Human beings are genuinely bad at distinguishing between "I feel prepared" and "I am prepared." This isn't a character flaw — it's cognitive architecture. We build mental models of complex systems, and those models feel accurate because they're the only models we have. When the last five deploys went fine, the brain updates its confidence level upward. The fact that those deploys were all additive migrations and this one is a column drop doesn't necessarily register as a material difference until it's too late.

Organizational dynamics make this worse. Teams that have built a reputation for smooth deployments face social pressure to maintain that reputation. Raising concerns before a deploy can feel like undermining the team's track record, or like you're the person slowing everyone down for no good reason. So concerns get swallowed. Runbooks get rubber-stamped. The pre-deploy checklist becomes a ritual rather than a genuine verification.

This is how teams end up confident and wrong at the same time.

The Runbook Problem

Runbooks are supposed to be the antidote to false confidence. Write down the procedure, verify the procedure, follow the procedure. In theory, a good runbook eliminates the need to rely on tacit knowledge or gut feel.

In practice, most runbooks have a shelf life that nobody is tracking.

A runbook written six months ago may reference a replica configuration that's since been updated, a connection string that's changed, or a rollback procedure that assumes a table structure that no longer exists. The team that wrote it is confident because they wrote it. The team that inherited it is confident because it exists. Nobody has validated whether it still reflects reality.

The other runbook failure mode is scope. Many runbooks cover the expected path in detail and the unexpected path barely at all. What happens if the migration runs twice as long as estimated? What happens if replication lag spikes during the migration window? What happens if the rollback itself fails? These scenarios get a sentence or two, or nothing, because writing them out requires confronting the possibility that things will go wrong — and teams that feel confident don't like sitting with that possibility.

Untested Failover Is Theater

One of the most reliable indicators of false confidence is a failover procedure that has never been executed under realistic conditions. Teams will document a rollback, review it, and nod at each other in the pre-deploy meeting — but if that rollback has never been run against a production-scale dataset, under time pressure, by someone who didn't write it, then what you have isn't a safety net. It's a prop.

The test that matters isn't "does this procedure make sense on paper?" It's "can the on-call engineer execute this correctly at 2 AM with three stakeholders asking for updates in Slack?" Those are radically different tests, and most teams only run the first one.

Game day exercises — structured, realistic simulations of failure scenarios — are the most direct way to close this gap. They're also chronically underfunded and deprioritized, because they're expensive to run and they produce results that are uncomfortable to look at. Nothing deflates a confident team faster than watching their rollback procedure fail in a simulation. Which is exactly why you should run them.

Frameworks for Stress-Testing Actual Readiness

So how do you tell the difference between genuine preparedness and the comfortable illusion of it? A few approaches that actually surface the gaps:

The stranger test. Hand your runbook to an engineer who wasn't involved in writing it and ask them to walk through it out loud, cold. Every place they pause, ask a clarifying question, or get stuck is a gap in the documentation. If they can't execute the rollback procedure without help, your rollback procedure isn't ready.

The failure injection exercise. Before a significant deploy, explicitly walk through three or four failure scenarios: the migration runs long, the primary goes down mid-migration, the rollback encounters a lock. For each one, name the specific person who makes the call, the specific action they take, and the specific tool they use. Vague answers are a red flag.

The time-pressure drill. Take a past incident and replay the response, but compress the timeline. How does the team perform when the window for decision-making is half as long? Confidence built on leisurely decision-making often evaporates under real time pressure.

The drift audit. Compare your current production schema against your documented expectations. Schema drift — the gradual divergence between what you think the database looks like and what it actually looks like — is one of the most common sources of deploy-day surprises. If you haven't audited recently, your runbook may be describing a database that no longer exists.

Confidence Is a Signal, Not a Verdict

None of this is an argument against confidence. Teams that are genuinely prepared should feel confident — that feeling is useful information. The problem is when confidence becomes a substitute for verification rather than a product of it.

The teams that deploy most reliably tend to have a healthy skepticism baked into their pre-deploy process. They ask hard questions not because they expect things to go wrong, but because they know that the questions are cheaper to answer before the deploy than after it.

"We've got this" is a great place to land. Just make sure you've actually done the work to get there.