Database Migrations Are a Coordination Problem, Not a Schema Problem

Ask an engineer to add a column to a database table and they’ll have the migration file written in five minutes. Ask them to ship that column to a production system serving live traffic, and the estimate quietly grows to a day, a sprint, or a change-freeze exception. The SQL was never the hard part. The hard part is that a schema and the code reading it are two separate deployables, released on two separate schedules, and for some window they have to both be true at once.

The window nobody schedules

A rolling deploy means old code and new code run side by side for minutes, sometimes longer. If a migration drops a column the old code still selects, or a new code path writes to a column the old code doesn’t know exists, that window is where the outage lives. Nobody puts it on the calendar because nothing about it looks like a discrete event — it’s a property of how the rollout happens to overlap with the schema change, and most teams only notice it exists the first time it breaks something.

The standard fix is the expand-contract pattern: add the new column without removing the old one, deploy code that writes to both, backfill, deploy code that reads only the new one, and only then drop the old column. That’s four separate deploys to do what looks like one ALTER TABLE. Teams that skip straight from step one to step four aren’t being reckless — they usually just don’t have a shared mental model of the deploy as a multi-stage process instead of a single push.

Migrations don’t get reviewed the way code does

A pull request with a risky migration gets the same review as a pull request that reformats a CSS file: one approval, a green check, merge. But a migration runs once, against real data, often before anyone can look at the query plan it produces. Locking behavior on a large table, whether an index build blocks writes, how long a backfill actually takes against production-sized data — none of that shows up in a diff. It shows up during the migration, live.

Teams that treat migrations as ordinary code changes eventually get burned by one that locks a hot table for longer than expected. Teams that treat them as a distinct category — requiring a dry run against a data volume close to production, an explicit rollback plan, and a named owner watching it execute — catch that class of problem before it reaches anyone’s dashboard.

Rollback is not “run it backward”

Application code has a trivial rollback: redeploy the previous version. Migrations don’t get that for free. A column drop can’t be undone without the data that was in it. A backfill that runs for twenty minutes doesn’t reverse in twenty seconds. “We can always roll back” is true for the code and false for the data, and conflating the two is how teams end up staring at a broken migration with no real way out except forward.

The fix isn’t clever tooling, it’s discipline: write migrations that are additive and reversible in the specific step being deployed, and never combine “add capability” with “remove old capability” in the same release. Anything destructive — dropping a column, dropping a table, changing a type in place — should be the very last step, done only after the code depending on the old shape has been out of rotation for a full deploy cycle with no incidents.

Coordination scales worse than the schema does

A single service with one database and one deploy pipeline can get away with a casual approach to migrations for a long time. The trouble starts once there are multiple services reading the same tables, or the same service is deployed in multiple regions on independent schedules. Now “the code and schema are in sync” isn’t a fact about one deploy, it’s a fact about the whole fleet, and any lagging instance can break the assumption the migration relied on.

This is why teams that scale past a handful of services usually end up formalizing what used to be tribal knowledge: a required checklist for schema changes, a bot that flags migrations touching large tables, a rule that destructive changes need a second reviewer from outside the team. None of that is about the SQL being correct. It’s about making sure the coordination step — the part with no compiler to catch mistakes — has a process standing in for the review that code changes get automatically.

The database doesn’t care how careful the migration author was with syntax. It cares whether the deploy that ran before it, and the one running after it, agree on what the table looks like in between.