Deployment, observability, rollout and schema migration

Why it matters

Many production incidents start with a change: a deploy, a configuration push or a database migration. The practices in this topic make changes small, observable and reversible, so that a bad change reaches few users and is undone quickly. Interviewers raise them in system design ("how would you roll this out?"), in incident discussions ("how did you notice, and what did you do?") and in project deep dives, and the expected depth rises with level: a graduate should know the vocabulary and the reasons, while a senior engineer should be able to plan a multi-service migration that never takes the system down.

Core ideas

Pipelines. Continuous integration merges small changes often and runs automated builds and tests on each, so integration problems surface within minutes rather than at release time. A deployment pipeline builds an artifact once (a container image, a binary) and promotes that same artifact through environments, with configuration supplied per environment; rebuilding per stage invites "it worked in staging" surprises. Reproducible builds with pinned dependencies make every release traceable to exact inputs.

Progressive delivery. A rolling update replaces instances gradually; blue-green keeps two complete environments and switches traffic between them; a canary sends a small slice of traffic to the new version and compares its behavior with a control population before widening. A useful canary has predefined success metrics (error rate, latency percentiles, saturation, key business signals), enough traffic and time to detect problems, and an automatic stop. Feature flags separate deploying code from releasing behavior: code ships dark, is enabled for internal users, then a share of tenants, then everyone, and a kill switch turns it off without a deploy. Flags are also debt, since each one multiplies the states you might need to test, so remove release flags once a feature is fully on.

Rollback and compatibility. Rollback is only fast if the old version can run against the current state. Old and new versions coexist during every rolling deploy, and if you may roll back from version N+1 to N, then N must cope with anything N+1 wrote. The standard technique is two-phase: first ship code that can read the new format while still writing the old one, and only when it runs everywhere ship code that writes the new format.

Expand, migrate, contract. A schema change becomes a sequence of individually safe steps. Expand: add the new column or table, nullable or with a default, and deploy code that writes both old and new forms. Migrate: backfill existing rows in throttled batches, switch reads to the new form and verify parity. Contract: once nothing reads or writes the old form, stop writing it and drop it. Each step can be reversed on its own and none needs downtime. Watch for statements that lock or rewrite large tables, and use online alternatives where your database offers them, such as PostgreSQL's CREATE INDEX CONCURRENTLY.

Observability. Logs record discrete events with context; make them structured and include request, trace and tenant identifiers. Metrics are cheap numeric time series for dashboards and alerts; the SRE book's four golden signals are latency, traffic, errors and saturation. Traces follow one request across services as a tree of spans and show where time went and which dependency failed. Alert on user-visible symptoms tied to service level objectives rather than on every internal cause, and give every alert an owner and an action.

Common misconceptions

• "Rollback is always the safe option." If the new version already wrote data in a new format or ran a destructive migration, the old version may fail; establish rollback compatibility before deploying.
• "Feature flags make deployments risk-free." A flag flip is a production change that needs the same care, testing and audit trail as a deploy.
• "A canary without errors proves the release is safe." Small or unrepresentative traffic, short bake times and missing metrics hide slow leaks and tenant-specific bugs.
• "Logs are enough." Logs alone make aggregate questions slow and expensive and do not show cross-service timing; metrics and traces answer different questions.
• "Rename the column in one migration during a quiet hour." A rename breaks every running instance that still uses the old name; expand, migrate, then contract.

How to approach it

Read the SRE book chapter on release engineering and the SRE Workbook chapter on canarying, then Sadalage and Fowler's article on evolutionary database design, Pete Hodgson's article on feature toggles, and the OpenTelemetry observability primer. Practice by writing a rollout plan for a real change you made, such as adding a column to a production table: list each deploy, what is compatible at each point, the metrics you would watch and the rollback for every step. In interviews, present the plan in phases with explicit verification gates, and say how you would detect a problem before users report it. Mid-level candidates are expected to change one service safely; senior candidates should coordinate changes across services, data and teams.