Platform engineering, infrastructure as code and safe migrations

Why it matters

As systems grow, the bottleneck moves from writing code to changing production safely: provisioning, deploying, migrating data and seeing what happened. Platform teams build the paved road that makes the safe way the easy way. Infrastructure roles probe this directly, and senior candidates in any backend or distributed role are expected to explain how a risky change (a schema change, a storage migration) reaches production without downtime.

Core ideas

Platforms as products. A platform offers self-service capabilities (compute, databases, queues, pipelines, observability) behind stable interfaces, with sensible defaults and guardrails. Success means product teams adopt it and ship faster with fewer incidents, not that it bundles many tools.

Declarative infrastructure as code. You describe the desired end state and a tool computes and applies the difference. With a plan step, the tool compares your configuration with its recorded state, refreshed against the real infrastructure, and shows the intended changes. Review plans before applying: a renamed resource can appear as destroy-and-recreate.

Drift is any difference between declared and real state, usually from manual changes or other tools. Detect it with scheduled plans or continuous reconciliation, then either re-apply the declaration or update the code to capture a deliberate change. Pull-based (GitOps-style) systems run a controller that keeps reconciling the environment toward a version-controlled declaration, the same loop Kubernetes controllers run.

Delivery pipelines. Build once, test, and promote the same artifact through environments. Release progressively: send a canary slice of traffic to the new version, compare its errors and latency with a baseline, widen in steps, and roll back automatically when the comparison fails. Feature flags separate deploying code from releasing behavior, so behavior can be turned off without a redeploy.

Observability by default. Every service on the platform should get request metrics (rate, errors, latency distributions), structured logs carrying request IDs, distributed traces and SLO dashboards without extra work. Adding them during an incident is too late.

Safe migrations: expand and contract. Split a backward-incompatible change into compatible steps:

1. Expand: add the new column, table or field; remove nothing.
2. Write both old and new (dual write), still reading the old.
3. Backfill historical data in small, throttled, idempotent, resumable batches.
4. Verify by comparing old and new (shadow reads, checksums), then switch reads.
5. Stop writing the old path, then contract: remove it. Every step must work with both the previous and the next code version, so you can roll back at any point.

Dual writes are a risk to manage. Writing two stores from application code is not atomic: a crash between the writes leaves them inconsistent, and concurrent updates can land in different orders. Prefer one source of truth plus change data capture or an outbox, and in every case run a verification job with a repair path. Data-format changes need two-phase deployment too: first ship code that can read the new format everywhere, then enable writing it, so a rollback never meets data it cannot parse.

Common misconceptions

• "Infrastructure as code means scripts in version control." The value lies in declarative desired state, reviewable plans and reconciliation, not in storing imperative scripts.
• "Drift is harmless if things work." Undeclared changes vanish or conflict on the next apply and make incidents harder to reason about.
• "A migration is one deploy plus a script." Zero-downtime changes take several deploys, each compatible with its neighbors, plus a verified backfill.
• "Dual writes keep two stores in sync." Without atomicity or reconciliation they drift apart; build detection and repair.
• "A clean canary proves the release is safe." It shows the slice was fine for that traffic and duration; compare against a baseline and watch for delayed effects such as data-format problems.

How to approach it

Read the CNCF platforms paper for vocabulary, then do the Terraform Docker tutorial and create drift on purpose. Read the Stripe online-migrations post, which walks through dual writes, a backfill and a verified read switch, and write the step list for a migration you might face, such as splitting a contacts table into person and company records. Graduates should know CI/CD and why migrations need care. Mid-level candidates should plan an expand-and-contract migration and a canary rollout. Senior candidates should design the platform interface, its guardrails, and the verification and rollback plan for a large migration.