Progressive delivery: canary, blue-green and flags

Deploy is not release

This is the framing the whole lesson hangs on. Deploying is putting new code on servers — the binary is running, the container is up. Releasing is directing user traffic to that code — deciding who actually hits it. The previous lessons got you to a published artifact; they did not say a single user should see it yet. The senior move is to treat those as two separate switches. Ship the code dark — deployed but receiving no traffic — and then turn the traffic dial up gradually while you watch the signals, ready to turn it back down the instant they degrade. Everything below is a different mechanism for that one idea: separate “the code is here” from “users are on it”, and put a blast-radius limiter on the second switch.

The four rollout strategies

# big-bang (recreate): stop everything, start the new version — downtime + 100% blast radius
strategy:
  type: Recreate          # old pods die, new pods start: a gap where nothing serves

# rolling: replace instances batch by batch — no downtime, but two versions run at once
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 0      # k8s default-ish: incremental, but NO explicit traffic gate

• Big-bang (recreate / all-at-once): stop the old version, start the new one. There is a downtime gap, and the moment it is up the blast radius is 100% — every user is on untested-in-prod code at once. This is the Hook’s failure. Acceptable only for tiny or internal apps where a blip is fine.
• Rolling: replace instances batch by batch (the Kubernetes default). No downtime, which is why it feels safe — but two versions run simultaneously, so they must be backward/forward compatible, and a bad version still reaches users incrementally with no explicit gate watching whether it is healthy. Rolling controls availability, not risk.
• Blue-green: stand up two full environments. Deploy to GREEN (idle), smoke-test it in isolation, then flip the load balancer / router from BLUE to GREEN atomically. The cutover is instant, and so is rollback — flip the router back to BLUE. The cost is roughly 2x infra during the switch, and the hard part is shared state: both versions usually talk to one database, so a schema change must be compatible with both BLUE and GREEN across the flip.
• Canary: route a small percentage of real traffic (1-5%) to the new version, watch its error rate, latency, and business metrics against the old version, then ramp 5 → 25 → 50 → 100% if it stays healthy, or abort if it does not. The canary catches problems with real production traffic at a tiny blast radius — which is exactly the bug class staging missed.

Feature flags: the release switch that needs no redeploy

A canary moves traffic at the infrastructure layer. A feature flag moves the release decision into the application layer: wrap the new behaviour in a conditional, deploy it with the flag OFF, and the code ships dark while users keep getting the old path. Then you flip the flag — for 1% of users, then 10%, then everyone — independently of any deploy. The new code was already on every server; you are just changing a config value.

// the new code is deployed everywhere, but dark until the flag opens it
if (flags.isEnabled("new-checkout-pipeline", { userId })) {
  return runNewCheckout(order);   // exposed to 1% -> 10% -> 100% via config, no redeploy
}
return runLegacyCheckout(order);  // everyone else, unchanged

Two properties make this the senior’s favourite tool. First, it is an instant kill switch: if new-checkout-pipeline misbehaves, you flip the flag back to OFF and every user is on the old path in seconds — no pipeline run, no image rebuild, no redeploy. Second, it does per-user / percentage targeting, which is what powers A/B tests and dark launches (run the new path, discard its output, just measure its load). When you own a flag that’s been “temporarily enabled” for six months, you’re already in the cost column. The cost is real: flag debt. Every flag is a branch in your code and a row in your config; stale flags that were never cleaned up become a combinatorial mess nobody dares delete. Flags are a release lever, not a permanent home — open them, learn, then remove them.

Automated rollback: the gate decides, not a human

Everything above is just safer if a human is watching. The senior version is automated. A progressive-delivery controller — Argo Rollouts, Flagger, or logic in your deploy pipeline — ramps the canary on a schedule and, at each step, runs an analysis query against your metrics: error rate, p99 latency, SLO burn rate. If the query breaches a threshold, the controller auto-aborts and rolls back — shifts traffic back to the stable version — with no human in the loop. That is the key number to internalise: the rollback decision is gated on metrics, not on vibes and not on someone happening to be staring at a dashboard at 4pm.

# Argo Rollouts canary with a metric-gated analysis step (sketch)
strategy:
  canary:
    steps:
      - setWeight: 5            # 5% of traffic to the new version
      - analysis:               # query Prometheus; auto-abort if it fails
          templates: [{ templateName: error-rate }]
      - setWeight: 25
      - analysis: { templates: [{ templateName: error-rate }] }
      - setWeight: 50
      - setWeight: 100          # full promotion only if every gate passed

Run the Hook’s incident through this. The bad version goes to 5% of traffic. The serialization bug fires under real concurrency and the analysis query sees the error rate on the canary spike past its threshold within seconds. The controller aborts at 5% and shifts traffic back to stable — automatically. Blast radius: 5% of users for a few seconds, instead of 100% of users for minutes. The bug still happened; progressive delivery just made it a non-event.