Engineering Practice
Feature flags: decoupling deploy from release without drowning in flag debt
On August 1, 2012, Knight Capital deployed new code to 7 of its 8 trading servers. The new code reused an old flag named Power Peg — a 2003 test feature that had been dead for years. On the eighth, unpatched server, flipping that flag woke the dead code. In 45 minutes it fired 4 million erroneous trades across 154 stocks and lost roughly $440 million — more than the firm’s market cap. A stale flag, left in the codebase and reused, ended a 1,400-person company.
Deploy is not release
A senior’s first reframe: deploying code and releasing a feature are two different events. Without flags they are welded together — the merge that hits main is the moment users see the change, so every risky launch becomes a high-stakes deploy at 2am with the whole team watching. A feature flag splits them. You deploy the code “dark” (present in production, wrapped in if (flags.newCheckout) {...} and returning false for everyone), then release later by flipping the flag — no rebuild, no redeploy.
That split changes how teams ship. Code merges continuously and small (trunk-based development stops depending on long-lived branches), incomplete features sit safely behind an off flag, and the release becomes a config change an SDK picks up in seconds. The Unleash client spec polls flag state on a default 15-second interval; LaunchDarkly streams updates over SSE so a toggle propagates to running servers in well under a second. Release stops being a deploy and becomes a decision.
The four flag types — and why their lifecycles differ
“Feature flag” hides four distinct kinds, and the senior mistake is treating them all the same. The type dictates how long the flag should live and who owns it.
| Type | Purpose | Lifespan | Owner |
|---|---|---|---|
release | Gate an in-progress feature; gradual rollout | Days to weeks — delete after 100% | Dev team |
ops / kill-switch | Disable a subsystem under load or incident | Permanent — kept on purpose | SRE / on-call |
experiment | Serve A/B variants, measure a metric | One experiment cycle — then delete | Product / data |
permission | Gate by plan, role, or entitlement | Long-lived — tied to the product model | Product |
A release flag that never gets deleted has quietly become flag debt. A kill-switch that someone “cleans up” because it looked stale has removed your safety net. Same mechanism, opposite correct fate — which is exactly why the type has to be recorded, not guessed.
Gradual rollout and the instant rollback
The release flag’s superpower is the percentage rollout. Instead of 0% → 100%, you ramp: 1% → 5% → 25% → 100%, watching error rate and latency at each step. If the new path breaks at 5%, you flip the flag off and the blast radius was 5% of traffic, recovered in seconds — no commit revert, no hotfix pipeline, no redeploy. That is the real argument for flags: rollback stops being an engineering event and becomes a config toggle.
The mechanism matters for correctness. A good rollout is sticky: the same user must keep getting the same variant across requests, or your UI flickers and your experiment data is garbage. SDKs do this by hashing a stable key (userId plus a groupId) into a 0–99 bucket; “25% rollout” means buckets 0–24 are on. The hash is deterministic and computed locally, so evaluation is sub-millisecond and needs no network call per check — the SDK holds the whole ruleset in memory and refreshes it in the background.
Why this works
Why hash locally instead of asking the server per evaluation? At scale a flag is checked thousands of times per request path. A network round-trip per check would add latency and a hard dependency: if the flag service is down, your app is down. Local in-memory evaluation with background sync means a flag check is a hashmap lookup, and a flag-service outage degrades to “last known config” rather than an outage of yours.
Every flag is a branch in production
Here is the cost a senior weighs against all that velocity. Each live flag is an if/else that both paths of are running in production simultaneously. Ten independent boolean flags is 2^10 = 1,024 possible runtime configurations — you cannot test them all, and the combination a user actually hits in prod may be one no test ever exercised. Flags multiply the state space of your system. They also rot: a flag left at 100% for months is dead config that still gets evaluated, still clutters the code with a branch nobody reads, and — the Knight Capital lesson — can be reused, waking code everyone forgot was there.
This is flag debt, and it is not hypothetical. LaunchDarkly’s own guidance defines a stale flag as one serving the same variation to everyone for over ~30 days, and recommends archiving on a schedule; tools like Uber’s Piranha exist specifically to AST-parse codebases and auto-generate the pull request that deletes a flag and its dead branch. The discipline is the whole game: a release flag must carry an expiry and a Jira ticket to remove it, kill-switches must be labeled permanent so nobody “cleans” them, and removal is part of the feature’s definition of done — not a someday-maybe.
A new checkout path is built and tested in staging. You want to ship it to production today but de-risk the launch. Pick the rollout approach.
What does decoupling deploy from release actually mean?
A release flag has been at 100% rollout for three months and nobody has touched it. What's the senior call?
Order the lifecycle of a release flag from creation to retirement:
- 1 Create the flag, default off; deploy the code dark to production
- 2 Release to 1–5% of users, sticky by userId; watch error rate and latency
- 3 Widen the rollout to 25% → 100% while metrics stay clean (or kill-switch off if not)
- 4 Once stable at 100%, remove the flag and delete the now-dead else branch
- 5 Confirm no references remain in code or config; close the cleanup ticket
- 01Explain how feature flags decouple deploy from release, and why that changes the way a team ships.
- 02What is flag debt, why is the Knight Capital incident the canonical example, and what discipline prevents it?
Feature flags decouple deploy from release: code ships dark to production and a runtime toggle — picked up by the SDK in seconds — decides exposure, so release becomes a decision rather than a high-stakes deploy. That buys gradual percentage rollout (1% → 5% → 25% → 100%, sticky by userId so variants don’t flicker) and instant kill-switch rollback, with evaluation done locally as a sub-millisecond hash lookup so a flag-service outage degrades gracefully. But the four flag types — release, ops/kill-switch, experiment, permission — have opposite correct lifespans, and every live flag is a branch in production, so N flags mean 2^N configurations you can’t fully test. The failure mode is flag debt: stale or forgotten flags that still evaluate, clutter the code, and can be reused — the exact mechanism that cost Knight Capital ~$440M in 45 minutes in 2012. The senior discipline is lifecycle: type every flag, give release flags an expiry and a removal ticket, label kill-switches permanent so nobody deletes the safety net, and make flag cleanup part of done.