backend · intermediate · 6d
Feature-flag service
Build a small flag service with targeting rules, percentage rollouts, and a typed SDK that evaluates flags client-side from a cached ruleset.
Deliverable
An API that serves a versioned flag ruleset (ETag + 304) and a typed SDK where flagOn('x', user) evaluates in-process with no per-call I/O, deterministic salted bucketing, and a kill-switch that propagates in seconds.
Milestones
0/5 · 0%- 01Ruleset model + ETag revalidation
Model flags and targeting rules with a typed, validated schema, then serve the ruleset so SDKs can cache it cheaply. Each flag has a key, a default value, ordered targeting rules (e.g. `email endsWith @company.com`, `country == DE`, `userId in allowlist`), and a percentage rollout config. The schema is zod-validated at write time — a malformed rule is rejected on create, not at eval time when it would silently mis-evaluate. Serve GET /flags with a strong ETag (hash of the serialized ruleset version); SDKs store the ETag and revalidate with If-None-Match, receiving 304 with no body when unchanged (only ~200 bytes of headers). This makes frequent polling cheap and lets thousands of SDK instances revalidate simultaneously without bandwidth spikes — the right default before streaming.
Definition of done- GET /flags returns the full ruleset with a strong ETag; a matching If-None-Match returns 304 with no body and correct Cache-Control/ETag headers.
- Flag + rule schema is zod-typed and validated at write time; a malformed rule (bad operator, missing field) is rejected with 400 and never stored.
Feeds fromSelf-review
Show a malformed rule rejected at write time and a 304 revalidation with headers. A senior reviewer checks the ETag is a hash of the ruleset version (not a timestamp) and that validation happens before storage.
- 02Deterministic salted rollout
Implement percentage rollout so the same user always gets the same answer and the enabled fraction is ~X% at X% rollout. The primitive is `hash(userId + flagKey) mod 100 < rolloutPercent`: salting with flagKey is load-bearing — hashing userId alone puts the same 10% of users in the first decile for every flag, a systematic bias that skews any A/B experiment relying on independent treatment assignments. Monotonic bucket expansion matters too: increasing rollout from 10% to 20% must never disable already-enabled users (you widen the threshold, not reshuffle buckets). Prove correctness with a simulated population (e.g. 10k users): at 10% ~1k are enabled, at 20% ~2k and the original 1k are still in, and a chi-squared test shows uniform bucket distribution. Document why `Math.random()` per request would be wrong here.
Definition of done- hash(userId + flagKey) buckets deterministically; at X% rollout ~X% of a 10k simulated population are enabled and the same user never flickers.
- Increasing rollout from 10%→20%→50% keeps all previously-enabled users enabled (monotonic) and a chi-squared or bucket-count check shows uniform distribution, not clustering.
Feeds fromSelf-review
Show the 10k-user bucket histogram and the monotonic expansion test (10%→20% keeps the original cohort). A senior reviewer checks the hash is salted with flagKey (not userId alone) and asks you to explain the A/B bias if it weren't.
- 03Typed SDK with in-process evaluation
Build the SDK that makes flag evaluation a local, typed, zero-network call. `flagOn('flagKey', user)` evaluates against the cached ruleset in-process: walk targeting rules in priority order, first match wins, fall through to the salted percentage rollout, then to the default. No per-call fetch — the ruleset was cached via ETag and revalidated on a background interval, so the hot path is a pure function `ruleset × user → boolean` with no I/O. Type safety matters: the SDK is generic over the flag key so `flagOn('nonexistent', ...)` is a compile error, and the return type narrows by flag (boolean vs string vs JSON). Prove it with a benchmark: 100k evaluations in-process vs the naive 'fetch ruleset per call' baseline, and show the latency difference (microseconds vs milliseconds). Document the evaluation order and what happens when a user matches multiple rules.
Definition of done- SDK flagOn evaluates in-process from the cached ruleset with no per-call network I/O; targeting rules are priority-ordered with deterministic fallthrough to rollout → default.
- SDK is typed so an unknown flag key is a compile error; a benchmark shows in-process eval at microseconds vs milliseconds for fetch-per-call, with numbers recorded.
Feeds fromSelf-review
Show the benchmark (100k evals in-process vs fetch-per-call) and a compile-error for an unknown flag key. A senior reviewer checks evaluation is priority-ordered with explicit fallthrough and that no I/O happens on the hot path.
- 04Kill-switch SLO and streaming propagation
Make the kill-switch real: turning a flag off must propagate to every SDK instance fast enough to be useful in a security incident. With polling alone and a 60s TTL, a flag turned off stays on for up to 60s in every SDK — an exposure window that is the hidden SLO of the flag service. Add a streaming update channel (SSE): the server pushes a 'ruleset version N+1 available' event, SDKs revalidate immediately, and propagation drops from TTL to ~seconds (SSE latency + one revalidation). Keep polling as fallback for clients that miss the stream. Measure both paths: with polling-only, flip a flag and record worst-case propagation delay (up to TTL); with SSE, record p50/p95 push-to-eval latency. Document the tradeoff: SSE eliminates the window but adds a persistent connection per SDK instance (fan-out cost); polling is connection-free but has a bounded staleness window. State when split-evaluation (some SDKs on old ruleset, some on new) is acceptable (gradual rollout) vs not (security kill-switch).
Definition of done- SSE channel pushes ruleset version bumps to SDKs; flipping a kill-switch propagates to all connected SDKs in seconds (push-to-eval p95 recorded), with polling as fallback when SSE is disconnected.
- Propagation SLO is documented with numbers: polling-only worst-case (TTL) vs SSE p95, and split-evaluation acceptability is stated for gradual rollout vs security kill-switch.
Feeds fromSelf-review
Flip a flag and show propagation latency polling-only vs SSE (TTL vs seconds). A senior reviewer checks you can state the kill-switch exposure window, the SSE fan-out cost, and when split-evaluation is unacceptable.
- 05Load-test, observe, and work an incident
Prove it under production-like load and make it observable when it misbehaves. Load-test the flag service with many SDK instances (e.g. 50 concurrent evaluators, mixed flag keys, 10% rollout flags, kill-switch flips mid-test) and find the QPS where ruleset revalidation — not evaluation — is the bottleneck (304s are cheap but still cost a round trip). Emit RED metrics (request rate, ruleset fetch rate, 304 ratio, evaluation rate, SSE connection count) and a trace span for the ruleset fetch so the flag system's own latency is visible in the waterfall. Then work an incident: flip a flag's targeting rule mid-load and watch split-evaluation — some SDKs on the old ruleset, some on the new — cause inconsistent flagOn results for the same user. Detect it from your metrics (304 ratio dip + evaluation inconsistency), mitigate (force revalidation / SSE push), and write a 5-line post-mortem whose prevention is not 'reduce TTL to 1s'.
Definition of done- A sustained load test (≥50 concurrent SDKs, mixed flags, mid-test flag flip) reports throughput, 304 ratio, evaluation rate, and SSE connection count on a dashboard with the ruleset fetch trace span visible.
- You reproduced a split-evaluation incident (rule flip mid-load), detected it from metrics, mitigated it, and wrote a post-mortem naming root cause and a prevention that isn't 'reduce TTL to 1s'.
Feeds fromSelf-review
Paste the dashboard and post-mortem. A senior reviewer checks the bottleneck is attributed to ruleset revalidation (not evaluation), the trace localized it, and the prevention addresses propagation (SSE/polling) — not just 'lower TTL'.
Starter
fallowlone/skein-projects
projects/feature-flags-service
- README.md
- src/flags.ts
- test/flags.test.ts
npx degit fallowlone/skein-projects/projects/feature-flags-service feature-flags-service Implement the stubs, then run the tests until they pass: bun test
Fork the repo and push your work — the grade workflow runs the suite plus static checks on your own runners.
Rubric
| Junior | Mid | Senior | |
|---|---|---|---|
| Deterministic rollout | Percentage rollout uses Math.random() per request; the same user gets a different answer on consecutive calls and the enabled fraction is approximate at best. | hash(userId + flagKey) mod 100 buckets a user deterministically: the same user always resolves to the same bucket, ~X% are enabled at X, and monotonic bucket expansion means increasing the rollout never disables already-enabled users. | You reason about bucket collisions between flags: if you hash userId alone (not userId+flagKey), users in the 0–10% bucket are always the same people for every flag — a systematic bias that skews A/B results. Salting with flagKey breaks the correlation. You demonstrate this with a chi-squared distribution test on a simulated user population. |
| Ruleset caching & propagation latency | The SDK fetches the full ruleset from the API on every evaluation call; a high-traffic service adds a network round trip to every request. | The ruleset is served with an ETag; SDK clients cache it locally and revalidate with If-None-Match, receiving 304 (no body transfer) when unchanged — evaluation is in-process with no per-call network I/O. | You reason about kill-switch propagation latency: a cache TTL of 60s means a flag turned off for a security incident stays on for up to 60s in every SDK. Streaming (SSE push) eliminates that window but adds a persistent connection per SDK instance. You document the chosen tradeoff and the worst-case propagation delay under your polling interval. |
| Eval consistency & targeting correctness | Targeting rules are applied in arbitrary order; a user matching multiple rules gets a non-deterministic result depending on evaluation sequence. | Rules are evaluated in priority order with an explicit fallthrough to the percentage rollout; the same rule model and same user always produce the same boolean, and the schema is typed and validated so a malformed rule is rejected at write time, not at eval time. | You address the stale-ruleset window: an SDK client evaluating a cached ruleset while the server has already changed a rule produces a split-evaluation inconsistency — some users see the old behavior, some the new. You document when this is acceptable (gradual rollout) vs. when it is not (a security kill-switch needing propagation in seconds), and show how the streaming channel closes the gap. |
Reference walkthrough (spoiler)
Why hash(userId + flagKey): hashing userId alone creates systematic bucketing correlation — the same 10% of users are always in the first decile for every flag. This biases any A/B experiment that relies on independent treatment assignments. Salting with the flagKey decorrelates buckets across flags; salting with an experiment-id decorrelates across re-runs of the same flag.
Kill-switch propagation latency is the hidden SLO: a flag service is usually a background concern until a security incident requires turning something off immediately. Polling-based SDKs add up to one TTL of exposure after a kill-switch is flipped. SSE streaming eliminates that window at the cost of a long-lived connection per SDK instance — the tradeoff is connection count (fan-out) vs. propagation time.
ETag caching for ruleset distribution: serving a versioned ETag with the ruleset and accepting If-None-Match means unchanged rulesets transfer only 200 bytes of headers, not the full payload. This makes frequent polling cheap and lets many SDK instances revalidate simultaneously without bandwidth spikes — the right default for polling-based SDKs before adding streaming.
Make it senior
- Add percentage-based experiment assignment with sticky bucketing so a user stays in the same variant even when the experiment restarts — and prove it with a distribution test.
- Add flag prerequisites (flag B only evaluates if flag A is on) and prove the DAG has no cycles, with evaluation short-circuiting.
- Add audit logging and flag-history diffing so every rule change is attributable and reversible, and show a rollback that restores the previous ruleset version.