Putting it together: build a resilient backend service

Build a small POST /charge service (Go, JVM, or Node) that fronts a flaky downstream 'payment provider' and a database, combining a bounded pool, idempotent retries, a circuit breaker, graceful shutdown, and edge load shedding — then prove under a fault-injected load test that it keeps goodput high and loses no data instead of cascading into collapse.

• Implement POST /charge on one code path that exercises every mechanism: parse with a body-size limit, set a per-request timeout budget in middleware, acquire a connection from a bounded pool with an acquisition timeout, call the downstream provider through a circuit breaker with its own call timeout, and persist the result under an idempotency key.
• Make the write idempotent atomically: a UNIQUE constraint on the idempotency key with INSERT ... ON CONFLICT (or equivalent), so a retried POST returns the original result and never double-charges, even under concurrent retries.
• Nest the timeouts: the acquire timeout, provider-call timeout, and write timeout must each be strictly smaller than the request budget and sum, with margin, to less than it; cap retries with exponential backoff and jitter.
• Add an edge load shedder: an admission concurrency limiter that returns a fast 503 with Retry-After when in-flight work exceeds a configured cap, dropping the least-valuable load first (e.g. retries before first attempts).
• Implement graceful shutdown on SIGTERM: stop accepting new requests, drain in-flight charges within a grace period, tear down in reverse dependency order (close the pool last), and force-exit at a deadline while logging any forced kills.
• Wire observability for the seams: RED metrics (rate, errors including 503s and breaker rejections, and a duration histogram for p50/p99/p999), plus pool saturation (in-use vs limit and acquire wait), breaker state and transitions, retry rate, and drain duration.
• Build a fault-injected load test: a provider that can be slowed (p99 to multiple seconds) and made to error, a client that retries, and a load level above saturation — run a baseline, then inject a slow provider during a simulated rolling restart, and capture the metrics.

• Under the slow-provider injection, p99 of unrelated requests stays bounded and the pool does not deadlock: the acquire timeout fires and the breaker trips, shown in the metrics rather than asserted.
• Under over-saturation load the shedder returns fast 503s and goodput (requests completed within their deadline) stays high while the service never falls to zero — demonstrated with a before/after table of goodput, p99, and 503 rate.
• A retry-storm test shows capped retries plus the breaker prevent a metastable collapse: when the provider recovers, the service recovers on its own without a manual bounce.
• A SIGTERM during sustained load drops zero in-flight charges (the idempotency table shows no duplicates and no lost writes) and the drain completes inside the grace period, with drain duration logged.
• A short write-up naming, for each requirement, which mechanism closed which failure mode, and which gates you consciously left partly open given the service's blast radius.