Distributed tracing and event-driven plumbing: X-Ray and EventBridge

Tracing: the third pillar, and why logs alone fail

Metrics and logs are per-component. A metric is a number from one box; a log line is one event on one box. Neither knows that this request to checkout fanned out into a call to cart, then payment, then a tax service, then DynamoDB — and that the 1.6 seconds of extra latency was spent waiting on the tax hop. To answer “where in the call chain did the time go?” you need a record that spans services and stitches them into one timeline. That is distributed tracing.

AWS X-Ray models a request as a trace: the collection of all work done to serve one request, identified by a single trace ID. Each service contributes a segment — a document recording that service’s name, the request and response, start and end times, and any errors or faults. Within a segment, every downstream call your code makes is a subsegment: a call to DynamoDB, an external HTTP API, or a SQL query, each with its own timing. For a downstream that doesn’t trace itself (DynamoDB doesn’t emit segments), X-Ray builds an inferred segment from your subsegment, so the dependency still shows up on the map. This is the structural difference from logs: a trace is a tree of segments and subsegments, not a flat stream of lines, and the tree is exactly the shape of your call graph.

The glue that ties segments across process boundaries is a propagated header: X-Amzn-Trace-Id. The first X-Ray-integrated service a request hits stamps it with a root trace ID and a sampling decision, and every instrumented hop forwards it downstream, adding its own parent segment ID so X-Ray can reconstruct parent/child edges.

X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Parent=53995c3f42cd8ad8;Sampled=1

X-Ray rolls every service’s segments for a trace ID into a service map (service graph): nodes for each service, edges for each call between them, each edge annotated with latency and error rate. The map is what turns “checkout is slow” into “the checkout→tax edge is at 1.6 s p99 and 0% errors” — a thing logs structurally cannot show you, because a log line lives inside one node and knows nothing about the edge.

Sampling: you trade completeness for cost on purpose

Tracing every request is expensive twice over: the per-trace bill and the latency the SDK adds collecting and shipping segments. So X-Ray samples. The default rule is a reservoir plus a rate: record the first request each second, then 5% of everything beyond that. The reservoir guarantees you always capture a baseline of traces even at low traffic; the 5% rate keeps a high-volume endpoint from recording millions of near-identical traces. You configure rules per service or request property — disable sampling (trace 100%) for state-changing or payment paths where every trace matters, and sample health checks and background polling at a low rate because they’re noise.

The cost is real, and it is the most common observability failure mode. X-Ray bills per trace recorded and, separately, per trace retrieved or scanned. As illustrative figures from the CloudWatch pricing page (US East / N. Virginia — prices are region-dependent, always check your region): traces recorded cost about $5.00 per million ($0.000005 each) beyond a free tier of 100,000 per month, and retrieved/scanned traces about $0.50 per million beyond 1,000,000 free. Those numbers look trivial until someone sets sampling to 100% on a path doing 5,000 requests/second: that is ~13 billion traces a month, roughly $65,000 in recording alone — the classic “we turned on full tracing in prod and got a five-figure surprise” story. Sampling is not a quality compromise to apologize for; it is the deliberate trade that makes tracing affordable.

EventBridge: decouple the hops you just traced

The tax-lookup incident had a second lesson: that synchronous call should never have been synchronous. When the payment service must call the tax service inline and wait, the two are tightly coupled — the slow one drags the fast one, and a deploy to either can break the other. Amazon EventBridge is the serverless event bus that breaks that coupling. A producer calls PutEvents to publish an event; it neither knows nor cares who consumes it. Rules on the bus match an event pattern against the event’s fields — source, detail-type, and arbitrary detail JSON — and route every match to one or more targets (a Lambda, an SQS queue, a Step Functions state machine, another bus) that run in parallel.

{
  "Source": "checkout.service",
  "DetailType": "OrderPlaced",
  "Detail": {
    "orderId": "o-4821",
    "region": "eu-west-1",
    "amountCents": 7400,
    "needsTax": true
  }
}

A rule that fans this out to a tax-calculation Lambda and an analytics queue matches on the pattern below; anything not matching is silently ignored, so adding a consumer never touches the producer:

{
  "source": ["checkout.service"],
  "detail-type": ["OrderPlaced"],
  "detail": { "needsTax": [true] }
}

The default bus receives events from AWS services themselves (an EC2 state change, an S3 upload), which is how you build automation that reacts to your own infrastructure; custom buses carry your application’s domain events, isolating them from the AWS-service firehose. A schema registry can infer and version the shape of events so consumers code against a contract instead of a guess. The payoff is choreography: checkout emits OrderPlaced and walks away; tax, analytics, and email each subscribe independently. No service holds a list of who to call next, so you add and remove consumers without redeploying producers — exactly the coupling that turned a third-party SDK change into a cross-service incident.

Structured logs: the third pillar, made queryable

Tracing tells you which hop; logs still tell you what happened inside that hop — but only if they’re queryable. A senior emits structured logs: JSON objects with named fields, not human prose. A line like “user 4821 failed payment, code 51” is unsearchable; a record like {“level”:“error”,“traceId”:“1-5759e988”,“userId”:4821,“event”:“payment_failed”,“declineCode”:51} lets you pull every log for one trace ID and join it to the X-Ray timeline. Carrying the trace ID into your logs is what fuses pillars two and three. CloudWatch’s Embedded Metric Format (EMF) goes further: emit a specially shaped JSON log and CloudWatch extracts metrics from it automatically, so you get a metric and its supporting log from one write.

But fields cost money and the trap is cardinality. CloudWatch Logs bills per GB ingested (illustratively about $0.50/GB beyond a 5 GB free tier in US East, region-dependent), and EMF turns high-cardinality dimensions into custom metrics at roughly $0.30 per metric per month. Put userId or requestId as an EMF dimension and every distinct value becomes its own billed metric — a million users becomes a million metrics and a four-figure monthly line item from one careless field. Log fields freely; keep metric dimensions low-cardinality (status, route, region — not user or request id). Over-instrumenting is a failure mode exactly like over-sampling.