Performance
Reading parent and child chains: where to apply the fix
A senior engineer opens pprof and looks at json.Marshal at 28% CPU. They do not reach for a faster JSON library. They check the parent chain. One wide parent: a request logger. The fix is not the library — it is the logger calling Marshal on every request regardless of log level.
Reading the parent chain
The parent chain answers: “where does this wide leaf get called from?”
Concentrated fan-in (one wide parent path): the leaf is called repeatedly from a single location. Fix the caller — call less often, cache results, batch, or eliminate the call.
Distributed fan-in (many narrow parents): the leaf is shared infrastructure used by many callers. Fix the leaf itself, because optimising any one caller leaves the other nineteen unchanged.
The classic example: json.Marshal wide with one wide path from a logger means the logger is the bug — don’t switch JSON libraries, fix the logger. The same json.Marshal wide and called from twenty handlers means JSON is the bottleneck — switch libraries or pre-encode.
Production-grade pprof and Pyroscope let you switch between leaf-up and caller-down views with one click; using both is the senior reading habit.
Reading the child chain (self-time vs cum-time)
The child chain answers: “does this function do the work, or does it delegate to a callee?”
Self-time-heavy leaf with narrow children: the function is doing the work itself. Fix is inside the function.
Cum-time-heavy parent with thin self-time: this function is a dispatcher. The cost sits one level down in a callee. Fix the callee, not this layer.
The reading rule: walk from the leaf with high self-time down the parent chain. Find the level where cum-time drops sharply as you move up. That level is the “boundary of meaningful work” — above it is dispatch, below it is execution. Apply the fix inside that boundary.
Misreading this boundary is how teams end up rewriting middleware that wraps a slow database call, or optimising HTTP routers when the cost is in the handler body.
| Signal | What it means | Fix layer |
|---|---|---|
| Wide leaf, one dominant parent | Concentrated call site | Caller (call less often / batch / cache) |
| Wide leaf, many thin parents | Shared infrastructure | Leaf itself |
| High self-time, narrow children | Function does the work | Inside the function |
| Thin self-time, wide cum-time | Dispatcher | Inside the callee (one level down) |
Fan-in amplification
When one function is called from many callers, optimising the leaf compounds across each caller. A 30% local improvement in a shared function used by twenty paths gives 30% on every one.
The inverse — fan-out: one caller hitting many callees — makes optimising each callee a single-path local saving.
Senior move: look at fan-in/fan-out shape before deciding leaf vs caller fix. High fan-in (shared infra) means leaf fix amplifies. High fan-out (one caller, many callees) means caller fix (batching, caching, eliminating the call) amplifies. This math often outweighs the fix-family choice: even a small fix on a high fan-in leaf can outscore a large fix on a single-caller path.
Bottom-up vs top-down views
pprof, Pyroscope, and Intel VTune all support two complementary views.
Bottom-up: rank functions by self-time, leaf-first. Best for finding which single function eats the most CPU.
Top-down: rank functions by cum-time from root, parent-first. Best for finding which subsystem is responsible for the most time.
Senior reading habit: open bottom-up first to name the hot leaf, then switch to top-down to see which entry-point owns it. A leaf that is hot under one entry-point is a localised fix; a leaf that is hot under many is shared infrastructure.
Why this works
Intel’s Top-Down Microarchitecture Analysis (TMA) extends this further, classifying each hot frame into front-end bound, back-end bound, retiring, or bad speculation. TMA is covered in the hardware counters lesson. The parent/child chain reading here is a prerequisite — TMA adds the CPU-level layer on top.
A Go API has p99 of 600 ms. Profile shows runtime.mallocgc at 18% and runtime.scanobject at 14%. Trace the diagnosis and fix.
A Node service has a wide leaf JSON.parse at 24% CPU. Trace why naively swapping parsers might not help.
Order the steps of the hot-path attack loop, in the order a senior engineer runs them:
- 1 Open the profile and identify the widest leaf by self-time
- 2 Read the parent chain (one caller or many?) and the children (is real cost one level down?)
- 3 Classify the hotspot: CPU, allocation, cache, lock, syscall, or JIT deopt
- 4 Pick the categorical fix family that matches the classification
- 5 Write only the predicted change, no scope creep
- 6 Capture a fresh profile under the same load
- 7 Diff: local frame shrank AND headline metric improved? Both must hold to ship
- 8 If headline did not move, find the new top hotspot — it was masked by the first
A wide leaf function is called from twenty different parent paths, each contributing a thin slice. Where do you fix?
- 01Why is it important to read both the parent chain AND the children when diagnosing a wide leaf, and what does each direction tell you?
- 02What does 'boundary of meaningful work' mean in pprof, and how do you find it?
The parent chain and child chain are two lenses on the same wide leaf. The parent chain reveals fan-in shape: one caller means fix the call site, twenty callers means fix the leaf. The child chain reveals whether the function itself carries the cost or is a pass-through to a callee. The boundary between dispatch and execution is where the effective fix location lives. Bottom-up and top-down profile views make both reads mechanical: bottom-up names the hot leaf, top-down shows which subsystem owns it. Together these two reads eliminate the most common fix-layer error in performance work.
- Hardware counters and Intel TMA: sub-category diagnosissenior
- False sharing and native-bridge hot pathssenior
- Hot paths in production: security, tail latency, and tooling lineagesenior
- Hot paths: diagnose and fix two shapessenior
- Hot paths: multiple-choice reviewsenior
- Hot paths: code and counter readingsenior
- Hot paths: free-recall reviewsenior
appears again in159
- The journey of a request: seven stops from socket to responsejunior
- Accept and parse: from kernel queue to a typed requestmiddle
- Routing and middleware: choosing what runs, and in what ordermiddle
- Handler and response: from business logic to bytes on the wiremiddle
- Streaming and backpressure: when the client reads slower than you writesenior
- Timeouts and tail latency: budgets, deadlines, and the fan-out trapsenior
- Middleware and DI: the two patterns that shape every backendjunior
- Writing middleware: signatures, next(), and the three framework modelsmiddle
- Inversion of control: how dependencies reach a classmiddle
- DI scopes and lifecycles: singleton, request, transientmiddle
- DI as a testing seam: fakes, mocks, and the boundary that matterssenior
- DI containers in production: resolution graphs, circular deps, and when not tosenior
- Blocking vs non-blocking I/O: two ways to waitjunior
- The event loop: one thread, ordered phasesmiddle
- What blocks the loop: CPU work and sync callsmiddle
- Offloading CPU work: worker threads and the libuv poolmiddle
- Backpressure and bounded concurrencysenior
- Throughput under load: tail latency and saturationsenior
- Why pool: the cost of creating a connectionjunior
- Pool sizing: why bigger is not fastermiddle
- Acquisition and timeouts: the wait queue is the real latency dialmiddle
- Retry strategies: backoff, jitter, and thundering herdmiddle
- Observability, production failures, and global-scale designsenior
- Tasks, microtasks, and scheduler.yield()middle
- Timer accuracy, throttling, and idle workmiddle
- Node.js event loop: phases, nextTick, and loop lagsenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- LCP: four phases, one dominant costmiddle
- INP: input delay, processing, presentationmiddle
- Lab vs field: why the two disagree and how to use eachmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- What an index is and how it speeds up queriesjunior
- The leading-column rule and composite index designmiddle
- Partial, expression, and covering indexesmiddle
- Index types: GIN, GiST, BRIN, Hash, Bloom, and HOT updatesmiddle
- Index-only scans, the Visibility Map, and INCLUDEsenior
- Production failure modes and the index audit playbooksenior
- Index design exercise: full-text search strategysenior
- EXPLAIN and execution plans: what the planner decides and whyjunior
- Scan types: Seq, Index, Bitmap, Index-Onlymiddle
- Join algorithms and the row-estimate cascademiddle
- pg_statistic, ANALYZE, and production observabilitymiddle
- Extended statistics: fixing correlated-column estimate failuressenior
- Plan cache, cost-constant tuning, and planner internalssenior
- Production failure modes and plan stabilitysenior
- Connection pools: amortising the cost of a Postgres backendjunior
- PgBouncer session, transaction, and statement modesmiddle
- Pool sizing: the (cores × 2) + spindles formula and the two-layer stackmiddle
- Pool exhaustion and idle-in-transaction: the 3 AM failure modemiddle
- Migrating to transaction mode: rollout playbook and PgBouncer 1.21 prepared statementsmiddle
- The Postgres process model and why raising max_connections degrades throughputsenior
- Pooler landscape 2026, serverless connection storms, and the full failure-mode taxonomysenior
- ADD COLUMN: instant in PG 11+ vs rewrite in older Postgresjunior
- The lock-queue failure mode: why instant DDL can freeze the databasemiddle
- Safe DDL patterns: NOT VALID, CONCURRENTLY, and unsafe-op fixesmiddle
- Migration failure taxonomy and production disciplinesenior
- Shard-key selection: hash, range, list, and directory strategiesmiddle
- Co-location and Citus: the invariant that makes sharding usablemiddle
- The hot-shard failure mode: detection, isolation, and durable policymiddle
- Online resharding, 2PC, and the operational cost of shardingsenior
- The seven acts: from CREATE TABLE to Citusjunior
- Acts 1–3 in depth: schema, indexes, and planner statisticsmiddle
- Acts 4–6 in depth: MVCC bloat, connection pooling, and safe migrationsmiddle
- Act 7 in depth: sharding, co-location, and the seven-tier tradeoff cascademiddle
- Observability, anti-patterns, and production triagesenior
- Bits on the wirejunior
- Latency mathmiddle
- Bufferbloat and congestionsenior
- The physical frontiersenior
- Sequence numbers and connection statemiddle
- Flow control and congestion controlmiddle
- BBR, production observability, and beyond TCPsenior
- CDN: putting content next doorjunior
- Anycast and GeoDNS: routing to the nearest edgemiddle
- Tiered cache and Cache-Controlmiddle
- Vary header and cache keysmiddle
- Stale-while-revalidate and cache stampedesenior
- Edge workers and edge-side compositionsenior
- CDN operations and observabilitysenior
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket vs SSE vs long-polling: choosing the right transportmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Balancing algorithms: round-robin to power-of-two-choicesmiddle
- L4 vs L7 load balancing and client-IP preservationmiddle
- Health checks, connection draining, and slow startmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- QUIC streams and head-of-line blockingjunior
- Integrated handshake and 1-RTTmiddle
- Connection IDs and network migrationmiddle
- Loss detection and congestion controlmiddle
- 0-RTT resumption and packet encryptionsenior
- Deployment tradeoffs and CPU costsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- The twelve layers: one URL, seven actorsjunior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Critical render path and Core Web Vitalsmiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- What the three signals are: logs, metrics, and tracesjunior
- Metrics and cardinality: the cost model of a time-series databasemiddle
- Logs and volume: the cost model of structured loggingmiddle
- Traces and sampling: the cost model of distributed tracingmiddle
- Join keys and exemplars: making the three signals composemiddle
- Observability 2.0: wide events and the cost shiftsenior
- Failure modes and engineering practice: cardinality budgets, PII, and samplingsenior
- Why structured logs exist: the diary vs the spreadsheetjunior
- The production log schema: fields every line must carrymiddle
- Log levels and alert routingmiddle
- Sampling strategies and log costmiddle
- PII redaction and log injectionsenior
- Trace context propagation in logssenior
- OTel Logs Data Model and audit logs as a subsystemsenior
- OTel signals, Semantic Conventions, and the OTLP wire formatmiddle
- Auto-instrumentation and manual spans: the 80/20 of OTelmiddle
- The OTel Collector: receivers, processors, exporters, and deployment patternsmiddle
- Sampling strategies: head, tail, and parent-basedmiddle
- Vendor neutrality, eBPF instrumentation, the Operator, and browser/serverless OTelsenior
- Operating the OTel Collector: reliability, version skew, failure modes, and governancesenior
- RED and USE: two checklists, one triage disciplinejunior
- Instrumenting RED in Prometheus: counters, histograms, and cardinality disciplinemiddle
- USE on Linux: CPU, memory, disk, network, and PSImiddle
- Golden signals, dashboard layout, and service mesh auto-REDmiddle
- Cardinality as a cost driver: labels, PII, exemplars, and samplingmiddle
- Native histograms, SLO tie-in, and production failure patternsmiddle
- Choosing SLIs and SLO targets: ratios, not feelingsmiddle
- Multi-window multi-burn-rate alerting: why AND beats ORmiddle
- Error budget policy, latency SLOs, and composite journeysmiddle
- Iceberg SLIs, composite SLO math, and SLA vs SLOsenior
- Flame graphs: reading the picture that shows where time goesjunior
- Sampling vs instrumentation profiling: why 99 Hz wins in productionmiddle
- Profile types: CPU, memory, off-CPU, mutex — which one to reach formiddle
- Continuous profiling: always-on flame graphs with eBPF and trace-id correlationmiddle
- How flame graphs are built from samples, and the production workflows that use themmiddle
- Linux perf, eBPF internals, PGO, and the limits of samplingsenior
- Profiling in production: security, war stories, OTel profiles, and the infrastructure designsenior
- The debugging funnel: SLO → RED → trace → profilejunior
- OTel architecture: one SDK, four signals, one wire formatmiddle
- Cost discipline: keeping observability under 5% of infra spendmiddle
- Scale, security, and the ROI of observable systemssenior