Databases
Partial, expression, and covering indexes
A tasks table has 100M rows. 80% have status = 'done' and are never queried by the dashboard. The full B-tree on (workspace_id, created_at) is 2 GB and every insert touches it. A partial index WHERE status IN ('open','in_progress') is 400 MB, 5× cheaper to maintain, and the dashboard queries run just as fast — because it only indexes the rows the dashboard actually needs.
Partial indexes
A partial index is a B-tree (or any index type) built only over rows matching a WHERE predicate.
CREATE INDEX ON orders(user_id) WHERE status = 'pending';This indexes only pending orders — typically a small fraction of the table. The result: smaller index, faster lookup on the hot subset, cheaper write overhead (the index is only touched when status = 'pending').
For the planner to use a partial index, the query’s WHERE clause must logically imply the index’s predicate. A query WHERE user_id = 42 AND status = 'pending' can use the index. A query WHERE user_id = 42 alone cannot (it would need to match all statuses, but the index only covers pending rows).
Production use cases:
- Soft-delete pattern:
WHERE deleted_at IS NULL— only active rows are indexed; historical deleted rows are excluded. - Hot-subset filters: a task table where 95% of rows are
status = 'done'— index only'open'and'in_progress'. - Feature flags:
WHERE feature_x_enabled = true— index only rows where the flag is active.
Partial unique indexes are a powerful variant. CREATE UNIQUE INDEX ON users (email) WHERE deleted_at IS NULL enforces “email is unique among non-deleted users.” This lets you soft-delete a user and later re-register the same email — impossible with a simple UNIQUE constraint on the email column.
Expression indexes
An expression index indexes a function of one or more columns, not the columns themselves.
CREATE INDEX ON users (LOWER(email));This lets WHERE LOWER(email) = 'alice@x.com' use the index. Without it, LOWER(email) is evaluated on every row, defeating any index on the email column.
Common patterns:
- Case-insensitive search:
LOWER(email),UPPER(code). - Date truncation:
DATE_TRUNC('day', created_at)for group-by-day queries. - JSON path extraction:
(data->>'event_type')— indexes one field inside a JSONB column. - Computed values:
COALESCE(first_name, '') || ' ' || COALESCE(last_name, '')for name-search.
The query must use the exact expression for the planner to recognize the index. WHERE LOWER(email) = ? uses the index. WHERE email = ? does not (unless the email is already lowercase and a regular index exists).
Covering indexes with INCLUDE
A regular B-tree index stores key columns in both internal nodes and leaf pages. The INCLUDE clause (Postgres 11+) adds extra columns to the leaf pages only — they are not part of the sort key, but they are stored in the leaf, available for the query.
When a query’s SELECT projection needs only columns that are present in the index (either as key columns or INCLUDE columns), Postgres can answer the query without fetching the row from the heap at all — this is an index-only scan. It is 10–100× faster than an index scan for hot reads because it eliminates the heap fetch entirely.
CREATE INDEX CONCURRENTLY idx_orders_workspace_pending_recent
ON orders (workspace_id, created_at DESC)
WHERE status = 'pending'
INCLUDE (id, total_cents);This index:
- Filters by
workspace_id(leading column, tenant scope). - Sorts by
created_at DESC(serves theORDER BYwithout a Sort step). - Partial
WHERE status = 'pending'— only covers ~20% of the table. INCLUDE (id, total_cents)— the typical dashboard projection is covered, no heap fetch needed.
- B-tree size, 8-byte key, 100M rows
- ~1 GB
- B-tree depth, 100M rows
- ~4 levels
- B-tree lookup time
- ~5-50 μs
- Index build (CREATE INDEX CONCURRENTLY)
- ~10-30 min / 100M rows
- Index write overhead per INSERT/UPDATE
- ~5-50 μs per index touched
- Typical index overhead vs table size
- ~10-30% per index
- Index-only scan vs index scan speedup
- ~10-100x on hot reads
- Partial index size, 5%-matching predicate
- ~5% of full B-tree
| Modifier | What it does | When to use |
|---|---|---|
| Partial (WHERE) | Index only rows matching predicate | Hot subset is <20% of table; or unique constraint on non-deleted rows |
| Expression | Index a computed value | Query filters on LOWER(col), DATE_TRUNC, JSON field, or other function |
| INCLUDE | Store extra columns in leaf pages without affecting sort | Query projects columns beyond the key; enables index-only scan |
Why this works
Why are partial indexes underused in practice? Because they require the developer to know the predicate before writing the query. Many teams add indexes by looking at slow queries after the fact, not by thinking about data distribution at schema design time. The discipline: when a new hot query always filters by a specific status, flag, or soft-delete condition, build the partial index at the same time the feature ships. The cost is near-zero at feature time; the benefit is 10–20× cheaper index maintenance across the table’s lifetime.
Pick the right index for a hot dashboard query
1/3A query: WHERE LOWER(email) = 'alice@x.com'. Which index does the planner use?
A table has 10M rows, 95% with status = 'done'. A dashboard queries only WHERE status = 'open'. Which is the best index strategy?
- 01Why can a partial unique index enforce a constraint that a plain UNIQUE constraint on a column cannot?
- 02What two conditions must hold for Postgres to use an index-only scan?
- 03A query filters on DATE_TRUNC('day', created_at) = '2026-01-15'. There is a plain index on created_at. Does the planner use it?
Three modifiers turn a plain B-tree into a targeted tool. Partial indexes (WHERE predicate) restrict coverage to matching rows — 5% coverage means 5% of full-index size and write cost. Expression indexes cover computed values like LOWER(email) or (data->>'field') — the query must use the exact same expression. INCLUDE columns (Postgres 11+) store projection-only columns in the leaf pages without affecting the sort key, enabling index-only scans that never touch the heap. Used together — partial + composite + INCLUDE — a well-designed index can cover an entire hot dashboard query with near-zero heap I/O.
appears again in174
- The journey of a request: seven stops from socket to responsejunior
- Accept and parse: from kernel queue to a typed requestmiddle
- Routing and middleware: choosing what runs, and in what ordermiddle
- Handler and response: from business logic to bytes on the wiremiddle
- Streaming and backpressure: when the client reads slower than you writesenior
- Timeouts and tail latency: budgets, deadlines, and the fan-out trapsenior
- Middleware and DI: the two patterns that shape every backendjunior
- Writing middleware: signatures, next(), and the three framework modelsmiddle
- Inversion of control: how dependencies reach a classmiddle
- DI scopes and lifecycles: singleton, request, transientmiddle
- DI as a testing seam: fakes, mocks, and the boundary that matterssenior
- DI containers in production: resolution graphs, circular deps, and when not tosenior
- Blocking vs non-blocking I/O: two ways to waitjunior
- The event loop: one thread, ordered phasesmiddle
- What blocks the loop: CPU work and sync callsmiddle
- Offloading CPU work: worker threads and the libuv poolmiddle
- Backpressure and bounded concurrencysenior
- Throughput under load: tail latency and saturationsenior
- Why pool: the cost of creating a connectionjunior
- Pool sizing: why bigger is not fastermiddle
- Acquisition and timeouts: the wait queue is the real latency dialmiddle
- Retry strategies: backoff, jitter, and thundering herdmiddle
- Observability, production failures, and global-scale designsenior
- Tasks, microtasks, and scheduler.yield()middle
- Timer accuracy, throttling, and idle workmiddle
- Node.js event loop: phases, nextTick, and loop lagsenior
- Rendering strategies: SSG, SSR, ISR, streaming, and hydrationjunior
- SSG, SSR, ISR, streaming, and RSC — how each worksmiddle
- Hydration cost: selective, progressive, islands, resumabilitymiddle
- Core Web Vitals: what LCP, INP, and CLS measurejunior
- LCP: four phases, one dominant costmiddle
- INP: input delay, processing, presentationmiddle
- Lab vs field: why the two disagree and how to use eachmiddle
- Metric tradeoffs, RUM attribution, and the CI+field loopsenior
- The full picture: URL to LCP to INP as a relay racejunior
- Eight layers traced: from the service worker to the second navigationmiddle
- Five canonical breaks: where production reliably diessenior
- The three-track method: reading traces and building a monitored systemsenior
- Bits on the wirejunior
- Latency mathmiddle
- Bufferbloat and congestionsenior
- The physical frontiersenior
- Sequence numbers and connection statemiddle
- Flow control and congestion controlmiddle
- BBR, production observability, and beyond TCPsenior
- CDN: putting content next doorjunior
- Anycast and GeoDNS: routing to the nearest edgemiddle
- Tiered cache and Cache-Controlmiddle
- Vary header and cache keysmiddle
- Stale-while-revalidate and cache stampedesenior
- Edge workers and edge-side compositionsenior
- CDN operations and observabilitysenior
- WebSocket: the HTTP upgrade handshakejunior
- WebSocket vs SSE vs long-polling: choosing the right transportmiddle
- WebSocket backpressure: when clients can''''t keep upmiddle
- Reconnection: jittered backoff, thundering herd, message resumptionsenior
- WebSocket at scale: HTTP/2 multiplexing, permessage-deflate, C10Msenior
- WebSocket in production: proxies, security, and distributed architecturesenior
- What reverse proxies dojunior
- Balancing algorithms: round-robin to power-of-two-choicesmiddle
- L4 vs L7 load balancing and client-IP preservationmiddle
- Health checks, connection draining, and slow startmiddle
- Retry storms, circuit breakers, and load sheddingsenior
- Resilient LB architecture: anycast, zone-aware routing, and observabilitysenior
- Why QUIC and not TCP+TLSjunior
- QUIC streams and head-of-line blockingjunior
- Integrated handshake and 1-RTTmiddle
- Connection IDs and network migrationmiddle
- Loss detection and congestion controlmiddle
- 0-RTT resumption and packet encryptionsenior
- Deployment tradeoffs and CPU costsenior
- DDoS: what it is and why it worksjunior
- Amplification attacks and state exhaustionmiddle
- Rate limiting: algorithms and architecturemiddle
- WAFs, firewalls, mTLS, and HSTSmiddle
- DNS cache poisoning and BGP hijackingsenior
- Defense-in-depth architecture and attack economicssenior
- The twelve layers: one URL, seven actorsjunior
- DNS, TCP, TLS in sequence: where the milliseconds gomiddle
- Critical render path and Core Web Vitalsmiddle
- Proxy intercepts and security gates: rate limiters, WAF, mTLSmiddle
- Alternate paths: QUIC 0-RTT, WebSocket upgrade, connection migrationmiddle
- Observability: distributed traces, USE/RED, and samplingsenior
- Resilience: cascading retries, circuit breakers, and error budgetssenior
- What the three signals are: logs, metrics, and tracesjunior
- Metrics and cardinality: the cost model of a time-series databasemiddle
- Logs and volume: the cost model of structured loggingmiddle
- Traces and sampling: the cost model of distributed tracingmiddle
- Join keys and exemplars: making the three signals composemiddle
- Observability 2.0: wide events and the cost shiftsenior
- Failure modes and engineering practice: cardinality budgets, PII, and samplingsenior
- Why structured logs exist: the diary vs the spreadsheetjunior
- The production log schema: fields every line must carrymiddle
- Log levels and alert routingmiddle
- Sampling strategies and log costmiddle
- PII redaction and log injectionsenior
- Trace context propagation in logssenior
- OTel Logs Data Model and audit logs as a subsystemsenior
- OTel signals, Semantic Conventions, and the OTLP wire formatmiddle
- Auto-instrumentation and manual spans: the 80/20 of OTelmiddle
- The OTel Collector: receivers, processors, exporters, and deployment patternsmiddle
- Sampling strategies: head, tail, and parent-basedmiddle
- Vendor neutrality, eBPF instrumentation, the Operator, and browser/serverless OTelsenior
- Operating the OTel Collector: reliability, version skew, failure modes, and governancesenior
- RED and USE: two checklists, one triage disciplinejunior
- Instrumenting RED in Prometheus: counters, histograms, and cardinality disciplinemiddle
- USE on Linux: CPU, memory, disk, network, and PSImiddle
- Golden signals, dashboard layout, and service mesh auto-REDmiddle
- Cardinality as a cost driver: labels, PII, exemplars, and samplingmiddle
- Native histograms, SLO tie-in, and production failure patternsmiddle
- Choosing SLIs and SLO targets: ratios, not feelingsmiddle
- Multi-window multi-burn-rate alerting: why AND beats ORmiddle
- Error budget policy, latency SLOs, and composite journeysmiddle
- Iceberg SLIs, composite SLO math, and SLA vs SLOsenior
- Flame graphs: reading the picture that shows where time goesjunior
- Sampling vs instrumentation profiling: why 99 Hz wins in productionmiddle
- Profile types: CPU, memory, off-CPU, mutex — which one to reach formiddle
- Continuous profiling: always-on flame graphs with eBPF and trace-id correlationmiddle
- How flame graphs are built from samples, and the production workflows that use themmiddle
- Linux perf, eBPF internals, PGO, and the limits of samplingsenior
- Profiling in production: security, war stories, OTel profiles, and the infrastructure designsenior
- The debugging funnel: SLO → RED → trace → profilejunior
- OTel architecture: one SDK, four signals, one wire formatmiddle
- Cost discipline: keeping observability under 5% of infra spendmiddle
- Scale, security, and the ROI of observable systemssenior
- Why profile first: measure where time actually goesjunior
- Amdahl''''s law and self-time: the ceiling on every speedup you can shipmiddle
- The measurement loop: microbench, macrobench, prod profile, observer effectmiddle
- Reading flame graphs: shapes, per-language profilers, and the 60-second scanmiddle
- Statistical baselines: why one run is not a measurementmiddle
- Profiler history and microbenchmark pitfalls: Knuth to GWPsenior
- Hardware counters, cold-start profiles, and profile securitysenior
- Continuous profiling at scale: costs, CI gates, trace correlation, and anti-patternssenior
- What makes a hot path: symptom vs causejunior
- Five shapes of hotspot: CPU, alloc, cache, lock, syscallmiddle
- Reading parent and child chains: where to apply the fixmiddle
- JIT deopt, the fix-and-verify loop, and PR-time profilingmiddle
- Hardware counters and Intel TMA: sub-category diagnosissenior
- False sharing and native-bridge hot pathssenior
- Hot paths in production: security, tail latency, and tooling lineagesenior
- Memory hierarchy: why the same O(N) loop can be 17x slowerjunior
- Row-major vs column-major: access order and the 9x gapjunior
- Branch prediction and branchless codemiddle
- Hardware prefetcher, TLB, and memory-level parallelismsenior
- GC basics: what the runtime taxes you forjunior
- GC algorithms: generational, concurrent, and per-runtimemiddle
- GC tradeoffs: pause, throughput, heap — and object poolingmiddle
- GC tuning: pacing, heap shape, and allocation observabilitymiddle
- GC internals: tri-color invariant, write barriers, and per-runtime deep-divessenior
- GC in production: observability, security, edge cases, and fleet governancesenior
- N+1: one logical operation, many round-tripsjunior
- Fix families: JOIN, IN, preload, and DataLoadermiddle
- Detecting N+1: query logs, APM traces, and CI gatesmiddle
- DataLoader: batching across resolver treesmiddle
- Cross-protocol N+1: HTTP fan-out and Redis MGETmiddle
- N+1 at scale: pool exhaustion, plan changes, and denormalisationsenior
- Batching: amortize fixed cost per operationjunior
- The batching window: size and wait timemiddle
- Batching in Kafka and Postgresmiddle
- io_uring and observability of batchingmiddle
- From Nagle to io_uring: evolution of batchingmiddle
- Backpressure, failure isolation, and batch security in productionsenior
- What a bundle actually costs: download, parse, compile, executejunior
- Core Web Vitals: LCP, INP, and CLSmiddle
- Code splitting: route-level, component-level, vendor splittingmiddle
- Tree shaking and compression: removing what you don''''t usemiddle
- Third-party scripts: the silent budget killermiddle
- CI enforcement and RUM: making budgets stickmiddle
- V8 JIT pipeline, HTTP priorities, and bundle securitysenior
- The performance loop: discipline, not a projectjunior
- Classify and fix: matching bottleneck families to remediesmiddle
- Observability stack and CI gates: catching regressions before they shipmiddle
- Incident to enforcement: SLO burn to verified fix in 35 minutesmiddle
- Culture, economics, and org-scale performancesenior