Data Engineering
Full-text search: multiple-choice review
Crux Multiple-choice synthesis across the search unit — inverted index, analyzer parity, BM25 tuning, GIN vs GiST, near-real-time refresh, and zero-downtime reindex.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. Each one mirrors a call you make designing or debugging real search — not a definition to recite, but a tradeoff to weigh against the corpus, the write rate, and the SLO.
Goal
Confirm you can connect the inverted index, the analysis pipeline, BM25 relevance, the Postgres-vs-engine decision, and the operational traps (refresh latency, reindex migrations) into one model of how search actually behaves.
Quiz
Completed
A query for 'running' returns zero results with no error in the logs. Documents were indexed with a stemming analyzer; the query path does not analyze the term the same way. What is the most likely root cause?
Heads-up Bloat slows lookups; it never turns a correctly-written token into a non-match. Zero results means nothing matched, which is a tokenization problem, not an index-health one.
Heads-up k1 only reshapes the ranking of documents that already matched. It cannot turn a match into a non-match — zero results is a matching failure, not a scoring one.
Heads-up Refresh latency causes a brief delay for freshly-written docs, not a permanent zero for already-indexed docs that a differently-phrased query would find.
Quiz
Completed
Why did modern engines move from raw TF-IDF to BM25 for term-frequency scoring?
Heads-up Speed is not the motivation; both are cheap once the posting lists are merged. BM25 is a relevance refinement, not a performance one.
Heads-up BM25 scores the candidates the inverted index already produced. It ranks the index's output; it does not replace the index.
Heads-up Stemming and tokenization happen before scoring. BM25 scores whatever tokens analysis produced — the two stages are independent.
Quiz
Completed
A SaaS app on Postgres needs search over ~2M support tickets: keyword match, basic relevance, modest writes, low ops budget. What is the right default?
Heads-up Over-engineered for these constraints. ES earns its keep on faceting, fuzzy matching, tunable BM25, or scale-out — none required here — at the cost of a cluster to operate and a dual-write consistency problem.
Heads-up A leading-wildcard LIKE cannot use a B-tree, so you still scan all 2M rows, with no stemming, no case-folding, and no relevance. This is the exact problem full-text search replaces.
Heads-up GiST shines under constant writes or hard disk limits; here reads dominate and writes are modest, which is GIN's sweet spot — GIN looks up roughly 3x faster, and GiST's lossiness forces table rechecks.
Quiz
Completed
You picked GIN for a tsvector index. Months later, on a now write-heavy table, search latency degrades intermittently with no query change. What is the most likely mechanism?
Heads-up k1 and b are static configuration; they do not drift with corpus size, and Postgres ts_rank does not use BM25 by default anyway. The symptom is write-driven index health, not scoring.
Heads-up More candidates add some ranking cost, but intermittent degradation tied to write volume points squarely at GIN's pending-list flush and bloat, not at ranking arithmetic.
Heads-up An analyzer change would cause wrong or empty results, not intermittent latency that tracks write volume. This is an index-maintenance failure mode, not a tokenization one.
Quiz
Completed
An Elasticsearch service writes a document and immediately reads it back by search, intermittently getting a miss. The team assumed write-then-read consistency. What is happening?
Heads-up An intermittent miss that resolves within ~1s is the refresh window, not a drop. A dropped write would never become searchable; this one does, a beat later.
Heads-up Ranking position cannot make a document briefly invisible and then visible with no data change. The visible-then-not timing is the refresh interval signature.
Heads-up GIN is a Postgres structure; Elasticsearch uses Lucene segments. The behaviour described is Lucene's refresh-to-searchable latency, by design.
Quiz
Completed
You need to fix a stemming bug by changing an existing field's analyzer across 200M documents in Elasticsearch. What is the correct migration, and why?
Heads-up Lucene cannot change the analyzer or type of an existing field — the committed tokens are immutable. There is no lazy re-tokenize; you must build a new index.
Heads-up Dropping the live index means clients see empty or failing search until reindex completes — hours at 200M docs. Designing behind a read alias avoids the outage entirely.
Heads-up refresh_interval controls write-to-searchable latency, not analyzer changes. It has nothing to do with re-tokenizing already-committed documents.
Recap
The through-line: matching is tokens, not strings, so the same analyzer must run at index and query time or you get silent zero results; BM25 ranks the candidates the inverted index produces by saturating term frequency and normalizing length; Postgres GIN is the right default until you need facets, fuzziness, BM25 tuning, or scale, at which point GIN’s write-bloat versus GiST’s lossiness is the inside choice; and the operational traps — near-real-time refresh latency and the immutable-tokens reindex — are why you design behind a read alias from day one.