AI / LLM Integration
RAG architecture: multiple-choice review
Crux Multiple-choice synthesis across the RAG unit — chunking, embedding cost, wide-then-narrow retrieval, lost-in-the-middle, and the confident-hallucination failure mode.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole pipeline. None is a definition to recite — each is a decision you make while a RAG system is silently returning confident, wrong answers in production.
Goal
Confirm you can connect chunking, embedding cost, two-stage retrieval, context assembly, and the retrieval-driven failure mode into one diagnosis — the synthesis the unit built toward.
Quiz
Completed
A support bot confidently reports a specific but wrong Q3 churn number; the answer survived review for a month. Where does the fault most often lie, and why does it stay invisible?
Heads-up The generator can only reason over the chunks retrieval hands it. The right chunk was never in the prompt, so a bigger model just hallucinates more fluently over the same missing evidence.
Heads-up A miss means the answer-bearing chunk was never fetched at all; window size can't include text retrieval never returned.
Heads-up The defining RAG failure is not a crash — it produces a plausible answer with no error, which is exactly why it survives review for weeks.
Quiz
Completed
A policy doc states a rule whose exception lives one paragraph later. Queries about the exception return half-truths. Which chunking change is the right first move?
Heads-up Smaller chunks embed precisely but fragment spanning rules even worse — you'd split the rule and its exception across even more boundaries, deepening the half-truth problem.
Heads-up A bigger net can help recall but doesn't fix a fact chunking severed; it floods context with noise and invites lost-in-the-middle while the seam problem remains.
Heads-up Embedding accuracy isn't the issue — the answer-bearing text was physically cut in two at index time. No embedding model recovers a fact that isn't whole in any single chunk.
Quiz
Completed
Index search p99 is too high at 3072 dims and storage cost is climbing. What's the senior trade before changing infrastructure?
Heads-up Matryoshka-trained embeddings are explicitly truncatable — you can drop dimensions and keep most of the accuracy, cutting both storage and search latency with no extra hardware.
Heads-up Exact scan removes ANN's millisecond speedup; at million-scale it makes latency far worse. The lever here is vector size, not abandoning the ANN index.
Heads-up ANN search compares the query against the index regardless of k; reducing k trims candidates returned, not the per-query search cost driven by dimensionality.
Quiz
Completed
Why is 'retrieve wide (k=20–50) then rerank narrow to 3–8 with a cross-encoder' better than a single embedding top-3?
Heads-up Cross-encoders are slow per item precisely because they jointly encode query+chunk — that's why you only run them on the shortlist from stage one, never the full corpus.
Heads-up Top-3 with no rerank is brittle: one ANN miss or boundary split and the answer isn't in the prompt. The two-stage pattern decouples recall (stage 1) from precision (stage 2).
Heads-up It does the opposite — it narrows 20–50 candidates down to 3–8, which cuts tokens, cost, and lost-in-the-middle while raising precision.
Quiz
Completed
You have a 128k-token window, so you stuff all 40 retrieved chunks in retrieval order and put the decisive one near the middle. What does the literature predict?
Heads-up Empirically attention is U-shaped: the 'lost in the middle' effect persists even in long-context models, so position within the window matters.
Heads-up It's the reverse: the middle is the least-attended region. Rerank, keep few, and place the strongest evidence at the very start or end.
Heads-up A bigger window is not a substitute for reranking; stuffing irrelevant chunks still buries the decisive one in the low-attention middle and costs tokens.
Quiz
Completed
A RAG bot that worked last week now returns last quarter's policy with no code change and no error. Most likely cause, and the structural fix?
Heads-up A frozen generator wouldn't flip a previously-correct answer to a specific older value — that pattern is the index serving stale embeddings, not a model change.
Heads-up The query embedding didn't change; the indexed document embeddings are stale relative to the updated source. The fix is re-indexing the corpus, not re-embedding queries.
Heads-up If the new policy was never re-indexed, no value of k retrieves it — it isn't in the index. Freshness is the missing piece, not breadth.
Recap
The unit’s through-line is one diagnosis: retrieval, not generation, is where production RAG fails. Chunking sets the ceiling (size to the answer-bearing unit, overlap so seams survive); embedding dimensionality is a real cost lever you can truncate; the recall-vs-precision split is solved by retrieve-wide-then-rerank-narrow; context ordering must dodge lost-in-the-middle by placing best evidence at the edges; and stale or poisoned indexes turn a silent miss into a confident wrong answer — which you defend against with a score gate, freshness, and an instruction to answer only from context or say “I don’t know.”