Start from zero: what LLM integration actually is
Integrating an LLM means calling a probabilistic text model as a service and engineering around its limits — finite context window, per-token pricing, and outputs that can hallucinate. This is the map and the eight words the rest of the track assumes you already know.
You have typed a question into ChatGPT and got a remarkably good answer. Now your manager wants that capability inside your product — the company’s own API key, the company’s own prompts, the company’s own guardrails. Suddenly the chatbot magic dissolves into a stack of engineering decisions: How do you pass context? What happens when the model invents a fact? How much will it cost when a thousand users hit it at once? This lesson is the map before the climb: eight words the senior units use without stopping to define them, and one paragraph showing how they fit together.
The one idea behind LLM integration
A large language model is not a database you query for stored facts. It is a statistical engine trained on text: given the tokens it has seen so far, it predicts the probability distribution over the next token, samples from it, and repeats. That process — inference — runs on powerful hardware someone else operates. You reach it over HTTP. Every engineering challenge in this track flows from that one mechanical reality: the model is probabilistic, bounded, and metered.
“Integrating” an LLM means writing the code that sits between your application and that HTTP endpoint — shaping the input, handling the output, caching aggressively, measuring cost, and guarding against failure modes that do not exist in deterministic APIs.
The eight words the rest of the track assumes
The senior lessons that follow drop these terms without stopping to define them. Before you can reason about prompt caching, hallucination guards, or retrieval pipelines, you need these eight words to already feel like furniture — so when you encounter one in a live incident, you reach for the right tool instinctively. Here they are, one sentence each — what it is and why it exists.
| Word | What it is | Why it exists |
|---|---|---|
| Token | The atomic unit the model reads and writes — roughly a word fragment (≈¾ of an English word). | So the model can handle any text as a fixed-size vocabulary; pricing and limits are always stated in tokens, not characters. |
| Prompt | The text you send to the model as input — instructions, context, examples, and the user’s question combined. | So the model has everything it needs to generate a useful reply in a single stateless request. |
| Context window | The maximum number of tokens the model can see at once — prompt + reply combined. | So you know the hard ceiling on how much history and data you can pass; exceeding it truncates or errors. |
| Model | A specific trained checkpoint you call by name (e.g. gpt-4o, claude-3-5-sonnet). | So you can pin your app to a known capability and cost profile and update deliberately. |
| Inference | Running the model on a prompt to produce output — the compute step that happens on the provider’s hardware. | So you understand what you are actually paying for: GPU time proportional to input + output tokens. |
| Embedding | A fixed-length vector of numbers that encodes the meaning of a piece of text. | So you can do semantic search and retrieval in a vector database — the backbone of RAG. |
| Hallucination | A plausible-sounding output the model generates that is factually wrong or entirely fabricated. | So you build verification and grounding steps; trusting raw model output in high-stakes paths is a design error. |
| Temperature | A number (0–2) that scales how peaked or flat the token probability distribution is before sampling. | So you control the trade-off between determinism (0 = most likely token) and creativity (higher = more varied). |
How they fit together
Read in order, the words tell one story: you write a prompt — a block of text made of tokens — and send it to a named model within its context window. The model runs inference: it samples the next token from a probability distribution shaped by temperature, repeats until it emits a stop token, and streams the result back. You pay per input and output token. If the model lacks grounding it may hallucinate, which is why production systems retrieve facts as embeddings before calling the model. That single paragraph is the entire track in miniature — every later unit zooms into one of those steps.
▸Why this works
Why not just use the chat UI and copy-paste the answers? Because your users need answers about your data, your business rules, your user’s state — none of which the public chatbot knows. Calling the API directly lets you inject that context into the prompt, stream responses into your own UI, cache aggressively to cut cost, and add guardrails the public UI does not expose. The engineering is the gap between “impressive demo” and “reliable product.”
You do not need to memorise this
Two honest notes before the climb. First: nobody holds all eight words at once on day one — you will meet each in depth in its own unit and it will stick then. This page is a coat-hook, not a test. Second: not every LLM integration uses every word. A simple summariser might never touch embeddings. The senior track teaches the full picture because that is what production at scale demands — but “start with the simplest prompt, add pieces when the pain shows up” is itself the senior instinct.
Why can an LLM return a confident but wrong answer?
Order the journey from your code to a streamed reply reaching the user:
- 1 Assemble a prompt (tokens) from system instructions, retrieved context, and the user's message
- 2 Send the prompt to the model endpoint — it must fit within the context window
- 3 The model runs inference, sampling tokens from a probability distribution shaped by temperature
- 4 Stream the output tokens back, validate for hallucination, and render to the user
- 01In one breath, what is LLM integration and what is its core engineering challenge?
- 02Trace the path from a user's question to a reply, naming each concept.
LLM integration is one idea with a lot of engineering hung off it: call a probabilistic text model as a service and build the scaffolding that makes it reliable, fast, and affordable in a real product. The model half is inference — predicting the next token from a probability distribution, bounded by a context window, billed per token. The engineering half is everything else: assembling a prompt that fits the window, caching repeated prefixes to cut cost, retrieving embeddings so the model has grounding instead of guessing, streaming tokens back without blocking the UI, setting temperature to balance creativity against determinism, and running evals so hallucinations surface in CI rather than in front of users. You do not need to hold all eight words at once — each gets its own unit ahead. Now when you see a colleague say “the model hallucinated” or “we’re hitting context limits,” you know exactly which lever is broken and where in the pipeline to look — that is the shift these eight words give you. Next: Unit 01, prompt caching — how to stop paying for the same tokens twice.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.