An AI chatbot does not uncover a completed paragraph hidden inside the model. It constructs the response while you watch, usually one token at a time. The model supplies next-token probabilities; the inference engine turns those probabilities into a running service.
That service does more than call a model. It formats the request, converts text into token IDs, schedules work on accelerators, manages temporary memory, chooses each output token, and decides when generation should stop. These surrounding choices help explain why two products using similar models can feel noticeably different.
The prompt becomes a sequence of numbers
Before the model sees a question, the chat application may add role markers, system instructions, conversation history, or retrieved material. The visible sentence is therefore often only part of the input.
A tokenizer then breaks that input into tokens. A common word may fit in one token; a rare word may split into several pieces. Punctuation, spaces, and parts of words can have their own token IDs. Token count is not word count, which is why code, unusual spelling, and different languages can consume context space at different rates.
Those IDs are mapped to numerical vectors. Positional information preserves order, allowing the model to distinguish “the dog chased the robot” from “the robot chased the dog.” The underlying tokens are similar, but their relationships are not.
Training is over before inference begins
Training and inference use the same learned model in different ways. During training, an optimization process repeatedly adjusts the model’s weights to improve predictions. During ordinary inference, those weights are generally fixed. Your prompt changes the temporary context for this request; it does not automatically rewrite the base model.
This is the distinction behind the broader explanation of how inputs become predictions and decisions. The trained parameters contain distributed numerical patterns. Inference applies those patterns to a new input.
Serving a large model also requires physical preparation. Its weights must be available in GPU or accelerator memory. A model too large for one device may be split across several. Keeping the weights loaded avoids the enormous delay of loading the model from storage for every question.
Production inference keeps model weights in accelerator memory, often spreading a large model across several devices.
Prefill builds a working representation of the prompt
The first major phase is commonly called prefill. The model processes the known prompt through its transformer layers. Attention mechanisms calculate which earlier positions are useful while representing each token.
In simplified terms, each position produces query, key, and value vectors. A query describes what the current position is looking for. Keys help determine which earlier positions are relevant. Values carry information that can be combined according to those relevance scores.
Attention compares query and key representations so information from relevant earlier positions can shape the current token.
This is not a readable database lookup. The relationships are distributed across many layers and parameters. A causal language model also uses a mask that prevents a position from consulting future answer tokens that do not exist yet.
At the end of the pass, the model produces a score, often called a logit, for every token in its vocabulary. Normalizing those scores creates a probability distribution for the next token.
Decoding turns probabilities into one choice
The engine now has possibilities, not a final answer. A decoding strategy determines how one becomes the next token.
Greedy decoding always chooses the highest-probability option. It can be consistent, but longer output may become rigid or repetitive. Sampling allows several plausible tokens to have a chance. Temperature reshapes the distribution: lower values concentrate probability on the leaders, while higher values give less likely candidates more room. Other controls can restrict selection to a smaller likely set.
Lower temperature concentrates probability on leading candidates; higher temperature leaves more room for less likely tokens.
For each output step, the engine repeats a short sequence:
- Obtain probabilities for the next token.
- Choose one token using the configured decoding strategy.
- Append it to the working sequence.
- Continue unless a stopping condition has been reached.
The text appears continuous because this loop runs quickly, not because the entire paragraph was selected in advance.
The KV cache avoids rebuilding the whole past
Repeating the loop creates an obvious cost. Without an optimization, the engine would recalculate attention information for every earlier token at every step.
The common solution is the key-value cache, or KV cache. During prefill, the engine stores the key and value vectors produced by attention layers. For each new token, it can reuse the stored state and calculate the new addition instead of reconstructing the entire history.
The trade-off is memory. The cache grows with the sequence unless the model uses a limited attention window or another cache policy. A long conversation therefore consumes more than input space on paper; it also requires temporary serving memory while the request remains active.
The runtime shapes what the user experiences
An inference engine may batch several users’ requests, use lower-precision weights, split computation across devices, reuse common prompt prefixes, or let a smaller model propose tokens for a larger model to verify. These techniques affect time to first token, later token speed, context capacity, and the number of people the hardware can serve.
This is why the model name alone does not describe the experience. The physical cost also extends beyond training: the article on how AI training turns electricity into intelligence explains why heavily used inference can eventually dominate a system’s lifetime workload.
Generation stops when the model emits a special end token, reaches a configured length, matches a stopping condition, or is interrupted by the application. The tokenizer finally converts output IDs back into readable text.
The completed sequence still needs a separate check of its claims. Retrieval, tools and validation can add evidence around generation; next-token selection does not perform that job by itself. Our guide to confident hallucinations follows the gap between a plausible continuation and a supported answer.
Visual guide
How the pieces fit together
