How an AI Answers: Context, Tokens, and Chain of Thought

Jul 25, 26

When you send a prompt to a language model, it can feel like the reply arrives whole, the way a person blurts out an answer. It does not. A model builds its reply the way you would lay bricks: one small piece at a time, left to right, each piece decided only after the one before it is already down. This post walks the whole path - from the moment your message is assembled into context to the moment the last word is chosen - and shows where the much-talked-about chain of thought actually lives.

It connects to something in the history post: the reason models are uneven at ASCII art is buried in step two below. If you have ever wondered why an AI can write a sonnet but botch the alignment of a text-drawn cat, this is the mechanism that explains it.

The shape of the whole thing

[system prompt] + [chat history] + [your prompt] + [tools / documents]
        │
        ▼
   tokenization  →  model (transformer)  →  probability of the next token
        ▲                                             │
        └──────────────  repeat, one token at a time  ┘
                              │  (until a stop signal)
                              ▼
                          the reply

Everything below is just that loop, explained.

Step 1: Assembling the context

The model never sees only your latest sentence. Before anything runs, your message is merged into a single sequence of text, tagged by role:

All of this has to fit inside the context window, a fixed budget of tokens. If the conversation grows past it, older material gets summarized or dropped. The quality of what lands in this window is the single biggest lever on the answer - a point we will return to at the end.

Step 2: Tokenization

The model does not read letters. The assembled text is chopped into tokens - chunks that are often whole words but sometimes fragments (token, iz, ation)

This is also the quiet villain of AI-made ASCII art. A run of spaces or a row of box characters gets split into tokens in ways that have nothing to do with visual columns, so a character the model “wants” on column 12 of line 5 has no reliable relationship to the token stream it is producing. The model is composing a two-dimensional picture through a one-dimensional straw. That is why converters and human artists still beat models at fresh, well-aligned ASCII.

Step 3: Predicting the next token

Now the transformer reads all the tokens at once. Through a mechanism called attention, it weighs which earlier tokens matter for what comes next, and produces a probability distribution over the next token - not a single answer, but a ranked field of candidates:

next token →   " the" 41%   " a" 12%   " your" 7%   ...

There is no answer stored anywhere. There is only “given everything so far, what is likely to come next.”

Step 4: Autoregressive generation

Here is the heart of it. The model emits one token, appends that token to its own input, and predicts again. Then again. The reply is built strictly left to right, each token conditioned on all the tokens - yours and its own - that came before it.

The consequence is worth sitting with: the model does not see its finished answer before writing it. It cannot revise line 9 to match line 5, because when it writes line 5 it has not thought about line 9 yet. It is always working at the leading edge of a sentence it is still inventing.

Step 5: Sampling

The distribution from step 3 still has to become one concrete choice. That is sampling, and a couple of dials control it:

Turn temperature to zero and the model becomes (nearly) deterministic, always taking the top candidate.

Step 6: Stopping

The loop ends when the model emits a special stop token, hits the max_tokens limit, or runs into a stop sequence the caller specified. Then the tokens are decoded back into the text you read.

Where chain of thought lives

Chain of thought (CoT) is not a separate machine bolted onto the model. It is step 4, used deliberately - reasoning tokens generated before the final answer:

[context] → model → [ thinking tokens ... ] → [ final answer tokens ]
                     └──── chain of thought ────┘

The model writes out its reasoning first - “first this, which implies that, so the answer is” - and only then writes the answer.

Why place it there, and why does it help? Because generation is autoregressive, every token the model writes is fed back as context for the next one. Forced to answer immediately, the model has to complete an entire multi-step argument inside a single prediction, which is hard. Allowed to think first, it works on paper: it writes intermediate results down as tokens and then attends to them as it goes. The scratchpad is real, and it measurably raises accuracy on layered problems - arithmetic, logic, planning.

There are two flavors you will meet:

  1. Prompted CoT - you ask it to “think step by step,” and the reasoning shows up inside the visible reply.
  2. Built-in reasoning - modern reasoning models produce a distinct thinking block before the answer, often returned separately from the final content. The raw chain is frequently summarized or hidden, so you see a tidy summary or just the conclusion rather than the full internal trace.

Two honest caveats. CoT spends tokens and context - the more it thinks, the slower and more expensive the turn. And a chain of thought that reads clean can still reach a wrong conclusion; plausible-looking reasoning is not proof, which is exactly why answers that matter still deserve to be checked.

The layers on top

Two more structures wrap the same loop in real systems:

Why any of this matters to you

Strip it all down and a model is a next-token predictor conditioned on its context. That single fact is the most useful thing to internalize, because it tells you where your leverage is: the context. Clear instructions, a concrete example, the right documents in the window - these bend the probability field toward the answer you want. Vague input bends it toward vague output. The old phrase fits perfectly: garbage in, garbage out.

None of this is unique to answering questions, either. It is the same left-to-right, one-piece-at-a-time process that makes a model draw a candle out of words, or struggle to keep a text cat’s whiskers aligned. Understanding the loop is the quickest way to get more out of the thing on the other side of the prompt - and, if you like, to draw with it in the editor.