In previous chapters we looked at how the prefill phase builds the KV substrate and how the decode phase generates text one token at a time by repeatedly reading from that substrate. We also saw that the quality of the KV substrate strongly influences how coherent and useful the final output becomes.
Even with a well-structured prompt and rich reference material, the model sometimes produces answers that feel direct but shallow or skip important logical steps. A simple and very effective technique can help with this: Chain-of-Thought prompting, commonly called CoT.
The Core Idea of Simple CoT
The primary goal of simple Chain-of-Thought is to generate an output that contains "thinking out loud" reasoning.
Instead of asking the model to jump straight from the prompt to a final answer, you explicitly instruct it to show its reasoning process step by step as part of the generated response.
The most common way to trigger this is by adding a short instruction such as:
“Think step by step.”
Or a more detailed version:
Compare different sorting algorithms. Think step by step about their time and space complexity, strengths, and weaknesses for various data scenarios. Then recommend the best algorithm for efficiently sorting a list of 1000 integers.How It Changes the Decode Phase
Without CoT, the model tends to compress its reasoning into the hidden state and move quickly toward a final conclusion. With CoT, the model is encouraged to externalize its reasoning — producing visible intermediate steps as tokens during the decode phase.
Those reasoning tokens become part of the growing KV substrate. Later tokens in the same generation can then attend to them. This often leads to:
- More structured and transparent reasoning
- Better handling of multi-step tasks
- Reduced tendency to take logical shortcuts
- More consistent final answers
Simple CoT as Caveman Debugging
A helpful way to think about simple Chain-of-Thought is that it functions as caveman debugging — the AI equivalent of rubber-duck debugging. Just as explaining your code line-by-line to a rubber duck (or adding printf() statements) forces you to slow down and spot flaws you missed when rushing, telling the model to “think step by step” externalizes its reasoning and often surfaces logical shortcuts.
In multi-agent orchestration, this becomes especially valuable: the visible reasoning trail turns opaque agent decisions into something you can read, audit, and debug. The orchestrator’s “thinking out loud” can reveal why it chose one routing over another, making failure diagnosis far easier.
However, like well-formatted printf() logs written by humans, the output can be deceptively smooth. The decode phase generates tokens autoregressively, so the visible steps sometimes act as a polished, post-hoc narrative rather than a faithful record of the model’s internal latent state. Much of the actual “navigation” happens in the hidden states — the high-dimensional guidance vectors in the KV substrate — which we still cannot inspect with true stack-trace granularity today.
As a result, CoT can occasionally produce longer, more confident-sounding reasoning that hides internal shortcuts or silent corrections. Treat the visible chain as a powerful but imperfect signal: excellent for caveman-level debugging and auditing, yet not a complete window into the underlying state machine.
Simple Before-and-After Example
Without CoT:
Explain how to implement a quick sort algorithm in JavaScript.With CoT:
Explain how to implement a quick sort algorithm in JavaScript. Think step by step.The CoT version typically produces a response that first walks through the reasoning process before delivering the final recommendation. This "thinking out loud" makes the output more reliable and easier to follow.
When Simple CoT Is Most Useful
Simple CoT tends to shine when:
- The task involves multiple logical steps (analysis → planning → implementation → verification)
- You have rich reference material in the prompt (such as a large coherent codebase)
- You want the reasoning to be transparent and auditable (with the important caveat that the visible trail is helpful but lossy)
Keep in mind that CoT increases the number of tokens generated and therefore adds some latency and cost. It is not a guarantee of correctness — the visible reasoning is a useful but smoothed reconstruction of the decode process — but it gives the model a better opportunity to use the information already present in the KV substrate more effectively.
Looking Ahead
Simple Chain-of-Thought is one of the easiest ways to encourage more deliberate reasoning during a single inference pass. In the following chapters we will explore other techniques for shaping output structure, managing long contexts, and balancing quality versus speed during inference.