Chain of Thought Prompting in Production

Why Naive CoT Fails in Production

Asking the model to "think step by step" works in a chat UI. In production it breaks in two ways:

1. The thinking and the output are mixed into one text block. You cannot parse the output reliably.

2. There is no validation. If the reasoning is wrong, the output is wrong and you have no signal.

The fix is to separate thinking from output and validate both independently.

The Two-Block Pattern

1
## Thinking
2
Before answering, reason through the problem. Write your reasoning inside <thinking> tags.
3
4
## Output
5
After your reasoning, return the final answer inside <output> tags.
6
Return only the answer in the output block. No reasoning, no caveats.
7
8
## Format
9
<thinking>
10
[your step-by-step reasoning]
11
</thinking>
12
<output>
13
[final answer only]
14
</output>

Parsing the Response

1
function parseCoT(response: string) {
2
  const thinking = response.match(/<thinking>([\s\S]*?)<\/thinking>/)?.[1]?.trim();
3
  const output = response.match(/<output>([\s\S]*?)<\/output>/)?.[1]?.trim();
4
5
  if (!thinking || !output) {
6
    throw new Error("Missing thinking or output block");
7
  }
8
9
  return { thinking, output };
10
}

If either block is missing, the parse throws. This catches malformed responses before they propagate downstream.

Validating the Thinking

For high-stakes tasks, log the thinking block and run a second validation pass:

1
Given this reasoning:
2
{{thinking}}
3
4
And this conclusion:
5
{{output}}
6
7
Does the conclusion follow from the reasoning? Return JSON: {"valid": true|false, "issue": "..."}

This is a second LLM call, which adds cost. Use it only for tasks where a wrong answer has real consequences.

When to Use CoT

Chain of thought helps with:

• Multi-step math or logic
• Tasks requiring information from multiple parts of a long input
• Classification where the decision criteria are complex

It does not help with:

• Simple retrieval (what is the capital of France)
• Format conversion (convert this JSON to CSV)
• Tasks where the answer is in the first paragraph of the context

Adding CoT to everything adds tokens without adding reliability. Use it where reasoning complexity is actually the bottleneck.