Introduction

How to Use This Book

This is a self-study guide that Sunday AI, a study group, compiled to read papers on agentic AI for an hour each week. It is easier going if you have a grounding in probability, statistics, linear algebra and basic programming. If natural language processing and reinforcement learning are new to you, or it has been a while, you can review them first in Chapter 0 (preliminaries). You are welcome to run a study group the same way elsewhere, or to read it alone.

1. How a meeting runs

Each meeting was planned to take about an hour: roughly 50 minutes of presentation and 10 minutes of discussion. Each chapter has a "Presenter's guide" that points to the sections, figures and tables at the center of the paper and suggests how to divide the time. The presenter prepares from it. The other participants will find the presentation and discussion easier to follow if they read "The question this chapter answers" and "Building intuition" in advance.

Every chapter has the same structure.

2. How many meetings? — Schedules

The bracket next to each chapter title gives its type and importance. Core chapters come from the list of papers first proposed in the study group; Optional chapters are supplements. ★★★ marks the backbone (leave it out and later chapters will not make sense), ★★ is recommended (later chapters still make sense without it, but one strand gets thin), and ★ can be skipped, with the side-paper notes elsewhere covering enough.

If you add optional chapters, we suggest this order: 18 (statistics of evaluation) → 4 (CoT) → 14 (SWE-agent) → 5 (ToT) → 16 (alignment). Chapter 18 comes first because it suits readers with a statistics background best. Chapter 4 is directly needed to read the comparison in Chapter 6, because ReAct uses CoT as a baseline. If you leave out Chapter 4, as the 15-meeting basic plan does, fill the gap the way the item "If your group skipped Chapter 4" in Chapter 6's presenter's guide describes: spend the first 10 minutes of the Chapter 6 presentation introducing three figures from the CoT paper.

Whether to include optional chapters is up to the group. You can decide at the start or as you go. The chapter numbers in this book are fixed at 1 to 20, so skipped chapters do not make the others harder to find, and you can read a skipped chapter on your own. When you cover two chapters in one meeting, the reading range and time split are under "When combining chapters" in that chapter's presenter's guide.

3. The twenty chapters at a glance

The principle of the order is "add parts to the augmented LLM one at a time". Thinking (4–5) comes before acting (6). That is because ReAct uses CoT as its point of comparison. Tools (7) come before context (8–10). In the broad sense, the retriever in RAG is one tool. Verification (11) follows context, and memory (12) follows verification. Reflexion's memory of reflections is the bridge to MemGPT's tiered memory. After single agents are covered come collaboration (13), then coding agents (14) as a real system with all the parts in place, and then security (15–16). The damage from indirect injection grows by passing through tools, retrieval, memory and code repositories, so it only makes sense once all the parts have been seen. Evaluation and operations (17–19) come after everything there is to evaluate has been covered, and synthesis (20) ends with LLM Wiki, where retrieval, memory, verification and governance come together again in one system.

How the twenty chapters are grouped into parts. The editor drew the diagram.

4. The two surveys of Part I — head and body

Every later chapter ends with "Back to the surveys", which returns to the two surveys of Part I. The Agentic Reasoning survey in Chapter 2 looks at the agent's head. It sorts how a model plans, uses tools and searches, and how it improves through feedback and memory, by "what is being optimized" (the reasoning trace $z$, the system state $S$, the parameters $\theta$). The Harness survey in Chapter 3 looks at the agent's body and fence: everything outside the model, namely the execution environment, tool protocols, context management, orchestration, observability, verification and governance (ETCLOVG). The notation and layer names of the two surveys are fixed in Appendices I and II, and every chapter uses that notation.

5. Conventions

• Video labels. [Author] marks a talk by the paper's authors or an official channel, [Review] a paper review by a lab or study group, and [Secondary] a third-party summary or what appears to be an automatically generated one. Watch [Secondary] videos alongside the original. The editor did not verify the content of the videos. Korean-language videos and other Korean-language materials are kept and marked "(in Korean)"; titles and channel names in Korean are given in English, translated or romanized by the editor.
• Boxes. The Links to the originals box holds links, the A bridge from the editor box holds analogies, and the This section is a reanalysis done outside the paper box states the premises of a reanalysis.
• Folded solutions. In the screen edition, click the "Solution" heading to open them. In the print edition they are all open.
• Chapter references. Wherever "Chapter n" appears, it links to that chapter in the screen edition.
• Notation. Appendix I. Terms are in Appendix II. Where the Korean edition paired a translated term with the original English at first use (for example, attribution, harness), this edition simply uses the English term.
• Tables from the originals. The caption of each reproduced table gives the table number in the original. When you put numbers on presentation slides, look at the original table again, not this book.

6. What to know before you read

1. Versions. Read the Agentic Reasoning survey in the revised TMLR version on OpenReview. The version on arXiv (2601.12538) differs from the revised version in its section and table numbers and in its categories. v2 only adds "Accepted by TMLR" on the arXiv abstract page; its body is the same as v1 (compared on October 5, 2026). For the other papers, the latest arXiv version and the official conference version are taken to have the same content, but page numbers may differ between versions.
2. Peer review. Several of the 2026 sources are preprints (the Harness survey, HarnessFix, the position paper among the side papers of Chapter 18, and WikiSkill in Chapter 20). The Harness survey is posted on OpenReview, but as a direct archive upload, not after conference or journal review. Do not cite their claims and numbers as settled results.
3. When the numbers date from. The performance numbers in the papers are for the models of their time (GPT-4, PaLM and so on). Do not use them to compare with today's models. When you carry a number over, give the sample and conditions with it.
4. What company documents are. The posts by Anthropic, OpenAI, Manus and Chroma are prescriptions drawn from building their own products. The companies have an interest, and the posts were not peer reviewed. Read them as design guidance, not as generalized empirical evidence.
5. Videos. Videos marked [Secondary] are third-party summaries or appear to be automatically generated. Watch them alongside the original.
6. Distribution and copyright. Do not redistribute paper PDFs; share links (arXiv, OpenReview, conference pages). CC BY documents may be shared with attribution. Under its terms, the GAIA data may not be reshared outside its gated repository or published in a crawlable form.
7. Security exercises. Do prompt-injection exercises only in an environment you own or in a public evaluation environment (AgentDojo and the like). Do not try them against real services or other people's accounts. Do not put internal company material or personal information into a practice agent.
8. Terms. Terms follow the glossary in Appendix II.
9. Telling analogy from claim. The "bridge" analogies (for example, reward model = a Bradley–Terry-type model) were laid down by the editor to aid understanding. Keep them apart from the papers' claims. Readings that differ from a paper are written as "can also be read as". They are points for discussion, not conclusions.
10. Links. All links were checked on October 4, 2026. Blogs and videos may disappear or change address.
11. "Rereading with a statistician's eye". This is a reanalysis the editor calculated using only the numbers printed in the paper. The paper did not do such an analysis, and the aim is not to overturn its conclusions. It is an exercise in how the same numbers can be read through a statistician's eye. In chapters without numbers to work with, it is a study-design exercise that asks "how would you measure this to test the claim?"

7. For presenters

• Do not put internal information from your organization in presentation materials.
• When you write a number, give the model, task, metric and number of items with it. Not "ALFWorld 71%", but "PaLM-540B, 134 ALFWorld games, best of six prompts 71%, mean 57%".
• You can prepare the 10 minutes on side papers from this book's "Side papers" section. For side papers you want to explore further, follow the links to the originals in that section.
• If you show only one figure, the "Must read" item of each chapter names it.

Chapter 0 · not a meeting

Chapter 0. Preliminaries — Language Models, Tokens, In-Context Learning, and Reinforcement-Learning Terms

The question this chapter answers: what do you need to know to read a language model as a probability model, and which terms do the later chapters take for granted?

What this chapter is for. It is for readers who have never studied natural language processing or reinforcement learning, to read for an hour or two before the first meeting (or before Chapter 1, if you are reading alone). It is not presented. The later chapters use "token", "context window", "temperature", "a few examples", "episode" and "policy" without explanation; this chapter defines them and ties them to concepts you know from statistics. If you already know them, skip ahead and just do the "Self-check" at the end.

One-paragraph summary. A language model is a probability model that outputs the conditional distribution of the next token, $p(x_t\mid x_{<t})$, and training it is maximum likelihood estimation that reduces the negative log-likelihood. Tokens are word pieces cut by BPE, and there is an upper limit on how many tokens can serve as the condition (2,048 for GPT-3). Temperature flattens or sharpens the distribution; sending it to 0 gives greedy decoding. The GPT-3 paper used the name "in-context learning" for leaving the weights as they are and having the model solve a new task by putting an instruction and a few examples into the input. InstructGPT, which made that model follow instructions, went through three steps: demonstrations → reward model → reinforcement learning. The vocabulary of reinforcement learning (state, action, reward, policy, episode, trajectory, return, value function) is borrowed as is by the agent papers later in the book.

1. Reading a language model as a conditional distribution

One chain rule. The joint probability of a token sequence $x_1,x_2,\dots ,x_n$ can be factored as follows for any distribution.

$$p(x_1,\dots ,x_n)={\prod }_{t=1}^{n}p(x_t\mid x_{<t}),x_{<t}=(x_1,\dots ,x_{t-1})$$

This is an identity, not an approximation. It just writes the chain rule in time order. When the GPT-3 paper calls its model an "autoregressive language model" (abstract), autoregressive means that the conditional distributions of this factorization are modeled with a single neural network that produces one token at a time from left to right. The substance of the model lies not in the factorization but in the family of functions used to approximate $p_{\theta }(x_t\mid x_{<t})$ (for GPT-3, a transformer, §2.1) and in how far back it looks (the context window, below). An $n$-gram model is a Markov chain that cuts the past down to the last few tokens.

The training objective is the negative log-likelihood. Add up the log of the probability the model gave to the actual next token at each position, and flip the sign.

$$ℒ(\theta )=-{\sum }_{t=1}^n\log p_{\theta }(x_t\mid x_{<t})$$

Reducing this is maximum likelihood estimation, and it is the same as reducing the cross-entropy. If the model gave the actual next token a probability of 0.25, the loss at that position is $\ln 4\approx 1.39$. The average loss turned back through the exponential, $\exp (\overline{ℒ})$, is called perplexity, and it reads as "on average, how many candidates is the model choosing among?" In this example it is 4. Because the next token is itself the answer, no labels made by people are needed. All the GPT-3 models were trained on 300 billion tokens (Table 2.1 caption). On the other hand, this objective treats every token the same. §5 of the paper lists as a limitation that it cannot tell which predictions matter more.

What is a token? The unit the model sees is neither a character nor a word. GPT-3 says it uses GPT-2's tokenization as is (the "reversible tokenization" of §2.1), and it counts the size of its data in "byte-pair-encoded tokens" (§2.2: 570GB of filtered Common Crawl is roughly 400 billion tokens). BPE (byte-pair encoding) takes bytes as the initial vocabulary and repeatedly merges the pair that most often appears together into a new symbol until the vocabulary reaches a set size. Common words become one token and rare words are broken into pieces. The paper explains that one BPE token is on average about 0.7 words, so the letter-scrambling tasks require the model to pull apart the insides of tokens (§3.9.2), and it guesses that the unusually weak English→Romanian translation may be due to reusing GPT-2's BPE tokenizer, which was built from almost entirely English data (§3.3).

The context window. The past $x_{<t}$ used as the condition has an upper limit. All the GPT-3 models use a context window of $n_{\text{ctx}}=2048$ tokens (§2.1), and training always uses sequences filled to the full 2,048 tokens. Short documents are joined with an end-of-text token between them (Appendix B). Estimated at about 0.7 words per token, that is a little over 1,400 English words (the editor's calculation). This limit is also the limit of in-context learning. Examples go in as far as the window allows, usually 10 to 100 (§2). The "context" of the later chapters is all the tokens in this window, and it fills up as an agent piles up thoughts, actions and observations. That information is not used evenly even within the window, depending on its position, is the subject of Chapter 9; designs that keep memory outside the window are the subject of Chapter 12.

Three of the column names in the table below appear here for the first time. $d_{\text{model}}$ is the length of the vector that represents one token inside the transformer (§2.1 of the paper describes it as the number of units in each bottleneck layer). Attention is the computation by which each position weights the earlier positions by how much to consult them and mixes them; a head is a unit that splits that computation into several branches done in parallel. The head dimension $d_{\text{head}}$ is the length of the vector one branch uses. The later chapters of this book do not use the inside of the transformer, so you can read the table only for "as the models grew, the layers and the width grew together" and move on.

Table 2.1 of the paper, with the batch-size and learning-rate columns removed. The context window is 2,048 for all eight models (§2.1). The comparison that follows is optional reading. The number of heads times the head dimension equals $d_{\text{model}}$ in six rows but not in XL (3,072 against 2,048) or 13B (5,120 against 5,140). Equality is not a required constraint and the paper does not explain it, so the original does not settle whether these are typos.

Sampling temperature and greedy decoding. For each token in the vocabulary $V$ the model outputs a real-valued score $z_i$ (a logit), and the softmax turns these into probabilities. Temperature $T>0$ is the number the logits are divided by before that.

$$p_T(i\mid x_{<t})=\frac{\exp (z_i/T)}{{\sum }_{j\in V}\exp (z_j/T)}$$

At $T=1$ you get the distribution the model learned, unchanged. Raising $T$ flattens the distribution; lowering it concentrates the mass on the largest logit. Here is the calculation for three tokens with logits $(2,1,0)$ (the editor's calculation).

If there is a single largest logit, the limit as $T\to 0$ gives that token probability 1. If two or more logits tie for the maximum, the limiting distribution is split evenly among them. So "temperature 0" means greedy decoding, which picks the most probable token at every position. With a rule fixed for ties, it is deterministic: the same input gives the same output. But picking the best token at each step does not maximize the probability of the whole sentence. Beam search, which carries several candidates along together, searches more widely for that. GPT-3 evaluated its free-form tasks with a beam width of 4 (§2.4). ReAct in Chapter 6 produces one trajectory by greedy decoding, while its comparison baseline CoT-SC samples 21 reasoning paths at temperature 0.7 and takes a majority vote (CoT-SC in ReAct §3.2; ReAct's greedy decoding in §3.3 footnote 4 and the Table 3 caption). Raise the temperature when you need varied samples, and lower it when you need one reproducible answer. Besides these there is top-$k$ sampling, which draws only from the $k$ highest tokens, and top-$p$ sampling, which draws only from the tokens up to cumulative probability $p$ (the poem samples in GPT-3 Appendix F use temperature 1 and $P=0.9$).

If you view next-token prediction as a multinomial logistic regression with tens of thousands of categories (the editor's analogy), the logits are the linear predictor, and dividing by the temperature is the same as sampling from a model with every coefficient multiplied by $1/T$. Unlike regression, though, the token just drawn becomes an explanatory variable for the next prediction, so one token that goes astray can pull what follows along with it.

2. In-context learning — learning that does not change the weights

The GPT-3 paper's definition. §1 of the paper introduces in-context learning like this.

"… using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by predicting what comes next."

The task is specified in the input text. The model takes an instruction or a few examples as the condition and carries on the rest as if predicting the next token. The weights stay the same. In the language of section 1, the $\theta$ of the conditional distribution $p_{\theta }(·\mid x_{<t})$ is held fixed and only $x_{<t}$ changes.

Outer loop and inner loop — Figure 1.1. The paper places this within the frame of meta-learning. Changing the weights by SGD during pre-training is the outer loop, and what happens within the forward pass over a single sequence is the inner loop, that is, in-context learning. SGD (stochastic gradient descent) draws part of the data, computes the gradient of the loss, and moves the weights a little in the opposite direction. A forward pass computes the output distribution from the input with the current weights held as they are. In the inner loop no gradient is computed and no weight is moved. When the examples placed in front change, the output computed with the same weights simply changes. The sequences in the figure repeat additions such as 5 + 8 = 13, spelling corrections such as gaot => goat, and translations such as thanks => merci within one sequence.

Four settings — Figure 2.1. §2 sets out four settings side by side by how much task-specific data they use, and draws them with a single English→French translation example.

Figure 2.1 and the definitions in §2 of the paper, arranged as a table. The heading of the fine-tuning panel on the right of the figure is "Traditional fine-tuning (not used for GPT-3)".

The input in the few-shot panel is these five lines.

Translate English to French:      ← task description
sea otter => loutre de mer        ← examples
peppermint => menthe poivrée
plush girafe => girafe peluche
cheese =>                         ← prompt

The few-shot panel of Figure 2.1 in the paper. The labels on the right are also in the figure, and "girafe" is spelled as in this panel (the fine-tuning panel has "giraffe").

All the model has to do is predict the token after cheese =>. The three example lines convey both the format and the task. In zero-shot, the one line of description has to do all of that. The paper writes that zero-shot is the most convenient but the hardest, and since even a person may find it hard to know the format without an example, as with a request to "make a table of world records for the 200m dash", it is in some cases "unfairly hard".

What does "shot" count? It is the number $K$ of examples (demonstrations) given as the condition at inference time. It is neither the number of gradient updates nor the size of the training data. For each evaluation item, $K$ examples are drawn afresh at random from that task's training set, and if there is a development set, several values of $K$ are tried and the best is used (§2.4). This book says "give a few examples", and keeps the names zero-shot, one-shot and few-shot only for the settings.

Scores rise with the number of examples and the size of the model. Here are two sets of numbers that §1 gives directly in the text (GPT-3 175B). CoQA is a reading-comprehension task in which questions and answers about one passage follow one another like a conversation; the answers are free-form (§3.6, and Figure G.18 in Appendix G). The score is F1, which matches the model's answer against the correct answer word by word: the harmonic mean of precision and recall, written on a scale of 0 to 100 (the definition of F1 is the editor's general explanation; the GPT-3 paper does not give it). TriviaQA is a set of quiz-style factual questions. The "closed-book" setting is the condition in which only the question is given, without retrieving and attaching related documents (§3.2). The score is the share of questions answered correctly.

Numbers from the text of §1 of the GPT-3 paper. The paper writes that the few-shot TriviaQA score of 71.2 was the best result at the time, above fine-tuned models in the same closed-book setting.

The paper sees a tendency for the gap between zero-, one- and few-shot to widen as the model grows, and interprets it as a sign that larger models may be better meta-learners (§1, Figures 1.2 and 1.3). The values read off the figures' axes are not carried over here. The sentence the paper stresses is this one.

"We emphasize that these 'learning' curves involve no gradient updates or fine-tuning, just increasing numbers of demonstrations given as conditioning."

How do the examples work? — what the paper says. The paper does not explain the mechanism; it deliberately leaves it open. Footnote 1 says that the names zero-, one- and few-shot "are intended to remain agnostic on the question of whether the model learns new tasks from scratch at inference time or simply recognizes patterns seen during training". §5 lays out that question as a spectrum. The model may already have seen examples from the same distribution during training, or the same task in a different format, or it may really be learning a new skill, and where on that spectrum it falls differs from task to task. The paper's guess is that artificial tasks such as letter scrambling are learned from scratch, while translation was learned in pre-training. As evidence, §3.9.2 points to the results that on the scrambling tasks the one-shot score drops to half or less of the few-shot score and that the model can rarely do them zero-shot, while adding the caveat that this cannot be confirmed with certainty.

How this connects to later chapters. CoT in Chapter 4, ToT in Chapter 5 and ReAct in Chapter 6 all sit on top of this setting. A model with frozen weights is shown a few worked examples written by people and made to follow their format (ReAct's HotpotQA prompt has 6 examples and its FEVER prompt has 3). The first optimization target the survey in Chapter 2 names, "in-context reasoning", belongs to this line: the parameters $\theta$ are held fixed and only the reasoning trace $z$ changes (Appendix I, section 4).

3. How models that follow instructions were made

GPT-3 was trained to guess the next token, not to follow instructions. §1 of the InstructGPT paper points to this gap. Predicting the next token on a webpage from the internet is a different objective from "follow the user's instructions helpfully and safely", so the paper says the language modeling objective is "misaligned", that is, not aligned. §5 of the GPT-3 paper also writes that useful language systems might be better thought of as taking goal-directed actions rather than just making predictions, and it names learning an objective function from humans and fine-tuning with reinforcement learning as directions for future work. Linking the two papers this way is this book's reading. InstructGPT's starting point is GPT-3 (§3.5).

The three steps of Figure 2 in the InstructGPT paper, redrawn. The wording of each step follows the figure and §3.1, the numbers of prompts follow §3.2, and the range of $K$ follows §3.5. The example prompt for step 3 in the original figure is "Write a story about frogs". The editor drew the diagram.

Step 1 — supervised learning on demonstrations (SFT). For a sampled prompt, a labeler writes the desired answer by hand. The example in the figure is the prompt "Explain the moon landing to a 6 year old" with an answer that begins "Some people went to the moon..." GPT-3 is fine-tuned on these demonstrations with supervised learning (supervised fine-tuning). The original says only "supervised learning" and does not give the loss. Reading it as the negative log-likelihood of section 1 applied to the labelers' answers is the editor's explanation.

Step 2 — a reward model from comparisons (RM). For one prompt, 4 to 9 model outputs are sampled and a labeler ranks them, and all $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairs that come out of one ranking are used in training (§3.5). The reward model takes a prompt and an answer and outputs one scalar score (the paper used only the 6B size). The loss (Eq. 1 in the paper) models the probability that the preferred answer gets the higher score with a logistic function, and Chapter 16 reads it again as a paired-comparison model.

Step 3 — reinforcement learning against the reward model (PPO). A policy that starts from the SFT model writes answers to new prompts, the reward model scores them, and PPO updates the policy with those scores as the reward. To keep the policy from chasing the reward model alone and writing nonsense, a penalty for how far it has moved from the SFT model (the KL divergence) is subtracted at each token (§3.5). The prompts in this step carry no human labels. Human judgment has already gone into the reward model in step 2. Steps 2 and 3 can be repeated, but in practice most of the comparison data came from outputs of the SFT policy (§3.1).

The result in one line. Outputs from the 1.3B-parameter InstructGPT were preferred in human evaluations to those of the 175B GPT-3 (abstract). The win rates in Figure 1 would have to be read off the axes, so they are not carried over. The details and the evaluation are covered in Chapter 16; from this chapter take only the flow and the words needed to read step 3 (policy, reward, episode).

4. Reinforcement-learning terms — episode, reward, policy, value

Reflexion in Chapter 11 has "Verbal Reinforcement Learning" in its very title, and DeepSeek-R1, among the side papers of Chapter 4, draws out reasoning ability with reinforcement learning, as its title says. The scoreboard in Chapter 6 includes imitation-learning and reinforcement-learning baselines. This section sets out, at textbook level, the words these papers use without defining them, with notation matched to Appendix I. At each time $t$ the agent looks at the state of the environment (or observes only part of it) and chooses an action; the environment moves to the next state and gives a reward. This repeats until the end.

Return and discounting. The return is the sum of the rewards received after time $t$, discounted by the discount factor $\gamma \in (0,1)$.

$$G_t={\sum }_{k\ge 0}{\gamma }^k{r}_{t+k}$$

If the agent succeeds and receives a reward of 1 only after its third action, then with $\gamma =0.9$ the return at the first time step is $G_0={0.9}^2=0.81$. After the tenth action it would be ${0.9}^9\approx 0.39$. Discounting rates policies that succeed quickly more highly. The value function is the expected return $V^{\pi }(s)={𝔼}_{\pi }[G_t\mid s_t=s]$, and $Q^{\pi }(s,a)$, which also fixes the action, is the action-value function. Reinforcement learning looks for the policy that maximizes the expected return. When the policy is determined by parameters $\theta$, this becomes the objective in Appendix I.

$$J(\theta )={𝔼}_{\tau ~{\pi }_{\theta }}\left[{\sum }_{t\ge 0}{\gamma }^t{r}_t\right]$$

The survey in Chapter 2 writes the environment as a partially observable Markov decision process (POMDP), $E=⟨X,O,A,T,\Omega ,R,\gamma ⟩$ (latent state space, observation space, action space, transition kernel, observation kernel, reward, discount factor). The terms in the table above each fill one of these slots.

Episode and trajectory. An episode is an event, one attempt at a task from start to finish; a trajectory is the record of that event, the data. The ReAct paper defines the WebShop success rate as the "percentage of episodes where the chosen product satisfies all requirements" (§4). Success is counted per episode, and the log of thoughts, actions and observations read in Chapter 6 is a trajectory. Reflexion calls one attempt a "trial" and writes that each trial produces a trajectory ${\tau }_t$ (§3, Algorithm 1).

Episode length also differs from task to task. One ALFWorld game runs to dozens of actions. Step 3 of InstructGPT is a bandit environment that ends in a single step: it presents one customer prompt, receives an answer, gives the reward set by the reward model, and ends the episode (§3.5). Reinforcement learning ran for 256,000 episodes, over about 31,000 distinct prompts (Appendix C.4). The same appendix states that no discount was applied in estimating the advantage.

What are the policy and the reward? In a language-model agent, the policy is the language model given a prompt. ReAct §2 writes the policy as $\pi (a_t\mid c_t)$, where the context $c_t$ is the text of the observations and actions so far, joined together. Even under the same ${\pi }_{\theta }$, what $\theta$ refers to differs from paper to paper. In Appendix I and in InstructGPT it is the weights, but Reflexion's Algorithm 1 sets $\theta =\{M_a,\text{mem}\}$, the actor model bundled with the memory of reflections. The paper writes that it parameterizes a policy "as an agent's memory encoding paired with a choice of LLM parameters" (§1), and states that it reinforces through verbal feedback without updating weights (abstract). The same goes for the reward. Reflexion's evaluator takes a trajectory and outputs a scalar reward $r_t=M_e({\tau }_t)$ (§3), while InstructGPT's reward comes from a reward model learned from human comparisons. Who assigns the reward is the definition of "what counts as having done well", so this touches the evaluation problems of Chapter 17.

Imitation learning against reinforcement learning. Imitation learning trains by supervised learning to copy the actions in expert trajectories. It needs demonstrations and no reward. Reinforcement learning revises the policy with the rewards received from interacting with an environment. It needs a reward and an environment, and no demonstrations. The baselines in ReAct §4 show both.

The account in the text of ReAct §4, arranged as a table. ReAct's Table 4 states that the IL and IL+RL figures come from the original WebShop paper (Yao et al. 2022).

Notice that what IL+RL added was not trajectories but "instructions". Reinforcement learning does not need records of people solving the task. You give it the task, let it solve it directly, and then give a reward. In this book's reading, InstructGPT has the same structure. Step 1, SFT, has the form of imitation learning, copying demonstrations, and step 3 is reinforcement learning that gives only prompts and revises with rewards. The policy-update algorithm that step 3 used is PPO (Proximal Policy Optimization, Schulman et al. 2017). In this book you only need to know the name.

5. The notation and words this book uses

The notation is in Appendix I, and the terms are in Appendix II. Let us fix just five words first.

• thought. Intermediate reasoning written out in words before an action. It does not go out to the environment; it only accumulates in the context. Symbol $z_t$.
• action. Output that goes out to the environment: a tool call, a search, a final answer. Symbol $a_t$.
• observation. What the environment returns in response to an action. Usually text. Symbol $o_t$.
• reasoning trace. Another name for thought. The survey in Chapter 2 treats its space $Z$ as either latent plans or thoughts written out in words.
• harness. Everything outside the model that runs the model: the execution environment, tool protocols, context management, orchestration, observability, verification and governance. It is the subject of Chapter 3.

Where the symbols overlap. This chapter has used some letters in two senses.

Self-check

Answer each question in a line or two. The answers are collected in the folded solution at the bottom.

1. What does "temperature 0" mean? You cannot put 0 into the formula, so why is it called that?
2. Run the same prompt twice at temperature 1 and the answers can differ. How did the CoT-SC baseline in the ReAct paper use this property?
3. Under what assumption does the autoregressive factorization $p(x_1,\dots ,x_n)={\prod }_{t}p(x_t\mid x_{<t})$ hold as an approximation?
4. What is the negative log-likelihood at a position where the next token was given probability 0.5? If that value kept coming up, what would the perplexity be?
5. Can a document of 2,000 English words fit in a model with a 2,048-token context window? What do you estimate it with?
6. What does the "shot" in few-shot count? How many times are GPT-3's weights updated in the few-shot setting?
7. How do an episode and a trajectory differ? What is the unit counted in WebShop's success rate?
8. How many steps make up one episode in step 3 of InstructGPT? Why is the discount factor not an issue?
9. How does the training data of imitation learning differ from that of reinforcement learning? What data did WebShop's IL+RL add to IL?
10. A reward of 1 comes only after the fourth action, and $\gamma =0.9$. What is the return $G_0$ at the first time step? What does $\theta$ in Reflexion's policy ${\pi }_{\theta }$ refer to?

Videos

• [Author] "[1hr Talk] Intro to Large Language Models" — Andrej Karpathy

The title and channel of the video were checked through YouTube oEmbed on October 4, 2026. The editor did not verify the content of the video.

Further study

• The rest of the GPT-3 paper. §3 (evaluation — zero-, one- and few-shot compared by model size across tasks in 9 categories), §4 (measuring contamination of test items), Appendix G (the prompt wording for each task).
• The InstructGPT paper. §3 (methods) and §4 (results) are read in Chapter 16.
• Within this book. Chapter 2 (formalizes the terms of section 4 as tuples), Chapter 4 (drawing out thinking with a few examples), Chapter 6 (greedy decoding, imitation- and reinforcement-learning baselines), Chapter 11 (reinforcement that changes memory instead of weights), Chapter 16 (the three steps of section 3 in depth).

Sources

Reading

• [S08] Tom B. Brown et al. "Language Models are Few-Shot Learners". NeurIPS 2020. arXiv 2005.14165 · NeurIPS 2020. Cited: abstract, §1 (Figures 1.1, 1.2 and 1.3, footnote 1), §2 (Figure 2.1), §2.1 (Table 2.1), §2.2, §2.4, §3, §3.3, §3.9.2, §5, Appendix B, Appendix F.
• [S07] Long Ouyang et al. "Training language models to follow instructions with human feedback". NeurIPS 2022. arXiv 2203.02155 · NeurIPS 2022. Cited: abstract, §1, Figure 1, Figure 2, §3.1, §3.2, §3.5, Appendix C.4.

Also cited

• [L02] Shunyu Yao et al. "ReAct: Synergizing Reasoning and Acting in Language Models". ICLR 2023. arXiv 2210.03629. Cited: §2, §3.2, §4, Table 4.
• [L11] Noah Shinn et al. "Reflexion: Language Agents with Verbal Reinforcement Learning". NeurIPS 2023. arXiv 2303.11366. Cited: abstract, §1, §3, Algorithm 1.
• [S06] DeepSeek-AI. "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning". Nature 645, 2025. arXiv 2501.12948 · Nature. This chapter cites only the title.

How this was checked. On October 5, 2026, the section, figure and table numbers and the figures in the text were checked against the PDFs of GPT-3 arXiv v4 (2020-07-22) and InstructGPT arXiv v1 (2022-03-04). GPT-3 Figures 1.1 and 2.1 and InstructGPT Figure 2 (PDF page 3) were read by rendering the pages. ReAct (arXiv v3) and Reflexion were checked against the PDF text. Authors, venues and links follow the bibliographic details in Appendix III, checked on October 4, 2026. The notation follows Appendix I; the AR survey itself was not checked separately for this chapter. The softmax probabilities at each temperature, the word-count estimate and the return calculations were done by the editor in Python (numpy). The reinforcement-learning terms are general textbook-level definitions; only the statements that "a paper writes it this way" were checked against the papers concerned.

Chapter 1 · Core · ★★★

Chapter 1. What Is an Agent? — A Practitioner's Map

The question this chapter answers: is every LLM with tools attached an agent? When should you not use an agent?

Core reading. Erik S. and Barry Zhang (Anthropic), "Building effective agents", Anthropic Engineering blog, December 19, 2024. This is a practical guide, not a paper. About 1,900 words of main text and about 2,700 with the two appendices (8 to 9 pages printed).

One-paragraph summary. The post distills what Anthropic learned from working with dozens of teams across industries and from building its own agents. A workflow ties LLMs and tools together through code paths fixed in advance; an agent is an LLM that directs its own process and tool use; together they are called agentic systems. The prescription fits in one line. Look for the simplest solution first, and raise complexity only when results actually improve. The post holds that for many applications a single LLM call with retrieval and examples is enough. It then lines up, in order of complexity, the augmented LLM, five workflows (prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer) and agents. It has no measurements. You can read the papers in later chapters by placing them in the cells of this map.

Presenter's guide

A 50-minute plan. The 10 minutes after the presentation are for discussion.

Must read. The two definitions in "What are agents?", the two paragraphs of "When (and when not) to use agents", the "When to use this workflow" paragraph attached to each of the five workflows, the "Agents" section and the three principles in "Summary". For the five workflows, the "when to use" paragraphs matter more than the names. Each states in a sentence or two what the pattern buys and what it pays.

Optional reading. Appendix 1 (the four conditions under which agents are worth their cost). Appendix 2 comes back in Chapter 7 and Chapter 14, so read only the closing anecdote about absolute paths.

What to cut from the presentation. "When and how to use frameworks". Remember only the gist (use the API directly first).

Spend the last 5 minutes on one example. The "Back to the surveys" section of this chapter runs a little over three pages and cannot be covered in 5 minutes. In the presentation, pick orchestrator-workers alone and show only that it sits under LLM-Driven Pipelines in §5.2.1 of the revised AR survey and in the Orchestration box of Figure 1 in the Harness survey. Leave Table 5, the notation table, the discussion of axes in §2.3 and §3.1.1, and Harness §2.1 and §11.2 for self-study.

Which version. The current page carries a note at the top: "Much of the tooling landscape described in this post has changed since December 2024." Compared with the Internet Archive copy of December 20, 2024, the body has changed in three places, apart from the new top note and the summary added under the title. The list of frameworks (LangGraph and Amazon Bedrock's AI Agent framework → the Claude Agent SDK and the Strands Agents SDK; Rivet and Vellum stay), the routing example (Claude 3.5 Haiku and 3.5 Sonnet "to optimize cost and speed" → Claude Haiku 4.5 and Sonnet 4.5 "to optimize for best performance") and the author credit (Erik Schluntz → Erik S.). The sentences of the definitions, recommendations, patterns and appendices are the same.

If your group skipped Chapter 0. The words "in-context examples" and "tokens" come up, so in the first 3 minutes point out only that putting a few examples in the prompt changes the model's behavior without changing its weights (in-context learning).

When you quote the numbers. The post itself has no measurements. When you carry over numbers from the side papers and the statistics section (token multiples, SWE-agent's resolve rates and so on), give the model, task and conditions each source states, and if the conditions are not stated, say so.

Building intuition

Two definitions — who decides the next step

"Agent" can mean a fully autonomous system that runs on its own for a long time and uses many tools to get complex work done, or a more prescriptive implementation that follows a predefined workflow. The post calls both agentic systems, but draws one structural line.

Workflows are systems where LLMs and tools are orchestrated through predefined code paths.

Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.

The two sentences differ in one thing: who decides the next step. In a workflow, code the developer wrote decides, and the LLM is one of the functions that code calls. In an agent, the LLM decides, and the code steps back to running the tool the LLM picked and returning the result. Ask whether what happens next is written in an if statement in the code or in the model's output. One caveat: the question is about the structure of the steps. Inside a workflow, too, the model can make judgments such as classifying an input or splitting a task. If those judgments happen inside branches and steps the code opened in advance, it is a workflow; if the model also decides the kinds of steps and their order, it is an agent. The patterns that sit on the boundary come back in section 5 of "The paper in detail".

Tools are not the criterion. An LLM with retrieval and tools attached is only the basic building block (the augmented LLM). Call that block in an order set by code and you have a workflow; let it choose its own next call and you have an agent. The post's answer to "is every LLM with tools attached an agent?" is no.

The statements in "When (and when not) to use agents" and in the opening of "Building blocks, workflows, and agents", set out as a ladder. This is not a figure from the post; the editor drew the diagram.

The editor's bridge — parsimony in model selection

The post's own analogies are two. It likens evaluator-optimizer to the way a person polishes a piece of writing, and it says to invest as much effort in the agent-computer interface (ACI) as goes into human-computer interfaces (HCI) (Appendix 2).

The paper in detail

1. What is an agent? — definition and scope (What are agents?)

A short section. Beyond the two definitions, two things deserve attention. First, it draws the scope wide. Fully autonomous systems and prescriptive workflows are all agentic systems, so instead of "agent or not" you get a continuous axis: how many decisions are handed to the model. The rest of the post moves along this axis. Second, the criterion for the distinction is architecture ("an important architectural distinction"). With the same model and the same tools you can build either a workflow or an agent.

2. When to use agents and when not to (When (and when not) to use agents)

Read the two paragraphs sentence by sentence.

When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all.

The prescription opens with "This might mean not building agentic systems at all." A post that teaches you how to build agents starts with a sentence telling you not to build one.

Agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense.

What you pay is latency and cost; what you buy is task performance. The post says only "often" and gives no size. Anthropic's own numbers for the size appear in a post half a year later (Side papers).

When more complexity is warranted, workflows offer predictability and consistency for well-defined tasks, whereas agents are the better option when flexibility and model-driven decision-making are needed at scale. For many applications, however, optimizing single LLM calls with retrieval and in-context examples is usually enough.

Three levels come out here. In summary:

The editor put the two paragraphs of the post into a table. "A ceiling on performance" and "lower predictability" are the editor's reading of the flip side of the advantages the post names.

"Summary" says the same thing again about success: "It's about building the right system for your needs. Start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when simpler solutions fall short." The last sentence of "Combining and customizing these patterns" says it too: "To repeat: you should consider adding complexity only when it demonstrably improves outcomes." The same recommendation appears three times, and the later two ("Summary" and "Combining") attach a condition of measurement ("comprehensive evaluation", "measuring performance", "demonstrably"). The first, in "When (and when not) to use agents", does not. So the real content of the recommendation can also be read as "do not raise complexity without evaluation". Read that way, the post lays down the evaluation problems of Chapter 17 and Chapter 18 already in Chapter 1.

On frameworks (cut from the presentation). Frameworks make low-level tasks easy, but their layers of abstraction hide the prompts and responses, which makes debugging harder, and they make it tempting to add unnecessary complexity. So the post says to use LLM APIs directly first, and, if you do use a framework, to understand the code underneath. The ground for this is the observation that incorrect assumptions about what is under the hood are a common source of customer error.

3. The building block — the augmented LLM (Building block: The augmented LLM)

The material of every pattern is a single LLM augmented with retrieval, tools and memory. The model writes its own search queries, chooses tools and decides what to keep. What to attend to is tailoring these capabilities to the use case and giving the LLM an easy, well-documented interface. As one route to implementation, the post names the Model Context Protocol, released 24 days before the post (November 25, 2024). We read MCP as a specification in Chapter 7.

The last sentence of the section is the key to reading the rest: "For the remainder of this post, we'll assume each LLM call has access to these augmented capabilities." Since every LLM call in a workflow can already use tools, one branch of a routing workflow or one worker may itself be a small agent that uses tools several times. The post does not take up this overlap. In the editor's reading, the distinction between workflow and agent is made for one layer of the control flow, not for the system as a whole. Also, planning, which Weng counts as one of the three components of an agent, is absent from L21's augmented LLM. Planning moves to the question of how the blocks are put together, that is, workflow or agent.

4. Five workflows (Workflow: Prompt chaining to Workflow: Evaluator-optimizer)

The post lays out the patterns it has seen in production, from the simplest up. Each pattern gets a definition, a "When to use this workflow" paragraph and examples. The post has a diagram for each pattern; here they are summarized in a table instead.

The editor summarized the five workflow sections of the post in a table. The links in the last column are this book's reading.

The gates in chaining. A gate is one of the "programmatic checks", a check written in code. At each intermediate step it confirms that the process is still on track. Chaining makes each call an easier task, paying latency to buy accuracy.

The rationale for routing. Without routing, a prompt tuned for one kind of input can hurt performance on other inputs. This is separation of concerns. The classification can be done by an LLM or by a traditional classification model. Even when an LLM makes the judgment, if the branches are written into the code in advance, it is a workflow.

The two variations of parallelization. A rule of thumb comes attached: for tasks with several considerations, calling a separate LLM for each consideration usually works better. Splitting guardrails from the core response, instead of handing both to one call, "tends to perform better", but no size is given. Voting resembles self-consistency in Chapter 4. But self-consistency samples several reasoning paths from the same prompt and picks by majority vote, while the post's voting examples balance false positives and false negatives with different prompts or different vote thresholds. This is a matter of tuning a decision rule's sensitivity and specificity through its threshold.

How orchestrator-workers differs from parallelization. The post itself admits the two are "topographically similar". The difference is flexibility. In parallelization the subtasks are defined in advance; in orchestrator-workers the orchestrator decides them after seeing the input. The example given is coding, where you cannot know how many files need changing, or how, before you see the task.

The two signs of good fit for evaluator-optimizer. That a response improves noticeably when a person states feedback in words, and that an LLM can give such feedback. Without the second, the generator may faithfully follow off-target feedback and get worse (this is the editor's inference, reading the two signs in reverse). It comes back in Chapter 11.

"Combining and customizing these patterns" adds that these blocks are not prescriptions but common patterns, to be combined, with performance measured and implementations iterated.

5. Agents (Agents)

The post writes that agents are emerging in production as four capabilities mature (understanding complex inputs, reasoning and planning, using tools reliably and recovering from errors), and sketches an agent's life. It starts with a command from a person or a discussion; once the task is clear, it plans and acts on its own, at each step gaining "ground truth" from the environment (the results of tool calls or code execution) to judge its progress. At checkpoints or when it hits a blocker, it waits for human feedback, and it ends when the task is done, though stopping conditions such as a maximum number of iterations are also common. The gist is these two sentences.

Agents can handle sophisticated tasks, but their implementation is often straightforward. They are typically just LLMs using tools based on environmental feedback in a loop.

This is the thought-action-observation loop of ReAct, which we read in Chapter 6 (the post does not cite ReAct). Because the skeleton of the loop is simple, the post moves straight to saying that what counts is designing the toolset and its documentation clearly and thoughtfully, and hands off to Appendix 2. Its judgment is that success or failure is decided outside the loop, and the concept of the harness in Chapter 3 can be read as turning this into a system.

When to use them, what they cost. Use them for open-ended problems where you cannot predict the number of steps needed and cannot hard-code the path. You "must have some level of trust in its decision-making", and agents are suited to scaling tasks in "trusted environments." What happens when the documents an agent reads cannot be trusted is the question of Chapter 15. The price is higher cost and compounding errors, so the post recommends extensive testing in sandboxes and appropriate guardrails. Its examples are Anthropic's own SWE-bench coding agent and its computer use reference implementation.

Getting a feel for compounding errors in numbers. The post does not give sizes. The editor calculates the simplest model. If each step succeeds independently with probability $p$ and a single failure fails the whole task, the probability that a $k$-step task succeeds is $p^k$.

Values the editor calculated as $p^k$. A simple model that assumes independence between steps and "one failure ends it".

A real agent does better than this when it looks at the ground truth and fixes failures from earlier steps, and worse when an earlier error contaminates the next input. Still, the table shows that the longer the task, the more steeply the demand on per-step reliability rises. The pass^k metric in Chapter 17 looks at the same multiplicative structure from the side of repeated trials.

The editor's formalization — where the function that sets the path lives. Write the two definitions in the notation of Chapter 2. The post has no formulas.

In a workflow, a function $g$ written by the developer decides the next call. It looks at the current step ${\ell }_t$ and the output $y_t$ that the step's LLM call just produced, and chooses the next step ($\ell$ and $y$ are symbols the editor introduced and are not in the notation appendix. $y_t$ gets a different letter because it is the output of an LLM inside the workflow, not an observation $o_t$ from the environment).

$${\ell }_{t+1}=g({\ell }_t,y_t)$$

For routing, $y_t$ is the classification result and $g$ is a table that turns it into a branch number. The LLM's output affects the path, but the set of possible paths was enumerated in advance by whoever wrote $g$.

In an agent, the next action comes from the model's execution policy.

$$a_t~{\pi }_{\text{exec}}(a_t\mid h_t,z_t),a_t\in A=A_{\text{tool}}\cup \{\text{finish}\}$$

The developer sets the tool list $A_{\text{tool}}$, but the model decides what to use, how many times, in what order and when to end with finish. The harness only sets outside limits such as a maximum number of iterations.

Written this way, two patterns in L21's own classification turn out to sit on the boundary. Orchestrator-workers is a workflow, yet its subtasks "aren't pre-defined". In the search example of evaluator-optimizer, the evaluator decides whether to search further, so the number of iterations is open. Read "predefined code paths" as "the skeleton of the calls is in the code", and the two statements fit together. In orchestrator-workers, the code holds the skeleton "decompose → execute → synthesize" and the model fills in only the number and content; an agent has no skeleton for the order in which to solve the task. An agent too has a skeleton — the loop that repeats "call → run the tool → return the result" and a cap on iterations — but that is the skeleton of the layer that wraps execution (the harness), not of the order in which the task is carried out. You can also read the two as joined by a continuous band rather than a line.

6. Appendix 1 — customer support and coding agents (Appendix 1: Agents in practice)

Appendix 1 names two applications that were especially promising in work with customers. Its first paragraph states what the two have in common. Agents add the most value for tasks that have these four features:

• They need both conversation and action.
• They have clear success criteria.
• They allow feedback loops.
• Meaningful human oversight can be attached.

A. Customer support. A chatbot interface with tools attached. It follows a conversation flow yet needs outside information and actions; tools that pull customer data and order history and actions such as refunds and ticket updates can be attached; and success can be measured through "user-defined resolutions." As evidence of viability, the post notes that several companies use pricing models that charge only for resolved cases, but it gives no company names or figures.

B. Coding agents. Solutions are verified by automated tests, the agent can iterate using test results as feedback, the problem space is structured and quality can be measured objectively. The post writes that its own implementation can solve real GitHub issues in SWE-bench Verified from the pull request description alone, but it gives no resolve rate. And it adds the caveat that whether a solution fits broader system requirements still needs human review.

In both cases, agents do well where the environment tells you "what success is". This observation pairs with the point in the previous section that agents gain "ground truth" from the environment.

7. Appendix 2 — prompt engineering for tools too (Appendix 2: Prompt engineering your tools)

This comes back in Chapter 7, so look only at the gist: give tool definitions as much prompt-engineering attention as the overall prompt. A file edit can be specified as a diff or as a rewrite of the whole file, and structured output as markdown or as JSON. To a person these are cosmetic differences that convert losslessly, but some formats are harder for an LLM. A diff requires knowing how many lines change before the new code is written, and code inside JSON requires escaping every newline and quote. So the post says to give the model enough tokens to "think", to keep the format close to what it would have seen naturally in text on the internet and to remove formatting overhead such as counting lines or escaping. As tips for the ACI it lists putting yourself in the model's shoes, writing parameter names and descriptions like a docstring for a junior developer, testing with many inputs to find mistakes and poka-yoke, changing the arguments so that mistakes are structurally harder.

The anecdote at the end goes like this. When building its SWE-bench agent, Anthropic spent more time optimizing its tools than the overall prompt. When the model made mistakes with relative paths after moving out of the root directory, the tool was changed to always require absolute paths, and the model used this method "flawlessly." How many times this was tested is not stated.

8. Rereading with a statistician's eye

The recommendation of "finding the simplest solution possible" rests on two empirical claims: that there is a trade in which agentic systems pay latency and cost to buy performance, and that in many applications the trade does not pay off. The post gives the size of neither. A post from the same company half a year later (How we built our multi-agent research system) put token multiples of about 4× and 15× on the cost side ("In our data"), but it does not say how many cases were measured or how widely they scatter. The performance side will differ by task, so to test the recommendation you would have the same tasks solved by a single call, a workflow and an agent, and compare them in pairs.

Exercise — turning the advice to start simple into an experiment. Suppose your organization has a set of recurring tasks (for example, drafting replies to customer queries, answering questions about internal rules, small code fixes). You want to have the same tasks solved by (W) a workflow built from prompt chaining or routing and (A) an agent using the same tools, and compare success rate, cost and variance. (a) How would you set up treatment and control? What must be held the same? (b) Choose three or more outcome measures. Variance of what? (c) What is the sampling unit, and how many do you need? Estimate under the assumption of a 70% success rate for the workflow and 78% for the agent. (d) Name confounders that would contaminate the results. (e) What can this design still not answer?

9. Limitations the paper states

This is not a paper, so it has no Limitations section. Here are the caveats and reservations scattered through it.

• The trade. Agentic systems often pay latency and cost to buy performance, and whether the trade makes sense is for the builder to judge.
• The price of autonomy. Agents cost more and errors can compound, so the post says to test in sandboxes and set guardrails.
• The premise of trust. You must be able to trust the agent's judgment, and agents suit trusted environments. Untrusted environments are not discussed.
• Human review. Even for coding agents, fit with broader requirements needs human review (Appendix 1).
• Shelf life. The note at the top of the current page says the tooling has changed a great deal since December 2024 ("Which version").

There are also things the post does not state. Its basis is that "We've worked with dozens of teams", but it does not say how it judged "the most successful implementations". If you pull common features out of the successes, the simple implementations that failed stay invisible, so the post alone can also be read as leaving unsettled whether "the simple implementations succeeded" means "simple means success". Still, the prescription itself is "justify any added complexity with measurement", so it does not collapse under this weakness.

Side papers — how they differ from the main paper

The side papers for this chapter are Weng's post and the OpenAI guide. We add one later post from the same company and three papers, by pattern, that we will meet in later chapters.

LLM Powered Autonomous Agents (Weng 2023 · supporting material for this chapter). A personal blog post from June 23, 2023. It puts the LLM as the brain, supported by three components: planning (breaking into subgoals, reflection and refinement), memory (short-term as in-context learning, long-term as an external vector store) and tool use. Weng divides by what an agent is made of, and L21 by who sets the path. So Weng's map has no distinction between workflow and agent. HuggingGPT, which Weng gives as a tool-use case, has four fixed stages — task planning, model selection, task execution and response generation — and what it decides after seeing the input is the task list and the model assigned to each task, so in L21's terms it is close to an orchestrator-workers workflow (this book's reading). Of the three challenges at the end, the reliability of the natural language interface (much of the demo code goes into parsing model output) points to the same place as L21's Appendix 2.

A practical guide to building agents (OpenAI 2025). A 34-page PDF in which another company covers the same subject (no publication date given; the PDF was created on April 7, 2025). It uses the same word in a different sense. In this guide, "A workflow is a sequence of steps that must be executed to meet the user's goal," that is, the work itself, and an agent is a system that carries out that work on your behalf with a high degree of independence (p. 4). It says that applications that integrate LLMs but do not use them to control workflow execution are not agents, so L21's workflows would mostly fall outside this definition of an agent. Its prescriptions to maximize a single agent's capabilities first (p. 16) and that a deterministic solution may suffice (p. 6) overlap with L21's advice to find "the simplest solution possible". Only the guide takes up model choice. It says to set a baseline with the most capable model and then try swapping in smaller models (p. 8). L21's "simplicity" is the simplicity of the system's structure, not advice to start with a small model. In L21, model size appears only in the routing example (easy queries to a small model, hard ones to a large model). The manager pattern and the decentralized (handoff) pattern resemble, in shape, L21's orchestrator-workers and routing respectively. The text on p. 31 explains that the preceding example uses the math_homework_tripwire guardrail, but the guardrail in the code on pp. 28–30 is churn_detection_tripwire.

How we built our multi-agent research system (Hadfield et al. 2025). A post in which Anthropic, half a year later, says it built its Research feature on an "orchestrator-worker pattern". The workers are subagents that use tools several times on their own, so the "agent inside a block" of section 3 of "The paper in detail" actually appears, and there are numbers. A Claude Opus 4 lead agent with Sonnet 4 subagents outperformed single-agent Opus 4 by 90.2% on an internal research eval (the number of items and the definition of the metric are not given); on BrowseComp, token usage by itself explained 80% of the variance in performance; and agents use about 4× more tokens than chat, multi-agent systems about 15×. It also adds the caveat that most coding tasks, with little to parallelize, are not yet a good fit.

Reflexion (Shinn et al. 2023 · Chapter 11 of this book). It stretches evaluator-optimizer across trials. An Actor produces a trajectory, an Evaluator scores it (with exact match, heuristics or another LLM) and a Self-Reflection model turns that into a written reflection stored in memory (§3). L21's loop polishes one answer and leaves nothing behind when it ends. Table 4 of the revised AR survey writes this difference in symbols. Self-Refine, which corrects within one answer, is $z$; Reflexion, which carries its reflection over to the next attempt, is $S$.

MetaGPT (Hong et al. 2024 · Chapter 13 of this book). The problem its abstract aims at is "cascading hallucinations caused by naively chaining LLMs", that is, the failure of prompt chaining without gates. Its answer is to encode standard operating procedures (SOPs) into prompt sequences and to have roles such as product manager, architect and engineer exchange structured outputs through a shared message pool (publish–subscribe) and check intermediate results (§3). Roles and order are fixed by the SOP, so on L21's map it is close to chaining with gates, and in that it hands out work, it also overlaps with orchestrator-workers. §5.2.1 of the revised AR survey classifies it under Manually Crafted Pipelines.

SWE-agent (Yang et al. 2024 · Chapter 14 of this book). A paper that measures, with ablations, the agent-computer interface (ACI) that L21's Appendix 2 discussed through anecdote and advice (arXiv, May 2024; L21 does not cite it). SWE-agent with GPT-4 Turbo solved 12.47% (286) of the 2,294 SWE-bench tasks and 18.00% (54) of the 300-task SWE-bench Lite (§5). In Table 3, which changes the interface one piece at a time on Lite, removing the edit command dropped 18.0 to 10.3, showing the whole file dropped it to 12.7, and iterative search, which pages through results one at a time (12.0), was lower even than having no search tool (15.7). It is a measurement showing that the advice to give tool design as much attention as the prompt shows up as differences of several percentage points with the same model.

Back to the surveys

Open the two surveys and you can see where L21's distinction sits in each.

Agentic Reasoning survey, revised version — "what does it optimize?" and "who sets the path?"

§2.3 — the axes the survey rejected. The revised version names three classification axes it rejected: capability lists, architectural composition (single-agent / tool-based / multi-agent) and learning-paradigm-first. It rejected the composition axis because it is orthogonal to adaptation, leaving self-evolution no natural home, and the axis it chose is environmental dynamics × optimization target ($z$, $S$, $\theta$). L21's axis (who holds the control flow) coincides exactly with neither. §2.1 writes that two systems with the same composition are separated when one searches $z$ and the other updates $\theta$. Put the other way round, L21's workflows and agents can fall into the same cell in the survey.

§3.1.1 and Figure 2 — the workflow is a layer, not an alternative. The revised version has Workflow Design as one branch of in-context planning: approaches that structure the planning process into stages such as perception, reasoning, execution and verification. And it writes:

This reactive workflow view unifies scripted stage design with online adaptation: the workflow provides interpretable structure and interfaces (what is done when), while the reactive loop supplies closed-loop grounding and error recovery (how it is done in context).

L21 sets workflows and agents up as things to choose between (two rungs of a ladder); the revised version sets them up as two layers of one system. It is the same picture as the reading in section 5 of "The paper in detail", where orchestrator-workers was reconciled as "skeleton in code, content from the model". The caption of Figure 2 groups workflow design with the families of the system state $S$, not $z$, and §2.2 warns that calling work that optimizes prompts, workflows and communication graphs without touching weights "post-training" conflates two regimes. The work of L21's developer, revising a workflow's code and prompts while watching evaluations, can be read in this frame as optimizing $S$ by hand (the editor's interpretation).

§5.2.1 — pipelines crafted by people and pipelines built by LLMs. In-context collaboration among multiple agents is split into two lines. Manually Crafted Pipelines fix agent roles, execution order and communication rules before execution (MetaGPT); LLM-Driven Pipelines have an LLM orchestrator decompose a goal, route the pieces to specialized agents or tools and refine the workflow from intermediate feedback (Magentic-One and others). The survey judges that the former offer interpretability and simple execution but are rigid and hard to adapt to ambiguous tasks. Apart from its scope being multi-agent, this is where L21's distinction is most closely reproduced in the survey.

§2.4 and Table 5 — which mode, and when. A design guide the revised version added during review.

Table 5 of the revised AR survey (§2.4), reproduced from the original.

Table 5 asks "what to optimize", and L21 asks "how much control to hand to the model". Still, they overlap. Everything L21 covers lies within the two weight-frozen rows, $z$ and $S$, and fine-tuning is never mentioned. The first rung of the ladder is the starting point on the same ground as the $z$ row's "Zero marginal setup cost", and the cost that workflows and agents pay is written in the $z$ row as "cost recurs on every call". The revised version itself stresses that the table gives "readings of a fast-moving and often confounded literature rather than settled results". Neither L21, an account of experience, nor the survey, which organizes 866 bibliography entries (§1.1), offers its design guidance as a measured law.

Notation table. Linking L21's terms to the survey's symbols. The correspondence is the editor's reading.

Harness survey — inside one box of harness engineering

§1 — where it is cited. The survey cites L21 once (Anthropic, 2024a). It contrasts academic work, which studies planning, memory and tool use as separate capabilities, with production-oriented accounts, which describe tool contracts, retries, handoffs and permission boundaries, and gives L21 as one of the latter.

§2.1 — its place in the timeline. The timeline has three entries, "Reasoning–action (ReAct) loops (2022)", "Tool integration and multi-agent coordination (2023)" and "The harness turn (2025)", and no slot for 2024. L21 (December 2024) falls between the second and the third. The first entry notes that early systems ran on "a while loop, a prompt template, and a small tool-dispatch table", the same observation as L21's "their implementation is often straightforward." The third entry writes that by early 2026, practitioner reports increasingly attributed agent reliability to the design of the surrounding infrastructure as well as to model capability. L21's Appendix 2, which says to design tools and their documentation carefully, can be read as a paragraph that pointed to this shift ahead of time.

§2.2 and Figure 1 — three scopes. The survey places prompt engineering, context engineering and harness engineering as progressively broader scopes. The three pillars of Figure 1 (p. 1 of the PDF) are these.

The text of Figure 1 of the Harness survey, set out as a table. In the figure, the description under the Example & Format box is printed exactly the same as the box above it: "Define task, persona and constraints".

L21's content spans all three pillars. The single call refined with retrieval and examples is prompt engineering and Retrieval & Loading; the memory of the augmented LLM is Memory & State; the ACI of Appendix 2 is Tools & Environment; sandboxes and guardrails are Governance & Security; and measuring performance and iterating is Verification & Evaluation. The workflow–agent distinction that gives the post its name goes into one box of the harness pillar, Orchestration ("Plan, act, verify, retry, or delegate"). §2.3, too, defines the Lifecycle/Orchestration (L) layer as the layer that organizes control flow, "from single-agent loops to multi-agent and issue-to-pull-request workflows." L21's distinction is a design choice within the L layer, one of seven layers.

§11.2 — the trade between capability and control. The more authority you give, the larger the control problem. Wider tool menus increase selection error and the prompt-injection surface, and permissive sandboxes enlarge the blast radius of misaligned actions. This can be read as the harness side restating the conditions L21 set for agents: "you must have some level of trust in its decision-making" and "trusted environments."

The AR survey has received only an "Accept as is" from its TMLR action editor; its OpenReview status is still "Decision pending" and there is no final published version. The Harness survey is a preprint that has not gone through conference or journal review. We do not cite the classifications of either survey as settled conclusions.

Common misreadings

Misreading 1. "An LLM with tools attached is an agent." In L21, an LLM with retrieval, tools and memory attached is only the basic building block (the augmented LLM). Call that block in an order set by code and it is a workflow; it is an agent only when the LLM directs its own process and tool use ("What are agents?"). The OpenAI guide also writes that applications that do not use an LLM to control workflow execution are not agents (p. 4).

Misreading 2. "Agents are a more advanced stage than workflows, so in the end you should go to agents." L21 divides the two by use, not by stage of development. For well-defined tasks a predictable and consistent workflow is better; when flexibility is needed at scale an agent is better; and for many applications a single call is enough ("When (and when not) to use agents").

Misreading 3. "Orchestrator-workers is another name for parallelization." The post itself admits the shapes are similar. The difference is who decides the subtasks, and when. Parallelization runs subtasks defined in advance at the same time; in orchestrator-workers, the orchestrator creates them on the spot after seeing the input ("Workflow: Orchestrator-workers").

Misreading 4. "The workflow of the OpenAI guide and the workflow of this post are the same word." L21's workflow is a system in which code sets the path; the OpenAI guide's workflow is a sequence of steps that reaches a goal, that is, the work itself (guide, p. 4). In the guide, agents are what "perform the same workflows" on the users' behalf. Quote the two together and the same sentence reads the opposite way.

Misreading 5. "The post's recommendations are results confirmed by experiment." The basis is experience from "dozens of teams" and the company's own implementations, and there are no measurements. The token multiples (about 4× and 15×) and the improvement (90.2%) that the same company published half a year later also come without task counts or scatter. The design guide of the revised AR survey (Table 5) also states that it is not a settled result. Read corporate documents as design guidance, not as generalized empirical findings.

Misreading 6. "To build an agent, you must first pick a framework." The post recommends using LLM APIs directly first, and, if you do use a framework, understanding the code underneath ("When and how to use frameworks"). The framework names in that section changed after publication, so take the logic, not the names.

Misreading 7. "Attach an evaluator-optimizer loop and the output always gets better." The evaluation criteria must be clear, a response must improve when a person gives feedback in words and an LLM must be able to give such feedback ("Workflow: Evaluator-optimizer"). Without these conditions the loop only adds cost. What the evaluator should be is taken up again in Chapter 11.

Discussion questions

1. Put your own task on the ladder. Where would you place one recurring task from your work: single call, workflow or agent? Is your reason whether the task is well defined, whether you know the number of steps in advance or what you lose if it fails? How would you show that moving up one rung made things better?

2. Parsimony in two directions. L21 says to build the structure up from the simple; the OpenAI guide says to bring the model down from the most capable. Do the two prescriptions conflict, or are they different axes you can use together? Just as forward selection and backward elimination do not always reach the same model, can the two directions arrive at different systems?

3. What is a trusted environment? L21 writes that agents are suited to scaling tasks in trusted environments. Of the documents, emails and web pages an agent would read in your organization, how far does "trusted" extend? Think ahead to indirect prompt injection in Chapter 15 and the governance layer in Chapter 3.

Videos

• [Author] "How We Build Effective Agents: Barry Zhang, Anthropic" — AI Engineer
• [Author] "Tips for building AI agents" — Anthropic
• [Secondary] "The 'real' AI agent building guide Anthropic published itself (5 patterns)" — AI Tutor Lab (in Korean)
• [Review] "[DSBA Study] Agent AI Week 1 - Basic concepts and definitions of AI agents (1)" — DSBA Lab, Department of Industrial Engineering, Seoul National University (in Korean)

The titles and channels of the videos were checked through YouTube oEmbed on October 4, 2026. Titles and channel names in Korean are given in English, translated or romanized by the editor. The editor did not verify the content of the videos.

Further study

• The rest of the post. The full Appendix 2 is read in Chapter 7. The post points to Anthropic's cookbook for sample implementations.
• Later posts from the same company. Writing effective tools for agents — with agents (Chapter 7), Effective context engineering for AI agents (Chapter 9), Effective harnesses for long-running agents (Chapter 3).
• Maps by other authors. The three sections on planning, memory and tools in Weng's post, and the guardrails section of the OpenAI guide (pp. 24–31; read it with Chapter 15).
• Reading on in the surveys. AR survey, revised version, §2.4 (the whole design guide), §3.1.1 (Workflow Design and the reactive loop), §5.2.1 (pipelines crafted by people and pipelines built by LLMs). Harness survey, §2.2 (three scopes), §11.2 (the trade between capability and control).
• Within this book. Chapter 2 (formalizing "agent"), Chapter 3 (where stopping conditions, sandboxes and guardrails belong), Chapter 6 (the prototype of the agent loop), Chapter 7 (the augmented LLM's tools and MCP), Chapter 11 (evaluator-optimizer extended), Chapter 13 (orchestrator-workers and SOPs).

Sources

Core reading

• [L21] Erik S. (as credited on the current page; Erik Schluntz in the Internet Archive copy of 2024-12-20) · Barry Zhang. "Building effective agents". Anthropic Engineering blog, 2024-12-19. Original. Cited: all sections and Appendices 1 and 2.

Surveys

• [L22] Tianxin Wei et al. "Agentic Reasoning for Large Language Models: A Survey". Under review at TMLR (action editor "Accept as is" on 2026-09-15; OpenReview "Decision pending"). OpenReview, revised version (CC BY 4.0) · arXiv 2601.12538 (the arXiv version — its section and table numbers differ from the revised version. On v2 the abstract page's Comments field carries "Accepted by TMLR", but the body is the same as v1; compared on 2026-10-05). Cited: §2.1, §2.2 (the three optimization targets, the definition of $S$), §2.3 (rejected alternatives, Table 3), Table 4, §2.4 and Table 5, §3.1.1 and the Figure 2 caption, §5.2 and §5.2.1. Section and table numbers follow the anonymized review copy (generated 2026-09-01).
• [L01] Junjie Li et al. "Agent Harness Engineering: A Survey". 2026, unpublished preprint. Project page · PDF. Cited: §1, §2.1, §2.2 and Figure 1, §2.3, §11.2.

Side papers

• [S09] Lilian Weng. "LLM Powered Autonomous Agents". Lil'Log, 2023-06-23. Original (Agent System Overview; HuggingGPT under Component Three: Tool Use; Challenges).
• [S35] OpenAI. "A practical guide to building agents". 2025 (PDF created 2025-04-07). PDF (pp. 4, 6, 8, 16, 17 and 28–31).
• [S18] Jeremy Hadfield et al. "How we built our multi-agent research system". Anthropic Engineering blog, 2025-06-13. Original (architecture overview, token multiples, evaluation section).
• [L11] Noah Shinn et al. "Reflexion: Language Agents with Verbal Reinforcement Learning". NeurIPS 2023. arXiv 2303.11366 · Code (§3).
• [L12] Sirui Hong et al. "MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework". ICLR 2024. arXiv 2308.00352 · Code (abstract, §3).
• [S20] John Yang et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering". NeurIPS 2024. arXiv 2405.15793 (abstract, §5, Table 3).
• [L02] Shunyu Yao et al. "ReAct: Synergizing Reasoning and Acting in Language Models". ICLR 2023. arXiv 2210.03629.
• Aman Madaan et al. "Self-Refine: Iterative Refinement with Self-Feedback". 2023. arXiv 2303.17651 (only its classification in Table 4 of the revised AR survey is cited).
• [S10] Anthropic. "Introducing the Model Context Protocol". 2024-11-25. Original.
• [S19] Carlos E. Jimenez et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024. Leaderboard.

How this was checked. For L21, the current page as of October 4, 2026 was compared sentence by sentence with the Internet Archive copy of December 20, 2024 (then at a research path), which found the three changed places; the remaining sentences were the same. The figures of the original were not reproduced, and the editor drew this chapter's diagram from the text alone. The blog posts among the side papers were checked against the web originals, and the OpenAI guide and the papers and surveys against their PDFs, for section, page and table numbers and figures. The arXiv number of Self-Refine was checked on its abstract page on October 5, 2026. Authors, publication status, licenses and code repositories follow the bibliographic details in Appendix III. The $p^k$ table in section 5 of "The paper in detail", the sample sizes in the statistics section (normal-approximation formula), the power of the exact McNemar test (multinomial simulation, 10,000 to 20,000 runs), the Wilson intervals and the design effect were calculated by the editor in Python (numpy and scipy) from assumed values.

Chapter 2 · Core · ★★★

Chapter 2. The Agent's Head — Thinking, Acting, and Learning in One Frame

The question this chapter answers: how do we separate "thinking", "acting" and "learning" within one probability model?

Core reading. Tianxin Wei et al., "Agentic Reasoning for Large Language Models: A Survey", under review at TMLR (action editor "Accept as is" on September 15, 2026; OpenReview "Decision pending"). This book reads the revised post-review version posted on OpenReview (an anonymized review copy). Of its 146 pages, this chapter reads §1–§2.4, a little over 16 pages of the main text.

One-paragraph summary. The survey tries to set hundreds of papers on agents on top of one probability model. It writes the environment as a partially observable Markov decision process (POMDP) and the agent as a separate tuple. It writes the agent's policy as the product of a part that produces "thoughts" and a part that produces "actions", and the survey itself states that this factorization (Eq. 1) is only the chain rule, the content lying in three assumptions, A0, A1 and A2. It then divides every method by "what does it optimize?" Does it search the reasoning trace $z$ on the spot (in-context reasoning), revise a system state $S$ that persists across episodes (system-level optimization) or change the model parameters $\theta$ (post-training)? It crosses these three modes with three layers set by how dynamic the environment is (foundational, self-evolving, collective) to make a 3×3 grid, and summarizes in one table (Table 5) which mode to choose under which deployment conditions. The papers of Chapter 4 through Chapter 13 of this book all sit in some cell of this grid.

Presenter's guide

A 50-minute plan. The 10 minutes after the presentation are for discussion.

Must read. §2.2 Preliminaries. It is the center of this chapter and runs a little over 4 pages. In the post-training paragraph, though, only the first two sentences, which say that $\theta$ is optimized to align the policy with long-horizon rewards, are required; Eqs. 3 and 4 and the details of GRPO go to the optional reading below. Next come Table 3 in §2.3 and Table 5 in §2.4. If you show only one figure, show Table 3, not Figure 1. Every paper in the later chapters comes back to one of its nine cells.

Optional reading. §1.1 Survey Methodology (p. 5, for those interested in statistics). The makeup of the 866 entries by year and by category is printed in numbers, and section 8 of this chapter practices on those numbers. The post-training paragraph of §2.2 (Eqs. 3 and 4, GRPO) is recommended for readers familiar with reinforcement learning or who want to know more. This book does not go deep into post-training.

What to cut from the presentation. The detailed subsections of §3–§5 are read in the later chapters (Chapter 5, Chapter 7, Chapter 8, Chapter 11, Chapter 12, Chapter 13), §6 in Chapter 14, §7 in Chapter 17 and Chapter 18, and §8–§9 in Chapter 20. Skip the guide to the organization on p. 6 and the table of contents on pp. 7–8.

Do not skip or combine this chapter. The "Back to the surveys" sections of Chapter 4 through Chapter 13 all presuppose this chapter's notation and grid. If your group skipped Chapter 1, introduce in the first 3 minutes only the distinction that "a workflow calls the model along fixed code paths, while an agent sets its own path". Point out that this distinction (who sets the path) and this chapter's (what is optimized) answer different questions.

When you quote the numbers. This survey has almost no performance figures. The numbers to carry over are about the makeup of the corpus (§1.1) and the row counts of tables, and those describe the literature the authors chose, not a sample of the whole field. Give the denominator, as in "of the 866 bibliography entries in this survey", and name the version (the revised version on OpenReview).

Building intuition

What to divide the "head" by

Chapter 1 looked at agents through control flow (who decides the next step), and Chapter 3 looks at the execution infrastructure that wraps the model. This chapter looks at what lies between: how the model thinks, acts and learns.

The survey uses a single question: "what does it optimize?" (§1). Agents that are "good at making plans" can search on the spot with prompts, revise a store of past plans or change their weights with reinforcement learning. Divide by a list of capabilities and the three go into one cell; divide by optimization target and they go into three. The survey chooses the latter.

The editor's bridge — from hidden Markov models to POMDPs

Instead of analogies, the survey states in the first paragraph of §2.2 that it will unify the capabilities of the three layers "under a single control-theoretic framework."

The paper in detail

1. Scope and question — Figure 1 and Table 1 (§1, §2)

Figure 1. The figure on p. 2 has four tiers. The top band is a flow in which a User proposes (Propose) a Task, an Agentic Reasoning System solves it (Solve), and the experience generalizes (Generalize) to Future Tasks. The second band, under the title "LLM Reasoning → Agentic Reasoning (▷§Section 2)", draws three pairs of transitions. Paradigm goes from Static Input to Dynamic Context, Compute from Passive to Interactive and Learning from Pre-trained to Evolving. "And More …" is attached at the right end. The four panels below are the body: Foundational (Complex Planning · Tool Use · Web Search, §3), Self-evolving (Feedback Loop · Agentic Memory · Self-Evolving, §4), Collective Multi-agent (Role Assigning · Collaboration · Co-evolving, §5) and Applications and Benchmarks (§6 and §7; healthcare, finance, legal, education, robotics, science, software, gaming and web).

The caption says the three layers are arranged "in order of increasing environmental dynamics": stable environments (foundational), environments with feedback and accumulated state (self-evolving) and environments shared by several agents (collective). This is the survey's first axis.

Table 1 — LLM reasoning against agentic reasoning. The opening of §2 contrasts the two: traditional LLM reasoning raises performance by "scaling test-time computation," while agentic reasoning emphasizes "scaling test-time interaction."

Table 1 of the paper. The original links each of the two pairs in a dimension with "↔".

Do not read "single pass" as meaning that the network's forward computation runs only once. Even while generating one answer, the forward computation repeats for every token (Chapter 0, section 1), and methods that scale test-time computation make exactly that generation longer. What the table separates is whether feedback from the environment arrives in the middle of generation. If the model takes the input and goes all the way to an answer with no response from outside, it is a single pass; if it emits an action, receives an observation and continues generating, it is multi-step (this gloss is the editor's reading).

The right-hand cell of the "Learning" row does not mean only retraining the weights. A good part of this survey's self-evolution leaves the weights alone and revises memory or tool libraries (four of the six representative families in Table 4 change $S$. Counting the 56 rows of Tables 9 and 10, the method tables of the self-evolving layer, gives $S$ 25 rows, $\theta$ 20 and $z$ 11 — section 5 of this chapter). The "learning" in this chapter's title has that broad sense too.

2. How it differs from neighboring surveys (§2.1, Table 2)

The survey compares itself with nine closely related surveys in one table, Table 2. The columns are Formalisation, Planning, Tool use, Retrieval/RAG, Feedback, Memory, MAS, MAS training, Applications, Benchmarks and, last, "Per-system optim. target". There are three marks: • (substantive treatment, a dedicated section or more), ◦ (partial treatment) and – (not covered). In the last column, only this survey gets a •; three of the nine (Lin et al. 2025c, Fang et al. 2025a, Gao et al. 2026) get ◦ and the other six get –. This column is where the authors claim the biggest difference. The caption states that "each mark is our reading of the survey's own section structure."

§2.1 treats the table carefully as the authors' own judgment. Of Ke et al. (2025a), it writes itself that since that survey already takes interactive settings as one of its axes, "it would be inaccurate to characterize this literature as confined to single-forward-pass reasoning", and it states the differences narrowly ("we state them rather than overclaiming"). Of Zhang et al. (2025f), whose coverage overlaps most, it concedes that it already treats parameter-level multi-agent training in depth. It also gives a reason for each of the three ◦ marks in the last column. For example, Gao et al. (2026) did tabulate a per-method "what evolves" matrix, but only over 24 representative methods, and its axes do not separate the reasoning trace from the surrounding context. The differences among the three surveys come back in "Side papers" in this chapter. Note, though, that the marks in Table 2 were assigned by the authors, who are a party to the comparison, not reassigned by a third party on the same criteria.

3. Two tuples — separating the environment from the agent (§2.2)

The first paragraph of §2.2 is headed "Formalizing Agentic Reasoning: A Latent-Space View." The problem is stated in one sentence: "Standard approaches often conflate the agent's context with the environment state." So the survey keeps the two syntactically separate and writes them as two tuples.

The environment.

$$E=⟨X,O,A,T,\Omega ,R,\gamma ⟩$$

$X$ is the latent state space the agent cannot observe, $O$ the observation space (user queries, API returns and so on), $A$ the external action space (tool invocations, final answers and so on), $T$ and $\Omega$ the transition and observation kernels, $R$ the reward and $\gamma \in (0,1)$ the discount factor.

The agent.

$$G=⟨Z,M,{\pi }_{\text{reason}},{\pi }_{\text{exec}},U⟩$$

$Z$ is the reasoning trace space. The survey's wording is "latent plans, optionally verbalized as chain-of-thought". $M$ is the agent's internal memory/context space, ${\pi }_{\text{reason}}$ and ${\pi }_{\text{exec}}$ are the reasoning and execution policies, and $U$ is the system-update operator that appears later in Eq. 5.

Why separate them. $Z$ and $M$ are design choices of the agent. The environment neither observes them nor is changed by them, so two agents with different $Z$ or $M$ face the same $E$. The survey's answer is that folding them into the environment tuple obscures the very distinction the survey is organized around.

Put in the language of experimental design, the environment is the subject population and the agent is the treatment. If you write the treatment down as an attribute of the subjects, it no longer makes sense to say you are comparing two treatments on the same population (this rendering is the editor's reading).

The two tuples and assumption A0 from §2.2 of the revised AR survey, redrawn. The editor drew the diagram.

History and memory. At time $t$ the agent conditions on the history $h_t=(o_{\le t},z_{<t},a_{<t})$. The inequality signs give the order: observe $o_t$, emit $z_t$, emit $a_t$. The history can be summarized by an internal memory state $m_t\in M$.

4. Eq. 1 and the three assumptions (§2.2)

The policy is factored like this.

$${\pi }_{\theta }(z_t,a_t\mid h_t)=\underset{\text{Internal Thought}}{\underbrace{{\pi }_{\text{reason}}(z_t\mid h_t)}}·\underset{\text{External Action}}{\underbrace{{\pi }_{\text{exec}}(a_t\mid h_t,z_t)}}$$(Eq. 1)

The very next sentence takes the weight off the equation itself.

Eq. 1 is an application of the chain rule and carries no content on its own; it holds for any joint distribution over $(z_t,a_t)$.

Eq. 1 is $p(z,a\mid h)=p(z\mid h)p(a\mid h,z)$, and it holds for any joint distribution. The survey says the content lies not in this equation but in the modeling commitments that accompany it, and that it states them explicitly because different families of systems accept different subsets of them.

The three assumptions of §2.2, set out as a table.

A0's "$T$ and $\Omega$ are functions of $a_t$ alone" is the original sentence ("T and Ω are functions of $a_t$ alone") carried over as it stands, but read literally it says that transitions and observations do not depend on the state either, which does not fit the POMDP defined just before. The transition still sets the next state from the current latent state and the action, and observations still come out according to the state. What the sentence rules out is a path by which, of what comes from the agent's side, the thought $z_t$, rather than the action $a_t$, affects the environment directly. For example, if you write the transition as $T(x_{t+1}\mid x_t,a_t)$, $z_t$ does not enter the conditioning (the editor introduced this notation; the original has no equation of this form).

Rewrite the three assumptions in the language of conditional independence and you can see what is at stake.

• A1 is $z_t⟂h_t\mid m_t$. Every system that summarizes an old conversation and puts the summary into the context relies on it. It breaks when the summary leaves out something the next thought needs. This is the subject of Chapter 12.
• A2 is $a_t⟂h_t\mid z_t$. All the information the action needs passes through the thought. As Chapter 6 shows, ReAct puts the object number from an observation into an action without passing it through a thought, so A2 does not hold.
• A0 is not a conditional independence but an assumption of ordering and blocking. The thought does not reach the environment, and of what the agent emits, the environment sees only the action.

That A2 is "not generally assumed" shows what the formalization is good for. Under the same Eq. 1, structures that accept A2 part ways with those that do not, and the difference lines up with differences in design. The survey names ReAct as an example where A2 breaks, but for the side where it holds it writes only "plan-then-execute architectures (Section 3)". A good example of the side where it holds is ReWOO in §3.1.1 (which separates the observation module from the reasoning module). When A2 holds, observation text cannot reach an action without passing through the plan, which connects it to security as well (Chapter 15). Both connections are the editor's reading.

The last line of the formalization is maximizing the expected discounted return $J(\theta )={𝔼}_{\tau }\left[{\sum }_{t\ge 0}{\gamma }^t{r}_t\right]$.

5. The three optimization targets — $z$, $S$, $\theta$ (§2.2)

§2.2 writes four paragraphs in this order: in-context reasoning, post-training, collective intelligence and self-evolution. Three of them produce the three modes. In-context reasoning optimizes $z$, post-training $\theta$ and self-evolution $S$. Collective intelligence is about the scope over which those three modes are applied to several agents. This chapter looks at the three modes in a row first and puts collective intelligence at the end.

In-context reasoning — search at inference time. The parameters $\theta$ are frozen. The agent searches over $Z$ and picks a reasoning trajectory that maximizes a heuristic value function $\widehat{v}(h_t,z)$. Inference is modeled as selecting a trajectory $\tau =(h_0,z_0,a_0,h_1,z_1,a_1,\dots )$. ReAct is greedy decoding over alternating thoughts $z$ and actions $a$. Tree of Thoughts (ToT) and the MCTS (Monte Carlo tree search) family treat partial thoughts as nodes $u$. The survey says these families do not share an objective and states them separately rather than collapsing them into one expression. Below, ${\widehat{v}}_{\varphi }$ is a value estimator that scores nodes or steps, and $\varphi$ is that estimator's parameter (it may be an LLM prompted to give scores or a trained verifier).

ToT-style search with pruning keeps, under a node budget $B$, only the children whose value satisfies ${\widehat{v}}_{\varphi }(u)\ge \kappa$, and returns the highest-scoring leaf (${argmax}_{u\in L_H}{\widehat{v}}_{\varphi }(u)$) among the set $L_H$ of leaves reachable at depth at most $H$. It is not a sum along the path. MCTS estimates values by repeating rollouts (the original says simulation) that play out from a node to the end. The sum of the results of the rollouts that passed through the branch from node $u$ to child $u^{\prime }$ is $W(u,u^{\prime })$, the number of times that branch was passed is $N(u,u^{\prime })$, and the next branch is chosen by an upper-confidence rule that adds to their mean $Q(u,u^{\prime })=W(u,u^{\prime })/N(u,u^{\prime })$ a bonus that grows for less-visited branches. A path sum is the right object only for process-reward and step-verifier aggregation.

$${\tau }^{\star }\in {argmax}_{\tau \in {\Gamma }_H}{\sum }_t{\widehat{v}}_{\varphi }(u_t)$$(Eq. 2)

${\Gamma }_H$ is the set of feasible trajectories of depth at most $H$ (a different symbol from the transition kernel $T$), and ${\widehat{v}}_{\varphi }$ is a step-level verifier or process reward model. The survey explains that maximizing without constraint makes the sum unbounded, so the problem is well posed only when restricted to ${\Gamma }_H$. In statistical terms, ToT is an optimization that picks a leaf by heuristic scores, and MCTS is sequential decision-making that picks while estimating values by repeated sampling. Chapter 5 looks at this in detail.

Post-training — policy optimization. $\theta$ is revised to fit long-horizon rewards $r_t$ such as correctness or safety. Examples are reasoning models like DeepSeek-R1 and systems like Search-R1 that learn to search with reinforcement learning. PPO is the standard, and methods in the GRPO (Group Relative Policy Optimization) family are widely used for reasoning tasks. GRPO standardizes the rewards of $G$ outputs for the same prompt within the group to make the advantage ${\widehat{A}}_i=(r_i-\mu )/(\sigma +\delta )$ (Eq. 4). $\mu$ and $\sigma$ are the mean and standard deviation of those $G$ rewards, and $\delta >0$ is a small constant that keeps the division from blowing up when $\sigma$ is 0. The advantage is a value that says how much better this output is than a baseline: if it is positive, the probability of producing that output goes up, and if negative, down. The key is what to take as the baseline. PPO separately trains a value network (a neural-network approximation of the value function in section 4 of Chapter 0) that predicts the expected return for each state, and uses it as the baseline. GRPO drops that value network and takes the mean of sibling answers to the same problem as the baseline. In statistical terms, it is a standard score within the group. In this book, post-training comes up in Chapter 7, Chapter 10 and Chapter 16.

Self-evolution — the meta-learning loop. Self-evolving agents optimize the system itself across episodes $k=1,\dots ,K$. Let $S_k$ be the evolvable system state (explicit memories, tool libraries, code and so on); the generic meta-update rule is this.

$$S_{k+1}\leftarrow U(S_k,{\tau }_k,F_k)$$(Eq. 5)

${\tau }_k$ is the trajectory of episode $k$, and $F_k$ is environmental feedback (rewards, execution errors). The survey divides self-evolution into three branches by the content of $S$.

The "Self-Evolving Agents" paragraph of §2.2, set out as a table.

To a reader familiar with statistics, Eq. 5 reads as the general form of sequential updating. Just as Bayesian updating makes a posterior from a prior and data, $U$ makes the next state from the current state, a new trajectory and feedback. The difference is that $S$ is not a probability distribution but a document or code a person can read (this comparison is the editor's).

Collective intelligence — widening the three modes to multiple agents. This paragraph does not add a fourth mode; it widens the scope over which the three modes apply. It extends the single-agent formulation to a decentralized partially observable setting (Dec-POMDP). Agent $i$'s observation $o_t^i$ includes messages $c_{t-1}^{-i}$ its peers produced at the previous time step, through a communication channel $C$. The survey stresses that communication is not merely signal transmission but an extension of reasoning: one agent's external action becomes a prompt that triggers another agent's internal reasoning. The three modes split here too. AutoGen and CAMEL are static role-playing with fixed policies; GPTSwarm and MaAS optimize the coordination structure with $\theta$ frozen; and agents trained with PPO or GRPO update $\theta$. Chapter 13 picks up this paragraph.

Summed up as three modes. The three optimization variables that emerge this way are the axis that divides the whole survey. The survey stresses that this is not a binary in-context/post-training split.

The "Three Optimization Modes" paragraph of §2.2, set out as a table.

Why separate $S$ from $\theta$ at all. The abstract, §1 and §2.2 repeat the claim. Research that optimizes prompts, workflows and communication graphs without touching weights is growing fast, and putting it under "post-training" because it has an optimization loop mixes two regimes with different cost profiles, deployment requirements and failure modes. The abstract's phrase is "otherwise misfiled as post-training."

Differences between versions arise here too. A Korean-language post that introduced the arXiv manuscript (AI Native Insight by Lee Seung-hyun, in Korean) presents the survey as "three layers × two optimization modes (in-context versus post-training)" (our translation) and explains Reflexion on the in-context side. In the revised version there are three modes, and Reflexion is classified as $S$ (Table 4, Table 9). Keep this difference in mind when you look at material that covers the pre-review manuscript (including the two secondary videos for this chapter). The authors' affiliations also differ from the title of the same post: of 29 authors, 22 are at UIUC, with 2 at Meta, 2 at Amazon, 1 at Google DeepMind, 1 at UCSD and 1 at Yale (the revised PDF is anonymized, so these were counted from the affiliations on p. 1 of the arXiv manuscript).

6. The 3 layers × 3 modes grid (§2.3, Table 3, Table 4)

§2.3 begins "A taxonomy is a choice, not a discovery," and writes that a survey owes the reader the alternatives it rejected and the cost of the one it kept.

Three rejected alternatives.

• Capability lists (planning / tool use / memory / …). Easy to look things up in, but they say nothing about how a capability is obtained. Prompted planning and planning learned by reinforcement learning sit side by side.
• Architectural composition (single-agent / tool-based / multi-agent). Orthogonal to adaptation, so self-evolution has no place in it.
• Learning-paradigm-first (prompting / SFT / reinforcement learning). It loses the environmental-dynamics dimension, so single-agent and multi-agent reinforcement learning fall into one bucket.

The chosen axis. "Environmental dynamics × optimization target". It separates where the difficulty comes from from what is spent to address it ($z$, $S$, $\theta$).

Table 3 — the grid.

Table 3 of the paper. "–" marks a family the grid predicts but for which no established literature was found.

Two gaps. Saying that "an absence of work is itself a finding", the survey marks two empty cells.

• Gap (i), self-evolving × $\theta$, tool creation. The authors are aware of no established line of work that updates $\theta$ to make a model better at "writing new tools". Tool "selection" is a different problem and is covered under foundational × $\theta$ (§3.2.2).
• Gap (ii), collective × $\theta$, shared memory. Multi-agent memory is optimized almost exclusively at the system level, and the authors are aware of no established work that post-trains agents to read and write a shared store (§5.3.2).

Boundary cases. The three layers are not disjoint. A planner that retrieves past plans is foundational in capability but self-evolving in mechanism. Tool creation is a foundational capability and also a permanent expansion of $A$, that is, Procedural Evolution. The survey does not force such intersections into one cell; it labels them as intersections.

Table 4 — the three vocabularies of self-evolution. The content of $S$ (verbal, procedural, structural) is the primary cut, and mechanism (§4) and temporal scope (intra-test-time versus inter-test-time, the terminology of Gao et al. 2026) are cross-cuts.

Table 4 of the paper. According to the caption, Reflexion's critique store persists across trials of one task, which makes it $S$ under the primary cut but intra-test-time under the temporal one.

Set the first and second rows side by side and you can see where the three-mode distinction does its work. Self-Refine and Reflexion are both "generate–critique–revise" loops. Self-Refine revises only the trace and leaves nothing behind, so it is $z$; Reflexion writes its critiques to notes that persist across episodes, so it is $S$ (§4.1.1). §3.1.2 applies the same distinction to Reflexion and Reflect-then-Plan (which refines only a belief state within an episode, $z$) and writes: "both are 'reflection', and only one of them accumulates".

7. The design guide — which mode under which conditions (§2.4, Table 5)

§2.4 opens by saying that the optimization axis is useful only if it answers a question a system designer actually faces, and gives in one table, for each deployment constraint, the mode the current evidence favors and what to watch for. It also attaches the caveat that these are readings of a fast-moving and often confounded literature rather than settled results.

Table 5 of the paper.

Four of the six rows recommend $S$. The warning in the last row (the write that caused a failure may be many episodes old) comes back in Chapter 19, when we trace the cause of a failure backward, and in Chapter 20, when wiki pages written in advance go stale. A concrete case of the second row's "an eviction and validation policy is required" is Chapter 12.

The "Why" column is mostly the logic of cost and operations, with no performance comparisons in numbers. Reading this table as a checklist for design judgment rather than as a prescription confirmed by experiment fits the survey's own caveat better.

8. Rereading with a statistician's eye

The corpus in §1.1 is printed in numbers. The survey states that it was not built from a single systematic query, so it reports the process actually followed rather than reconstructing a PRISMA-style protocol after the fact. It monitored five arXiv listings, the ACL Anthology, OpenReview and the proceedings of major conferences, closed at a primary cutoff in January 2026, and repeated citation-chaining until new candidates were mostly already in the set. The inclusion criteria are work that treats reasoning as unfolding through interaction and reports a method or an evaluation artifact; the exclusion criteria are single-forward-pass prompting without external interaction, pure long-context modeling and position papers.

The numbers in "Corpus profile" of §1.1, set out as a table. The editor added up the totals to check them.

Both columns add up to 866. Applications and benchmarks together come to 407 entries, 47.0% of the total. This is the number the survey calls "roughly half", and the survey reads it as reflecting how quickly the evaluation layer is growing rather than as a bias in its search.

The year column grows steeply: 2.13 times from 2023 to 2024 and 1.46 times from 2024 to 2025. The 56 entries for 2026 combine the primary cutoff (January 2026) with a small number of later additions, so they are left out of this comparison. One sentence in the definition of years is open to more than one reading. It says that for entries with no peer-reviewed venue the year was taken from the arXiv identifier, and that "the four entries that do carry one keep their venue's year." Yet the bibliography shows several entries with venues, such as CoT (NeurIPS 2022) and Toolformer and Reflexion (NeurIPS 2023). What "the four entries" refers to cannot be settled from the text alone.

The relation between the inclusion criteria and the 866 entries is also worth noting. The 866 are the whole bibliography. By the exclusion criteria, prompting without external interaction is not under review, yet CoT, its leading example, is cited in §1 and §3.2.1 and is among the 866. This chapter reads the 866 as "the citation list", not as "the works under review that passed the inclusion criteria".

Exercise 1 — count the "optimization target" column of the method tables. The survey's list of contributions says "the framing is used: every method table records which of the three a system optimizes." Counting the last column of Tables 6–11 gives the following (the editor counted row by row from the PDF text).

(a) Combine by layer into a 3×3 contingency table and find the shares of the three modes in each layer. (b) Run a chi-square test of the null hypothesis "layer and mode are independent". Run the same test at the level of tables (6×3). Why do the two results differ? (c) Some systems, such as ReAct (Tables 7, 8 and 9), appear in more than one table. Does removing duplicates within the same layer change the conclusion? (d) What is the $p$ value of this test a statement about?

Exercise 2 — can you divide the rows of a table by the number of corpus entries? Set the number of corpus entries by category beside the number of rows in the method tables: planning 23/111, tools 17/70, search 17/21, feedback 26/27, memory 30/61, multi-agent 29/118. (a) Find each ratio. (b) Can you say "96% of feedback papers made it into the table"? Answer by comparing how the numerator and denominator are defined.

9. Limitations the paper states

• Known coverage gaps (§1.1). It names four: non-English literature; industrial systems without public write-ups; reinforcement-learning infrastructure and systems papers (cited only where they change what an agent can do); and the game-playing and negotiation literature.
• The scope boundary of agentic search (§1.1). Much of the RAG literature lies outside the inclusion criteria. When you read Chapter 8, remember that this survey does not cover RAG as a whole.
• The cost of the taxonomy (§2.3). Methods that adapt structure and parameters together must be assigned to the dominant mechanism, and reasonable people can disagree about that judgment. The survey writes that this is a real limitation of its axis, one a capability-list taxonomy does not have.
• The status of the design guide (§2.4). It states that Table 5 is a reading of a fast-moving and often confounded literature, not a settled result.
• Anonymized review copy. The address of the companion repository (paper collection, Tables 2–5, benchmark inventory, citation-audit log) is omitted for anonymous review and is to be included in the camera-ready version (§1.1, footnote 1). The authors' paper-list repository, to which p. 1 of the arXiv version points, is the likely candidate, but as of October 5, 2026 it holds only lists of papers and benchmarks and one PDF of presentation slides, not Tables 2–5 or the citation-audit log.

If we add one limitation the paper does not state, it is that assumptions A0–A2 are not used again outside §2.2. The names "A0", "A1" and "A2" could not be found outside their definitions in §2.2 (the editor searched the PDF text). The body of §4.2 does not discuss long-horizon failures under the name of A1 either. By contrast, the optimization targets $z$, $S$ and $\theta$ are recorded in every row of Tables 6–11. It can also be read that in this survey, the formalization that bears load is the three optimization targets rather than the three assumptions. Discussion question 1 takes this up.

Side papers — how they differ from the main paper

The list first proposed in the study group has no side papers for this chapter. Instead, we contrast three neighboring surveys that this survey compares itself with in Table 2, and three core papers from other chapters of this book that each illustrate one of the three modes. All descriptions are limited to what this survey writes about that literature.

A Survey of Frontiers in LLM Reasoning (Ke et al. 2025). The survey gives this as an example of a survey of LLM reasoning that reaches into interactive settings. It is organized as the product of reasoning regime (inference scaling versus learning to reason) and system architecture (standalone LLM / single-agent / multi-agent). The difference is the axis. Ke et al. divide by architecture, this survey by optimization target. The survey notes that Ke et al. has no memory section, and explains separately that not covering benchmarks is a scope decision Ke et al. themselves state (§2.1).

The Landscape of Agentic Reinforcement Learning for LLMs (Zhang et al. 2025). The work the survey admits overlaps most in coverage. It divides by the product of a capability axis and a task-domain axis, and as its formalization sets the single-step MDP of conventional LLM reinforcement learning against the POMDP of agentic reinforcement learning. Its per-system tables record the backbone model, the reward type and the reinforcement-learning algorithm. This survey records "what does it optimize?" in the same place. That is why the two part ways in the last column of Table 2.

LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios (Zhao et al. 2025). Another work the survey counts as close. It puts forward a unified formal description of the reasoning process and divides into single-agent / tool-based / multi-agent. There are two differences. Its axis is architectural composition, so two systems of the same composition go into the same cell even if one searches $z$ and the other changes $\theta$. And it restricts itself to training-free frameworks, leaving SFT and reinforcement learning out of its classification. In terms of this survey's grid, the whole $\theta$ column is missing.

Chain-of-Thought (Wei et al. 2022 · Chapter 4 of this book) — the prototype of $z$. Defining $Z$, the survey writes "optionally verbalized as chain-of-thought". At the same time it distinguishes CoT from agentic reasoning: CoT assumes a static context and a short horizon (§1) and works in a closed world confined to the model's internal knowledge (§3.2.1). CoT itself does not appear as a row in Tables 6–11. In this book's reading, CoT is a $z$ whose only external action is the final answer, the closest neighbor of the "single forward pass" that A0 separates off.

Reflexion (Shinn et al. 2023 · Chapter 11 of this book) — the representative of $S$. The survey gives Reflexion as the first example of Verbal Evolution (§2.2). In Table 6 and in Table 9 (self-reflective, I-a) alike, its optimization target is $S$, contrasted with Self-Refine ($z$), the same generate–critique–revise loop (§4.1.1). Compare this with the post that introduced the pre-review manuscript and explained Reflexion on the in-context side, and you can see where the three-mode distinction does its work.

Toolformer (Schick et al. 2023 · Chapter 7 of this book) — the representative of $\theta$. The survey lists Toolformer in Table 7 (training method "Self-sup. + SFT") and Table 8, as $\theta$ in both. §3.2.2 explains that it generates, verifies and retains useful API calls within unlabeled text and fine-tunes on the filtered data. Among "agents that use tools", ReAct (first row of Table 7, $z$) and Toolformer sit in different columns. The "same cell by capability list, different cells by optimization target" of section 6 is these two rows.

Back to the surveys

Since the survey itself is the subject of this chapter, this section serves as a place to contrast it with the Harness survey of Chapter 3 (H for short; this survey is AR). AR looks at the agent's head: how the model plans, uses tools and improves through feedback and memory. H looks at the body and the fence: the execution environment, tool contracts, context management, orchestration, observability, verification and governance outside the model.

What is taken as the unit of analysis

The first sentence of H's abstract says that reliable task execution depends on the infrastructure that surrounds the underlying model, the harness, as well as on the model itself. §2.4 defines the harness as "the engineered wrapper that turns model calls into bounded, stateful, tool-mediated task execution", and makes clear that the unit of analysis is "not the foundation model or prompt alone."

Set the scope statements of the two surveys side by side.

The editor set the scope statements of the two surveys side by side.

Both exclude standalone prompting techniques, and they part ways on post-training. The side that looks at the head takes research that changes weights as one axis; the side that looks at the body puts foundation-model training out of scope, keeping only training that meshes directly with harness mechanisms.

Overlap seen through the tables of contents

Put the tables of contents against each other and the overlaps are tools (AR §3.2 against H §4), memory (AR §4.2 against H §5.4–5.5), multi-agent (AR §5 against H §6.3) and evaluation (AR §7 against H §8). Even on the same topic they ask different things. Take tools: AR asks how tool use is learned, H how tools are connected. For memory, this book reads the classification from AR (Table 10) and the implementation layers from H (Chapter 12). The places with no overlap at all tell you more. The execution environment and sandbox (H §3) and observability (H §7) have no section in AR, and the distinction of optimization modes, assumptions A0–A2 and the search objectives are absent from H.

Laying the two maps over each other

In the editor's reading, the two surveys meet at $S$. The examples of $S$ the survey gives — explicit memories, tool and skill libraries, prompts, communication topologies and the agent's own code — become, carried over to the layers of the Harness survey, objects of context (C), tools (T) and orchestration (L) respectively. What is "learned state" on the head side becomes "infrastructure to design and operate" on the body side. The warning in the last row of Table 5 (persistent state creates its own audit problem) can also be read as saying that this state is, on the harness side, an object of observability (O) and governance (G). Neither survey writes this correspondence out directly. Laying $S$ over each layer of ETCLOVG in Chapter 3 makes good review.

Common misreadings

Misreading 1. "Eq. 1 is this survey's key finding." The survey itself writes that Eq. 1 is an application of the chain rule and carries no content on its own (§2.2). It holds for any joint distribution. The content lies in assumptions A0–A2 and in the three modes, set by what is optimized.

Misreading 2. "In an agent, the action is decided by looking only at the thought (A2)." A2 is not generally assumed (§2.2). It holds only for plan-then-execute architectures, and breaks in interleaved architectures such as ReAct, where the action is conditioned on the raw observation as well. What the survey states is "assumed by every system we survey" is A0, and judging from the phrase "Unlike A0 and A1" in the A2 entry, it places A1 on the generally assumed side too. A2 is the only one not generally assumed.

Misreading 3. "If the weights do not change, it is all prompting, that is, in-context reasoning." Correcting this is the point of the revised version. Revising memory, tool libraries, prompts or communication topologies across episodes with the weights frozen is the $S$ mode (§2.2, Three Optimization Modes). Whether anything persists is what separates $z$ from $S$. Self-Refine is $z$, Reflexion $S$ (Table 4).

Misreading 4. "If there is an optimization loop, it is post-training." This is the classification the survey opposes most strongly. It writes that putting research that optimizes prompts or communication graphs under "post-training" conflates regimes with entirely different cost profiles, deployment requirements and failure modes (abstract, §1, §2.2). Conversely, Self-RAG, often introduced as a prompting technique, fine-tunes its generator on a corpus with reflection tokens, so it is $\theta$ (§3.3.3, Table 8).

Misreading 5. "The 866 entries are a sample of the field, and since applications and benchmarks are half, so is the field." The survey itself states that it did not dress up a PRISMA-style protocol after the fact (§1.1). The 866 are the whole bibliography cited in the body, and the inclusion criteria put much of the RAG literature out of scope. This is the makeup of this survey's citations, not a sample of the whole field.

Misreading 6. "Table 5 is a prescription verified by experiment." The survey writes that Table 5 is a reading of a fast-moving and often confounded literature, not a settled result (§2.4). It has no performance figures, and the multiple-roles row directly warns that reported comparisons are confounded.

Discussion questions

1. Is the formalization decoration, or does it bear load? Eq. 1 is the chain rule, and assumptions A0–A2 are not used again by name outside §2.2. By contrast, $z$, $S$ and $\theta$ are recorded in every method table. If you were a reviewer, how would you ask the authors to show that this formalization bears load? Design an experiment that compares a structure where A2 holds with one where it breaks, on the same model and the same task, and measures how much the action wobbles when a sentence irrelevant to the observation is inserted (treatment, outcome measures, sampling unit, confounders). We use it again in Chapter 6 and Chapter 15.

2. To whom is this classification useful? If you brought an agent into your work, which row of Table 5 would apply? Put the recommendation of $S$ when "Auditability or regulatory review required" together with the warning "the write that caused a failure may be many episodes old". How would you manage the records that pile up in $S$? This connects to wiki maintenance in Chapter 20.

3. Where is the boundary between head and body? The same memory store is the optimized state $S$ in this survey and an object of implementation in the context layer (C) in the Harness survey. How does the answer to "what to fix when performance gets worse" change with the point of view? Check your answer in Chapter 3 and Chapter 19.

Videos

• [Author] "LLM Agents MOOC | UC Berkeley CS294-196 Fall 2024 | LLM Agents: History & Overview by Shunyu Yao" — Berkeley RDI
• [Review] "[DSBA Study] Agent AI Week 2 - Cognitive Architectures (1)" — DSBA Lab (in Korean; a different framework, CoALA, for contrast)
• [Secondary] "[AI papers, January 2026] Agentic reasoning: a next-generation AI roadmap linking thinking and acting / Agentic Reasoning for Large Language Models" — Easy Paper: papers to listen to at ease (in Korean; covers the pre-review version)
• [Secondary] "2601.12538 - Agentic Reasoning for Large Language Models" — AI Paper Cast (covers the pre-review version)

The speaker in the first video, Shunyu Yao, is not an author of this survey but the first author of ReAct and ToT.

The titles and channels of the videos were checked through YouTube oEmbed on October 4, 2026. Titles and channel names in Korean are given in English, translated or romanized by the editor. The editor did not verify the content of the videos.

Further study

• The rest of the paper. §3.4, §4.4 and §5.4 (what the evidence for each layer supports), §7.3 on the limits of benchmarks, §9 Conclusion and Appendix A, the broader impact statement.
• The list and the reviews. The paper-list repository. The public reviews and author responses on the OpenReview forum record how the survey was revised.
• Reading on in the surveys. Harness survey, §2.2 and §2.3. Zhang et al. 2025, whose coverage overlaps most.
• Within this book. Chapter 4 (getting the model to write out $z$), Chapter 6 (where A2 breaks), Chapter 7 (learning tools through $\theta$), Chapter 11 (Verbal Evolution of $S$), Chapter 12 (A1 and memory), Chapter 3 (the body and the fence).

Sources

Core reading

• [L22] Tianxin Wei et al. "Agentic Reasoning for Large Language Models: A Survey". Under review at TMLR (action editor "Accept as is" on 2026-09-15; OpenReview "Decision pending"). OpenReview, revised version (CC BY 4.0) · arXiv 2601.12538 (the arXiv version — its section and table numbers differ from the revised version. On v2 the abstract page's Comments field carries "Accepted by TMLR", but the body is the same as v1; compared on 2026-10-05) · List repository. Cited: §1, §1.1, §2–§2.4 (Figure 1, Tables 1–5, Eqs. 1, 2, 4 and 5, assumptions A0–A2), §3.1.2, §3.2.1, §3.2.2, §3.3.3, §4.1.1, the optimization-target column of Tables 6–11, the bibliography. Section and table numbers follow the anonymized review copy (generated 2026-09-01).
• [L23] AI Native Insight by Lee Seung-hyun (posting as epoko77). "Agentic Reasoning for Large Language Models (by Meta, Google deepmind, Amazon, etc.)". Blog, 2026-01-22. Original (in Korean). Cited as an introduction to the two-mode scheme of the pre-review manuscript. The affiliations in the title are left as in the original (see the text for the actual affiliations); the final word of the title, Korean for "etc.", is translated.

Surveys

• [L01] Junjie Li et al. "Agent Harness Engineering: A Survey". 2026, unpublished preprint. Project page · PDF. Cited: abstract, §1, §2.2, §2.3, §2.4, §2.5.

Side papers

• Zixuan Ke et al. "A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems". 2025. arXiv 2504.09037 (bibliographic details as in the AR survey's reference list, Ke et al. 2025a).
• Guibin Zhang et al. "The Landscape of Agentic Reinforcement Learning for LLMs: A Survey". 2025. arXiv 2509.02547 (bibliographic details as in the AR survey's reference list, Zhang et al. 2025f).
• Bingxi Zhao et al. "LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios". 2025. arXiv 2508.17692 (bibliographic details as in the AR survey's reference list, Zhao et al. 2025b).
• [S01] Jason Wei et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models". NeurIPS 2022. arXiv 2201.11903. Described within the scope of AR survey §1, §2.2 and §3.2.1.
• [L11] Noah Shinn et al. "Reflexion: Language Agents with Verbal Reinforcement Learning". NeurIPS 2023. arXiv 2303.11366. Described within the scope of AR survey §2.2, §3.1.2, §4.1.1 and Tables 4, 6 and 9.
• [L03] Timo Schick et al. "Toolformer: Language Models Can Teach Themselves to Use Tools". NeurIPS 2023. arXiv 2302.04761. Described within the scope of AR survey §3.2.2, §3.3.3 and Tables 7 and 8.

How this was checked. On October 5, 2026, the chapter was checked against the revised PDF (146 pages, anonymized review copy). Section, table and equation numbers, quotations and the numbers in §1.1 were checked in the extracted text, and Figure 1 (p. 2) and Table 3 (p. 15) were read by rendering the pages. The optimization-target column of Tables 6–11 was counted row by row and then checked again against the count of symbols in each table. Because the PDF is anonymized, authors, publication status, license and repository follow the bibliographic details in Appendix III (checked October 4, 2026). The arXiv numbers of the three neighboring surveys were taken from the AR survey's bibliography and matched against the titles on their arXiv abstract pages. The public OpenReview reviews (reviewers 8QGv and sncv, the author responses, the action editor's decision) were opened in a browser on October 5, 2026 to check the original text. The Harness survey was checked against the PDF on its project page (71 pages). The ratios and chi-square tests in section 8 (Pearson, scipy) were calculated from the printed numbers and the row counts the editor made.

Chapter 3 · Core · ★★★

Chapter 3. The Agent's Body and Fence — The Harness

The question this chapter answers: why do performance and safety differ from system to system when the model is the same?

Core reading. Junjie Li et al., "Agent Harness Engineering: A Survey", 2026, unpublished preprint (uploaded directly to the OpenReview archive; it has not been through conference or journal review). 51 pages of main text, 71 with the references and appendices.

One-paragraph summary. As agents came to work for long stretches in repositories, browsers and terminals, the success of a task stopped being decided by the model's responses alone. This survey calls the operational infrastructure around the model a harness and sets it up as an engineering object in its own right: the infrastructure that feeds in context, calls tools through stable interfaces, preserves state, recovers from failures, verifies results and keeps the agent inside its permissions. The framework is a seven-layer taxonomy, ETCLOVG (Execution, Tooling, Context, Lifecycle/Orchestration, Observability, Verification, Governance). The authors place more than 170 public repositories on these layers to draw a map, then gather three tradeoffs between layers. It is a map, not an experimental paper. The evidence that harnesses change performance comes from other people's reports, and the authors themselves say that this evidence is concentrated on coding agents.

Presenter's guide

A 50-minute plan. The 10 minutes after the presentation are for discussion.

Must read. All of §1 (under three pages), §2.1–2.3, §10–11 (two pages together) and §13 (one paragraph). If you show only one figure, make it Figure 3. Four layers are pillars, one looks down from above, one supports from below and one surrounds the whole.

Optional reading (for those interested in statistics). §2.4–2.9 (scope, collection procedure, inclusion criteria, coding protocol, limitations of the corpus and aggregate analysis, two pages together) and Table S1 in Appendix A. The reanalysis in section 7 uses these. Table S2 in Appendix B (which layers each of six reference implementations covers) is half a page.

What to cut from the presentation. The layer-by-layer chapters, §3–§9, are read one at a time in the later chapters (the table in section 4). Also skip the references (pages 52–66) and the body of the project list in Appendix A.

If your group will not hold Chapter 14. Put SWE-agent's Table 3 (the ablation of the agent-computer interface) from "Side papers" into the presentation. It is the table that shows, as a controlled experiment, the "fix the model and change only the harness" that the survey speaks of only through other people's reports.

When you quote the numbers. Most of the performance figures in the survey were not measured by the survey. They were carried over from blogs and technical reports. When you write a number, give the model, the benchmark and the original report with it, and say that the survey itself is a preprint that has not been peer-reviewed. The ecosystem figures are a snapshot of May 8, 2026.

Building intuition

Same model, different scores

At the outset the survey gives three cases in which the model's weights stayed fixed and only the surroundings changed (§1).

§1 of the Harness survey put into a table. The survey does not give the sample sizes or the number of repetitions of the original reports.

§1 immediately adds a cautious sentence. "The strongest controlled evidence currently comes from coding-agent benchmarks, and these results do not establish that the harness matters more than the model in every setting." Performance depends jointly on the model and its execution system. How to divide up and name what lies outside the model is the work of this survey.

The editor's bridge — the controlled conditions and measuring instruments of an experiment

The survey's own framework is closer to control theory. §11.3 writes that changing a context policy, tool schema, verifier or recovery loop changes the closed-loop controller $C_H$ and therefore the measured behavior of the same model (citing Zhang et al. 2026c). The model is the plant; the harness is the controller.

The paper in detail

1. What a harness is — definition and boundary (§1, §2.4)

Definition. §1 lists seven things an agent must do to accomplish a task and calls the surrounding operational infrastructure that provides them the agent harness.

An agent must receive relevant context, invoke tools through stable interfaces, preserve state, recover from failures, expose its actions for inspection, verify outcomes, and operate within defined permissions. We refer to the surrounding operational infrastructure that provides these functions as the agent harness.

Harness engineering is the work of designing and integrating these mechanisms so that model capabilities can support reliable execution.

Boundary. §2.4 narrows the definition on purpose. A harness is not "any software around an LLM" but "the engineered wrapper that turns model calls into bounded, stateful, tool-mediated task execution". The line is drawn by function, not by product category. Frameworks that expose reusable mechanisms such as stateful orchestration, tool routing, policy hooks or trace collection are in scope. Thin API wrappers, prompt collections, static datasets, general-purpose containers, vector databases, APM dashboards and content filters are out of scope unless they have been adapted to agent execution. The vector-database retriever of Chapter 8 is not, by itself, a harness either.

Three questions. §1 names three questions that earlier work found hard to answer: where the harness begins and ends, how its interacting mechanisms are organized, and which parts of the design space are well covered. Academia studies planning, memory, tools, verification and safety separately, while practitioners talk about context pipelines, tool contracts, retries and permission boundaries, so the same problem goes by different names. That is the survey's diagnosis.

The contributions are numbered in two ways within the document. The abstract goes ① establishing the harness as a distinct engineering object ② ETCLOVG ③ the ecosystem map, while the list in §1 goes ① ETCLOVG ② the ecosystem map ③ the cross-layer synthesis. The "Claim 1, 2 and 3" of the body (the start of §3, §10) follow the order of the abstract. When you refer to them by number in a presentation, say which list you mean.

2. The timeline — from a single loop to infrastructure (§2.1, Figure 2)

§2.1 divides the change in agent systems into three periods, as changes of architecture and infrastructure.

• Reasoning–action loops (2022). ReAct popularized the loop that alternates reasoning, action and observation. The early infrastructure was a single while loop, one prompt template and a small tool dispatch table. When AutoGPT and BabyAGI widened the pattern with task queues, memory and broader tool use, the need for explicit termination, context control, state recovery and monitoring came into view.
• Tool integration and multi-agent coordination (2023). Gorilla, ToolLLM and Toolformer showed tool use as an ability that can be learned or elicited, and CAMEL, ChatDev and MetaGPT put forward coordination patterns. The evaluation infrastructure grew with SWE-bench, AgentBench, WebArena and GAIA, and MCP and A2A pushed standard interfaces.
• The harness turn (2025). Reports attributing reliability to the design of the surroundings increased. OpenAI used the term "harness engineering" for Codex-style work, Meta-Harness outperformed hand-built scaffolds through automatic optimization, and the 52.8% → 66.5% report above appeared. The survey sums up that harness design has become "a measurable experimental factor rather than a fixed implementation detail." The subheading says 2025, but the text begins "By early 2026," and all the examples are from 2026.

Reading Figure 2. The figure places 51 representative systems on a timeline and colors them by the seven ETCLOVG layers. Enlarged and counted, it gives the following.

The editor counted the entries of Figure 2 of the Harness survey by layer.

By year there are 1 entry in 2022, 18 in 2023, 12 in 2024, 16 in 2025 and 4 in 2026. All five entries colored G date from January 2025 or later. The picture fits §2.9's statement that governance matured late, but the 51 are "representative" systems chosen by the authors, not a sample, so it is not read as a trend. That "Beyond Permission Prompts" (Oct 2025), which deals with permissions, is colored E rather than G can also be read as a sign that permissions sit where the boundary between G and E blurs.

3. Three kinds of engineering — prompt, context, harness (§2.2, Figure 1)

§2.2 divides the engineering around LLM systems into three expanding scopes.

The three paragraphs of §2.2 of the Harness survey put into a table.

Figure 1 sets the three columns from left to right and gives them the subtitles "Optimize the model input" (Instruction & Role, Example & Format, Prompt Iteration), "Optimize what the model sees" (Retrieval & Loading, Memory & State, Compression & Filtering) and "Optimize how the model runs" (Tools & Environment, Orchestration, Verification & Evaluation, Governance & Security), each with its items. The last paragraph of §2.2 writes that harness engineering does not replace the other two but integrates them into a broader operational process. The caption says that this three-part distinction was articulated early in a February 2026 blog post (MadPlay 2026) and that similar formulations followed (OpenAI 2026c, Meng et al. 2026). S22, a supplementary source for this chapter, is OpenAI 2026c.

Scope or era? The caption of Figure 1 compares the three kinds of engineering as "progressively broader scopes", while §13 calls the same thing a "three-phase engineering evolution". The word "phase" by itself makes the earlier ones sound like eras that have passed, but the last paragraph of §2.2 writes that harness engineering integrates the earlier two rather than replacing them (Misreading 2). So the two descriptions are not an either-or. Read them as saying that the emphasis of engineering moved outward from period to period while the earlier kinds remain as inner concentric circles, and the two fit together.

4. ETCLOVG — the seven layers (§2.3, Figure 3, Figure 4)

The names in §2.3 are Execution, Tooling, Context, Lifecycle, Observability, Verification and Governance (the abstract and §1 use slightly longer names: Execution environment, Tool interface, Context management, Lifecycle/Orchestration, …). The first four are the structural core, and the last three are the control plane around it.

The layout of Figure 3 of the Harness survey and the sub-items in each box. O reaches down to the four pillars, V reaches up to them, and G is the outer border. The editor drew the diagram.

Here is what each of the seven layers is responsible for and where this book meets that layer again, gathered in one table.

The definitions of §2.3. The subcategories of each layer are in Figure 4. "Where we meet it again" is this book's arrangement, and Chapter 20 maps all seven layers onto the parts of the LLM Wiki.

Three choices that differ from earlier taxonomies. The survey contrasts its taxonomy with Meng et al. (2026), who described one harness as six components: an execution loop, tool registry, context manager, state store, lifecycle hooks and evaluation interface. ① It divides by engineering surface rather than by minimal runtime role, so that it can map an ecosystem that mixes complete harnesses, protocols, sandboxes, memory systems, observability tools and evaluation frameworks. ② It raises observability (O) to an independent layer. The reason is that practice has dedicated tools (Langfuse, Arize Phoenix, OpenLLMetry) and its own routines (OpenTelemetry instrumentation, cost attribution). ③ It makes governance (G) a first-class layer that holds model-level defenses, system-level permissions and gateways, and organization-level audit and human approval.

As a result, state management goes into L together with the execution flow that reads and writes state, while lifecycle hooks and policy enforcement go into G. The C box in the figure also has "session state", which makes this easy to confuse. The split is this. C is about what the model sees at each step, that is, choosing the content of state and putting it in. State management in L is the control flow that creates, updates and resumes state during execution (§1 writes "where state is created, updated, resumed, and consumed during execution", and §6.1 lists remaining subtasks, tool results, session persistence and resuming from checkpoints). Summarizing a long conversation and putting it into the next call is C; restarting an interrupted task from a checkpoint is L. Apply this taxonomy and even the hooks familiar to readers who have used Claude Code become parts of G.

5. The corpus — how the 170-odd entries were collected (§2.4–2.9, Appendix A)

Collection (§2.5). The authors write that they built the corpus as a systematic mapping of harness artifacts with public documentation, and that, following the reporting discipline of systematic reviews (PRISMA 2020, Page et al. 2021), they made their sources, search strategy and screening process visible. Candidates came from four streams: earlier surveys and benchmark papers, GitHub searches run reproducibly, curated lists and package registries, and corporate engineering blogs and release notes. The search terms were combinations such as agent harness, coding agent, MCP server and agent observability, and the metadata was frozen on May 8, 2026. Figure 5 draws this as three stages: candidate collection (through deduplication) → inclusion criteria → boundary checks and layer assignment. Note that this flow diagram has no counts per stage. The heart of a PRISMA flow diagram is recording how many records came in at each stage and how many were dropped and why.

Inclusion and exclusion (§2.6). An artifact was included if it had public documentation, implemented or specified a concrete harness-level mechanism, and gave grounds for assigning at least one layer. Chatbot demos, prompt collections, thin client wrappers, datasets and leaderboards without a runtime, and product pages whose behavior could not be confirmed from public documentation were excluded. Boundary cases were decided by mechanism, not by label.

Coding (§2.7). Coding was multi-label, based on public artifacts (READMEs, documentation, papers, examples, release notes), and the most central mechanism was marked as the primary layer. Then comes this sentence.

The current snapshot uses a single-primary-coder protocol with author audit rather than a formal multi-coder agreement study, so we do not report Cohen's kappa or a comparable inter-annotator statistic.

One coder did the coding and the authors audited it; agreement between coders was not measured. Where the public evidence was unclear, the layer assignment was withheld.

Limitations (§2.8). The corpus is not a census but "a map of the visible agent-harness ecosystem" (the biases are in section 8). Absence from a layer means "not publicly evidenced" rather than "not implemented".

Aggregation (§2.9). All of the body's judgments about the ecosystem are qualitative. Execution environments, tool interfaces, lifecycle orchestration and verification are densest; context and memory are buried inside large frameworks; observability and governance are thin in open source. The body has no numbers behind these judgments.

The numbers are in the appendix. Appendix A gives the scale of the May 8, 2026 snapshot as 171 public entries, 146 GitHub entries, 142 GitHub entries in project categories and 0 broken URLs. Table S1, which excludes reading lists, ecosystem maps, list repositories and tutorials, gives each artifact one primary layer and its GitHub star count.

The entry counts are taken from the category header lines of Table S1 in Appendix A of the Harness survey ("20 entries; primary layer: E" and so on), and counting the rows of the table directly gave the same numbers. The shares, medians and counts of "10k stars or more" were calculated by the editor from the star counts printed in the table.

Two things stand out. First, "170+" and the 138 of Table S1 differ. The table does not say how many of the missing 33 came from which categories. Second, L accounts for more than one in three. As the name of the L category, "Harness Architecture & Orchestration", suggests, this can be read as the result of complete harnesses with several layers, such as Claude Code, OpenCode and Codex CLI, landing in L when one primary layer had to be chosen. Six of the ten entries with the most stars are L. Table 2 in §6 also gives star counts, but its reference date (May 12, 2026) is different and so are its values, so do not mix the two.

6. Between the layers — the trilemma, capability against control, coupling (§10, §11)

The diagnosis of §10 is that many failures in practice arise at the interfaces between surfaces. The execution environment (E) constrains which orchestration strategies (L) are practical, context management (C) affects the reproducibility of evaluation (V), and governance (G) imposes identity, permission and audit constraints on every layer. A harness design should therefore "be read as a dependency structure, not as a checklist of separable components." The gaps that keep appearing at the boundaries are interoperability between tools, cost attribution, failure recovery, coordination across several repositories, and handoff between humans and agents.

The abstract and §1 introduce the problems that cut across layers as three, but the opening of §11 speaks of "five system-level effects" and adds 11.4 (from frameworks to platforms) and 11.5 (the research agenda). The last two are closer to trends and agenda. Read the three as the core.

§11.1–11.3 of the Harness survey put into a table.

The conclusion of 11.1 is that quality cannot be treated as a single scalar objective. It is the same picture as the balance among sample size, measurement precision and survey cost.

11.3 answers this chapter's question most directly. A harness change has to be tested as a system change. A component that is beneficial on its own can make the whole worse once combined with the rest of the control loop. So a score cannot be cleanly attributed to the model without specifying the surrounding controller. §8.1 puts it this way: "A harness-aware evaluation should treat the reported score as a property of the model–harness pair, not of the model alone." Evaluation should either lock the harness across models or vary it as an explicit experimental factor.

When models improve (§12.5). Each wrapper, reset, verifier, memory rule and permission gate encodes an assumption about "what the model cannot do reliably on its own." When the model changes, the effect of the intervention has to be estimated again. The survey writes that a factorial model × harness evaluation can reveal whether an intervention helps all models, helps only certain families or reverses the rankings (citing Zhang et al. 2026c), and it passes on Anthropic's report that context resets which had been useful for one model became unnecessary with a stronger model, and removing them cut cost with no loss of quality. It also warns that a harness that optimizes itself against a narrow benchmark can become brittle.

The strength of the conclusion (§13). §13 sums up the survey with the "binding-constraint thesis": "infrastructure quality, not model capability alone, sets the ceiling on real-world agent reliability." It is a stronger statement than the cautious sentence in §1. With the qualifier "not … alone" it is not a contradiction, but the survey contains no comparative experiment to support "sets the ceiling".

7. Rereading with a statistician's eye

This survey has no scoreboard to put confidence intervals on. Instead, the results of its coding are printed in the appendix (the 138 rows of Table S1). So this chapter takes the reanalysis form. The material is the entry counts and star counts by layer, and SWE-agent's table, which measured "same model, different harness" under control.

There is no inter-coder agreement. There was one coder, and no kappa was reported (§2.7). This means the reliability of the measurement called layer assignment is unknown. Choosing one primary layer is easy to dispute in boundary cases, especially whether to put a complete harness in L.

L alone makes the distribution. A goodness-of-fit test against the null hypothesis that the seven layers are equally represented gives ${\chi }^2=49.5$ (6 degrees of freedom), $p\approx 6\times {10}^{-9}$. The distribution is not even. But run the same test on the other six layers alone (91 entries), leaving out L, and you get ${\chi }^2=7.04$ (5 degrees of freedom), $p=0.22$. This is a reference calculation that assumes a multinomial model in which entries fall into layers independently and evenly, and since the corpus is not a probability sample this $p$ is not read as an inference about a population (Exercise 1(c)). Under that assumption, all one can say is that the differences among the six layers do not exceed the fluctuation that even assignment would produce. §2.9 writes that E, T, L and V are densest and O and G thin, but by primary layer T has 12 entries, sixth of the seven layers and fewer than O (15) or G (14). The judgment in §2.9 may refer to "coverage" that includes secondary layers assigned through multi-label coding, but the secondary-layer assignments are not printed anywhere. How to reconcile the two statements remains open.

Stars are even more skewed. Of the roughly 2.25 million stars across the 138 entries, 56.3% sit in L. The median for L is 13.9k; for the other 91 entries it is 4.0k. A Kruskal–Wallis test of the null hypothesis that the distributions are the same across layers gives $H=29.2$ (6 degrees of freedom), $p\approx 5.6\times {10}^{-5}$. Attention goes to complete harnesses, and the components of V and E, with a median of 1.8k, get the least. Star counts are a proxy for attention, not for use, and they are a one-day snapshot.

Exercise 1 — how far can the layer distribution in Table S1 be trusted? (a) Find the 95% Wilson confidence intervals for the shares of L (47/138) and T (12/138), and Goodman simultaneous confidence intervals for all seven shares. A Goodman interval is the Wilson interval formula with $z^2$ replaced by $A={\chi }_{1,1-0.05/7}^2\approx 7.24$ (the critical value from splitting a familywise error rate of 5% across 7 categories, $\sqrt{A}\approx 2.69$). That is, for a category with $n_i$ entries the interval is $\{A+2n_i\pm \sqrt{A[A+4n_i(n-n_i)/n]}\}/\{2(n+A)\}$. (b) Reproduce the two goodness-of-fit tests above. (c) What population are these intervals about? Answer on the basis of §2.5 and §2.8. (d) If ten boundary cases moved between L and other layers, how would the conclusions change? What should the next edition report so that this question can be answered?

Exercise 2 — same model, different harness: how large must a difference be to show? The SWE-agent paper (Yang et al. 2024) fixed the model, changed only the interface and measured the results on the 300 items of SWE-bench Lite (Table 1, Table 3; reproduced in "Side papers"). (a) Convert Shell-only's 11.00% and SWE-agent's 18.00% into counts and compare them as if they were independent samples. (b) With item-level results not published, what range can the $p$ value of an exact McNemar test take? (c) If each of the eight ablations in Table 3 is compared with the default configuration, how many survive multiple comparisons? (d) How many tasks per harness would be needed to detect the 52.8% → 66.5% cited in §1 at a 5% significance level with 80% power? With a paired design?

8. Limitations the paper states

• The evidence is concentrated on coding agents. It does not show that the harness matters more in every setting (§1).
• Bias of the corpus. It is skewed toward English, GitHub, open source and coding agents, and commercial systems are underrepresented (§2.8). The conclusion also names extending the analysis to closed systems and beyond coding as future work (§13).
• No inter-coder agreement. The coding was done by one coder with author audit (§2.7).
• The taxonomy is descriptive. Turning it into a normative framework that guides design decisions is the next step (§13).
• Unsolved problems. Context drift, in which behavior gradually degrades over long interactions, is the hardest open problem of the C layer (§5.7), and a self-optimizing harness risks overfitting to benchmarks (§12.5).

Two things the survey does not state are noted here as well. One is that the survey is a preprint that has not been peer-reviewed and did not measure the figures for harness effects itself. The other is that all six authors of Zhang et al. (2026c), which the survey cites in §5.7, §8.1, §11.3 and §12.5 as the basis for treating the harness as an experimental factor, are on the author list of this survey. Self-citation is common, but it is worth pointing out before reading that claim as an independently confirmed result. This remark is this book's reading.

Side papers — how they differ from the main paper

Effective harnesses for long-running agents (Justin Young 2025, Anthropic). The post the survey cites several times as "Anthropic 2025e". The problem is long-running work that spans several context windows. A new session starts with no memory of the previous one, and compaction alone was not enough. The failures were trying to do everything at once and running out of context midway, and a later session declaring the work finished. The remedy is for an initializer agent in the first session to create init.sh, a progress file, a first git commit and a JSON list of more than 200 features all marked "failing", after which a coding agent implements just one feature per session, tests it end to end through the browser, then leaves a commit and a progress note. A footnote says the two "agents" differ only in their initial user prompt and share the same harness. What differs from the survey is that this is one company's design case rather than a map, and it reports the effect in words without numbers. The progress file sits in C, git in L, the feature list and browser tests in V, and init.sh in E (this mapping is this book's reading).

SWE-agent's ablation of the agent-computer interface (Yang et al. 2024 · Chapter 14 of this book). Most of the reports the survey offers as evidence are blogs whose comparison conditions are not published. SWE-agent fixed the model, changed the interface components one at a time and measured on the 300 items of SWE-bench Lite. Table S2 in Appendix B of the survey also sees SWE-agent's key contribution as the agent-computer interface (ACI) and writes that commands, observations, editing, tests and execution backends become "part of the measured harness".

Table 3 of the SWE-agent paper. Resolution rates on SWE-bench Lite (300 items). "Default" marks the default configuration that the original indicates with an icon; the numbers in parentheses are the drops given in the original. §5.1 of the text calls the configuration for this ablation "SWE-agent w/ GPT-4", and Table 1 gives the same 18.00% as "GPT-4 Turbo".

In Table 1 of the same paper, with the same model (GPT-4 Turbo), the Shell-only agent scored 11.00% and SWE-agent 18.00%, and the paper calls this a 64% relative increase due to the ACI. The result that iterative search does worse than no search command at all (12.0 against 15.7) is put down to agents paging through results one by one until they exhaust their budget, and it shows that an interface built for people may not suit agents. How statistically solid the cells are was calculated in Exercise 2 of section 7. L21 in Chapter 1 also advises putting as much effort into the ACI as into the HCI.

Harness engineering: leveraging Codex in an agent-first world (Ryan Lopopolo 2026, OpenAI · Original · Korean edition (in Korean)). The post the survey cites as "OpenAI 2026c" across §1, §2, §6.4 and §12.5. It is a five-month report from a team that built an internal beta product with Codex alone, without people writing a single line of code by hand. Starting from an empty repository in late August 2025, it reached about a million lines of code and about 1,500 merged PRs, and each of the three engineers on the team at first produced an average of 3.5 PRs per day. That it took one-tenth of the time hand-written code would have is the authors' estimate. The post writes that the people's work moved from writing code to designing environments, specifying intent and building feedback loops.

Its devices hang on nearly all of the survey's layers (the mapping is this book's reading). Each worktree boots the app with a disposable observability stack for logs, metrics and traces per task (E, O). The agent handles the screen directly through Chrome DevTools Protocol and skills for DOM snapshots and screenshots (T). An AGENTS.md of about 100 lines serves as a table of contents rather than an encyclopedia, and docs/ is treated as the system of record, disclosed only as needed (C). Execution plans are committed to the repository along with progress and decision logs (L). The direction of dependencies between layers is enforced mechanically by custom linters and structural tests, and the lint error messages carry instructions for fixing the problem into the agent's context (V, G). A "doc-gardening" agent that scans the documentation and background jobs that look for violations of "golden principles" open cleanup PRs (V, L).

It parts ways with the survey in three places. First, the name. "Harness engineering" appears only in the title and is not defined in the body. Neither the three-stage distinction among prompt, context and harness engineering nor the term "task runner" is in the body. The statement in §2.1 that OpenAI adopted the term is supported by the title. But when the caption of Figure 1 and §2.2 lean on this post for the three-stage distinction, they are reading its content through the survey's framework. In content, the passage that goes from one prompt through bug reproduction, the fix, a demo video, a PR and the merge fits the task runner of §6.4 well. Second, §12.5. The survey writes that OpenAI sees harness engineering not simply as adding more scaffolding but as aligning human attention, repository state and agent execution (OpenAI 2026c;d). The statement that human time and attention are the one truly scarce resource, and the statement that the repository serves as the system of record, are in the body. But the contrast with scaffolding is not in this post; on the contrary, its conclusion says "the discipline shows up more in the scaffolding rather than the code." Nor is the contrast in 2026d, cited alongside it. 2026d is Michael Bolin's post that unrolls the agent loop of Codex CLI ("Unrolling the Codex agent loop", January 23, 2026). It calls the agent a "harness" and explains how the prompt is assembled as a list of items, the prompt cache that hits only when the prefix matches exactly, requests that keep no state, and conversation compaction. It does not deal with scaffolding, human attention or what happens as models improve. The contrast in §12.5 should be seen as the survey's synthesis from reading the two posts together. Third, the nature of the evidence. The figures are one team's self-report with no comparison condition. The post itself says that this autonomy depends heavily on the structure and tooling of this repository and should not be assumed to generalize without similar investment, and it writes that it does not know how this system will change as models improve. If the SWE-agent ablation above showed the effect of the harness through a controlled comparison, this post backs the same claim with operational experience. Its last reservation is the same question as §12.5's demand that "harness interventions should be re-estimated" as models improve.

Agent Harness for Large Language Model Agents: A Survey (Qianyu Meng et al. 2026, Preprints). The earlier synthesis that the survey sets directly against its own taxonomy (Preprints 202604.0428. The nine authors in the survey's references match v1 and v2 (April 7 and April 9, 2026); v3 of April 28, 2026 has eleven authors). If this one is an anatomy chart that describes one harness as six components, the Harness survey is a map that sorts many projects into seven boxes (section 4 of "The paper in detail"). There is also a different survey with a similar name (Guo et al., "From Question Answering to Task Completion: A Survey on Agent System and Harness Design"), so do not mix up the bibliographic details when searching.

Stop Comparing LLM Agents Without Disclosing the Harness (Yunbei Zhang et al. 2026 · Chapter 18 of this book). The position-paper preprint the survey cites as "Zhang et al. 2026c". What the survey passes on is that a score is a property of the model–harness pair (§8.1), that the harness is the closed-loop controller $C_H$ (§11.3), and that a factorial model × harness experiment can reveal reversals of ranking (§12.5). Where the survey places this idea as one conclusion of an ecosystem map, this paper sets it up as a demand on evaluation practice. The problem with comparisons that do not disclose the harness is taken up again in the side papers of Chapter 18.

Building effective agents (Erik S. and Barry Zhang 2024, Anthropic · Chapter 1 of this book). Chapter 1 separated workflows (LLMs and tools orchestrated through predefined code paths) from agents (the LLM dynamically directs its own process and tool use), and advised starting from the simplest solution. If Chapter 1 asks what to build, this chapter asks what to wrap it in. The parsimony principle of Chapter 1 meets §12.5's demand that harness interventions be re-estimated as models improve.

Back to the surveys

The subject of this chapter is the Harness survey itself, so this section reads it against the revised version of the Agentic Reasoning survey of Chapter 2 (the revised AR survey below). The two cover the same object in the same year, yet barely overlap.

Head and body — the questions the two surveys ask

The revised AR survey asks of every method "what does it optimize?", divides methods into in-context search over the reasoning trace $z$, updating the system state $S$ and post-training of the parameters $\theta$, then crosses these with the dynamics of the environment (foundational, self-evolving, collective) to form a 3 × 3 grid (§1, §2.2, Table 3). It is a survey that looks at the agent's head. The Harness survey asks which surface outside the model is responsible for what. Body and fence.

The revised AR survey does not use "harness" as a concept. Search its 146 pages and the word appears only in three reference titles ("Harnessing …"). Conversely, the Harness survey has no formalization such as $z$, $S$ and $\theta$. One draws the same agent as a probabilistic model, the other as a system diagram.

Comparing the tables of contents

The editor set the tables of contents of the two surveys side by side. "None" means that the table of contents has no section on that topic.

The three topics without a section in the revised AR survey — context management, execution environment and observability — are exactly the ones the Harness survey set up as independent layers.

Placing it in the formalization — where in the tuple is the harness?

§2.2 of the revised AR survey deliberately separates the environment $E=⟨X,O,A,T,\Omega ,R,\gamma ⟩$ from the agent $G=⟨Z,M,{\pi }_{\text{reason}},{\pi }_{\text{exec}},U⟩$. Where does the harness go? The mapping below is this book's reading; neither survey makes this mapping.

The editor's mapping, not a claim of either survey.

In Chapter 2, the revised AR survey wrote that "two agents with different Z or M face the same E." Placed as above, when a harness change touches tool permissions or the execution environment, $A$ and $T$ change along with $M$. In that case even the same model ($\theta$ fixed) in effect faces a different environment. Conversely, a harness change that touches only the context policy (C) or the observability instruments (O) counts as a case where only $M$ or the measuring side differs within the same $E$. This is one formal answer to "why does performance differ when the model is the same?"

The $S$ of the revised AR survey is memory, tool libraries, prompts, communication topology and the agent's own code, and it overlaps a great deal with the parts list of a harness. The difference is that $S$ is updated across episodes (Eq. 5, $S_{k+1}\leftarrow U(S_k,{\tau }_k,F_k)$), while a harness is usually designed by people and held fixed. Meta-Harness, which optimizes the harness automatically, and the HarnessFix of Chapter 19, which fixes harness defects from failed trajectories, become cases of the $S$ mode in the language of the revised AR survey.

Comparing methods — how did the two surveys disclose their corpora?

The editor set §1.1 of the revised AR survey against §2.5–2.8 and Appendix A of the Harness survey.

Their attitudes toward PRISMA are opposite. The revised AR survey says it did not follow it and prints the counts in the text. The Harness survey says it followed it, but its flow diagram has no counts. The gaps they acknowledge are alike. Industrial systems that are not public show up poorly on either map.

Design guidance and governance

Table 5 of the revised AR survey recommends which mode to choose for each deployment condition: $z$ for one-off tasks without a training budget, $S$ for repeated tasks in a fixed domain, $\theta$ when the task distribution is known and serving latency matters. §11 of the Harness survey records tradeoffs between layers rather than a prescription table. One asks which one to choose; the other asks whatever you choose, what you must accept.

On governance the two surveys arrive at the same structure. §8.7 of the revised AR survey divides governance into the model level (alignment), the agent level (tool permissions, sandboxing, restriction of the action space, human confirmation) and the ecosystem level (responsibility among agents run by different principals), and writes that the engineering controls at the agent level are "currently underspecified relative to how widely agents are deployed." The G layer of the Harness survey holds model-level defenses, system-level permissions and gateways, and organization-level audit and human approval. The agent level that the revised AR survey calls underspecified becomes, in the Harness survey, the content of two layers, E and G. Likewise, §8.7 of the revised AR survey requires provenance for every write to $S$, and §12.2 of the Harness survey requires provenance and staleness markers for remembered facts. A survey that started from the head and one that started from the body meet at auditability.

Common misreadings

Misreading 1. "The harness matters more than the model." The survey does not claim this. §1 writes that the strongest evidence comes from coding-agent benchmarks and does not show that the harness matters more in every setting. The "sets the ceiling" of the conclusion also carries the qualifier "not model capability alone". The video title "Why The Harness Matters More Than The Model" is not a sentence of the survey.

Misreading 2. "Harness engineering is the next stage that replaced prompt and context engineering." §2.2 writes that harness engineering integrates the two rather than replacing them. Writing prompts well is still needed inside a harness.

Misreading 3. "ETCLOVG is the order in which to build a harness and a design guide." §13 writes that the taxonomy is descriptive and that turning it into a normative framework is the next step. The seven layers are boxes for sorting what exists, not a process for building upward. The table closer to "what to choose" is Table 5 of the revised AR survey.

Misreading 4. "The layer distribution of the 170-odd projects is the shape of the ecosystem." §2.8 says to read the corpus as a map of the visible ecosystem. Table S1 gives 138 entries one primary layer each, and agreement between coders was not measured. Leave out L, and the differences between layers are not pronounced even in a reference calculation that assumes even assignment (section 7). That the corpus is not a probability sample comes first.

Misreading 5. "This survey showed the effect of harnesses experimentally." The survey ran no experiments. Its figures on harness effects are carried over from blogs and other studies, and their sample sizes and numbers of repetitions are not given. An example of a controlled comparison is SWE-agent's Table 3 in "Side papers".

Misreading 6. "Observability (O) and verification (V) are the same job." §2.3 sets them up separately. O captures signals; V turns them into evaluation, failure attribution and regression feedback. §12.3 cites LangChain's 2026 survey that 89% of teams use observability tools while 52.4% run offline evaluations, and reads this as a state in which teams see what agents did but do not systematically judge whether it was right.

Misreading 7. "Harnesses are a story about coding agents." Coding agents are overrepresented because they leave many public traces (§2.8). The conclusion names extending the map beyond coding as future work. The LLM Wiki of Chapter 20 is a case of meeting the seven layers again outside coding.

Discussion questions

1. The harness as a control variable. If you have compared two models, did you hold the prompts, tools, retry rules and evaluation scripts fixed? If not, what could you have attributed to the difference between the models? Of the two options in §8.1 (lock the harness or vary it as a factor), which is realistic for you? Check your answer in Chapter 18.

2. Capability against control: where to draw the line. Just as you decide how far to give developers write access to a production database, you have to draw a line when giving an agent file-system, shell or external-API permissions. Of the devices for reducing the blast radius in §11.2, which would you put in first, and what would you give up in exchange? Think ahead to Chapter 15.

3. Should the harness shrink as models improve? §12.5 says that each device of a harness is an assumption about "what the model cannot do reliably on its own." What could be removed a year from now? What experiment would show whether it is safe to remove? We ask again in Chapter 19 and Chapter 20.

Videos

• [Secondary] "What exactly is harness engineering? | Harness engineering, explained by a practitioner" — Jeon Hyunjun's Field Agents (in Korean)
• [Secondary] "Understand harness engineering in 15 minutes | Concepts, core principles, AI agents" — Lee Donghun's Root AI (in Korean)
• [Secondary] "Why The Harness Matters More Than The Model | YC Paper Club" — Y Combinator (panel discussion)

The titles and channels of the videos were checked through YouTube oEmbed on October 4, 2026. Titles and channel names in Korean are given in English, translated or romanized by the editor. The editor did not verify the content of the videos.

Further study

• The rest of the paper. §3–§9 are read in the later chapters (the table in section 4 of "The paper in detail"). Read the five open problems of §12 (execution environments, maintaining long-term state, diagnosing failures from traces, standard handoffs, and harnesses as models improve) while preparing for Chapter 19 and Chapter 20. Table S2 in Appendix B shows the layers covered by six reference implementations (Claude Code covers E, T, C, L and G; SWE-agent covers E, T, L and V).
• Repository. The awesome-agent-harness repository. The Hugging Face side holds only the survey PDF and a README as of October 2026, with no catalog data. The GitHub list has kept growing since the May 8, 2026 snapshot (Table S1) to about 370 entries, so if you redo the reanalysis of section 7 there, match the date. Checking whether the secondary-layer assignments are published is also a good exercise.
• Reading on in the surveys. Revised AR survey §1.1 (building the corpus), §2.4 and Table 5 (design guidance), §8.7 (the three levels of governance).
• Within this book. Chapter 7 (T), Chapter 9 (C), Chapter 13 (L), Chapter 14 (E), Chapter 15 (G), Chapter 17 and Chapter 18 (V), Chapter 19 (O), Chapter 20 (a mapping of all seven layers).

Sources

Core reading

• [L01] Junjie Li et al. "Agent Harness Engineering: A Survey". 2026, unpublished preprint (uploaded directly to the OpenReview archive; posted 2026-05-14, last modified 2026-07-18). Project page · PDF · OpenReview · Data · Repository. Cited: §1–2 (Figures 1–5), §5.7, §6 Table 2, §8.1, §10–13, Appendix A Table S1, Appendix B Table S2. Checked against main.pdf (71 pages) from the project page.

Surveys

• [L22] Tianxin Wei et al. "Agentic Reasoning for Large Language Models: A Survey". Under review at TMLR (action editor "Accept as is" on 2026-09-15; OpenReview "Decision pending"). OpenReview, revised version (CC BY 4.0) · arXiv 2601.12538 (the arXiv version — its section and table numbers differ from the revised version. On v2 the abstract page's Comments field carries "Accepted by TMLR", but the body is the same as v1; compared on 2026-10-05). Cited: §1, §1.1, §2.2–2.4, Table 3, Table 5, §8.7, §9, the table of contents. Section and table numbers follow the anonymized review copy (generated 2026-09-01).

Side papers

• [S21] Justin Young (Anthropic). "Effective harnesses for long-running agents". Anthropic Engineering blog, 2025-11-26. Original
• [S20] John Yang et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering". NeurIPS 2024. arXiv 2405.15793 · NeurIPS 2024 · Code. Cited: §5, §5.1, Table 1, Table 3.
• [S22] Ryan Lopopolo (OpenAI). "Harness engineering: leveraging Codex in an agent-first world". OpenAI blog, 2026-02-11. Original · Korean edition (in Korean). The author, date and text were checked on the English original page on 2026-10-05.
• [S32] Yunbei Zhang et al. "Stop Comparing LLM Agents Without Disclosing the Harness". 2026, preprint. arXiv 2605.23950. What this chapter says about it is limited to the Harness survey's citations (§5.7, §8.1, §11.3, §12.5).
• [L21] Erik S. and Barry Zhang (Anthropic). "Building effective agents". Anthropic Engineering blog, 2024-12-19. Original
• Michael Bolin (OpenAI). "Unrolling the Codex agent loop". OpenAI blog, 2026-01-23. Original · Korean edition (in Korean). The Harness survey's "OpenAI 2026d". The author, date and text were checked on the original page on 2026-10-05.
• Qianyu Meng et al. "Agent Harness for Large Language Model Agents: A Survey". Preprints, 2026. DOI 10.20944/preprints202604.0428 (v2, 2026-04-09; v3 is 2026-04-28). The bibliographic details in the Harness survey's references were checked against the Crossref metadata.

How this was checked. On October 5, 2026 the section, figure and table numbers, the figures and the quotations were checked against the Harness survey PDF (71 pages), the revised AR survey PDF (146 pages, anonymized review copy) and the SWE-agent PDF. Authors, publication status and links follow the bibliographic details in Appendix III, checked on October 4, 2026. Figures 1, 2, 3 and 5 of the Harness survey and SWE-agent's Table 3 were read by rendering the PDF pages, and the editor counted the 51 entries of Figure 2 and their numbers by layer on the enlarged figure. S21 was read as text downloaded from the original at the link above, and S22 was read as text downloaded from the English original page on October 5, 2026 and checked against every place the survey cites it (§1, §2.1, 2.2, 2.4 and 2.5, Figures 1 and 9, §6.4, §12.5). That "harness engineering", "task runner" and the distinction among the three kinds of engineering are absent from the body was confirmed by full-text search. OpenAI 2026d, which the survey cites alongside (the URL in the survey's references), was downloaded the same day and searched to confirm that "scaffold", "attention" and "human" do not appear. The calculations in section 7 were done by the editor in Python (scipy and statsmodels) from the printed numbers. The 138 rows of Table S1 were parsed and confirmed to match the entry counts in the category header lines, and Wilson intervals, Goodman simultaneous intervals, chi-square goodness-of-fit tests, a Kruskal–Wallis test and binomial tests were calculated. The SWE-agent comparisons used a normal-approximation test of two proportions, the two extremes of the exact McNemar test and the Bonferroni correction; sample sizes were found from the normal-approximation formulas for comparing two proportions and for the McNemar test.

Chapter 4 · Optional · ★★

Chapter 4. Getting the Model to Write Out Its Thinking — Chain-of-Thought

The question this chapter answers: why does accuracy go up when the model is made to write intermediate steps before answering, and when does it not?

Core reading. Jason Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022. 9 pages of main text, 43 with the appendices.

One-paragraph summary. A language model with frozen weights is shown eight examples, and in each example a worked solution written by a person is inserted between the question and the answer. The model then also writes a solution first for a new question and gives its answer. This is chain-of-thought (CoT) prompting. On GSM8K, a benchmark of grade-school arithmetic word problems, the accuracy of PaLM 540B rose from 17.9% to 56.9%. The effect appeared only once models grew to around 100 billion parameters, and in small models accuracy actually fell. On GSM8K the effect disappeared when the model was made to write only an equation, to write dots as long as the solution, or to write the solution after the answer. The authors read this as evidence that the effect comes from writing a natural-language solution before the answer. It is the starting point of the "thought" that the later chapters of this book talk about.

Presenter's guide

A 50-minute plan. The 10 minutes after the presentation are for discussion.

Must read. All of §1–2 (including Figure 1), §3.1–3.4 and the §6 Discussion. If you show only two figures, make them Figure 1 and Figure 4. Figure 1 shows what the method is, Figure 4 when it works. Read the numbers for Figure 4 from Table 2 in Appendix B rather than from the figure.

Optional reading. Appendix A (FAQ), especially A.1 (why the model has to be large) and A.2 (the role of prompt engineering). Appendices D.1 and D.2 (people reading and classifying 100 solutions written by LaMDA 137B) are short as well.

What to cut from the presentation. The "Side papers" section of this chapter stands in for §7, Related Work, and Appendix C (extended related work). Of Appendices F (input–output examples) and G (full prompts), skim only Table 20 (the eight examples for arithmetic word problems). Skip Appendix E (reproducibility, resources and licenses).

If you skip this chapter. Introduce only Figure 1 and Figure 4 in the first 10 minutes of the Chapter 6 presentation (the first subsection of "Building intuition" in this chapter and the table of gains by scale in section 3 of "The paper in detail"). Chapter 6 uses this paper's CoT and the follow-up self-consistency (CoT-SC) as baselines, and Chapter 5 widens this chapter's single-line reasoning path into a tree. To read those two chapters you need only this: "put solutions in the examples so that the model writes a solution first, and greedily draw a single path".

When you quote the numbers. The scores in this paper are results from 2022 models given a few examples and sampled once with greedy decoding. The arithmetic benchmarks use eight examples (four for AQuA), and the commonsense and symbolic tasks differ by task (the first 10 items of the evaluation set for the two BIG-bench tasks, 6 for SayCan, 4 for last-letter concatenation — §4 and §5 and Appendix G). Only LaMDA is averaged over five orderings of the examples; the others use one ordering (§3.1). Give the model and its size, and the benchmark and its number of items, with every number, and since the same value differs from table to table in places (sections 3 and 4 of "The paper in detail"), also say which table you took it from.

Building intuition

Figure 1 — one problem, two prompts

Figure 1 sets standard prompting on the left beside CoT prompting on the right. The upper panel is the input given to the model and the lower panel is the model's output.

Standard Prompting
  Model Input
    Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can
       has 3 tennis balls. How many tennis balls does he have now?
    A: The answer is 11.
    Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6
       more, how many apples do they have?
  Model Output
    A: The answer is 27.                                         (wrong)

Chain-of-Thought Prompting
  Model Input
    Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can
       has 3 tennis balls. How many tennis balls does he have now?
    A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis
       balls. 5 + 6 = 11. The answer is 11.
    Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6
       more, how many apples do they have?
  Model Output
    A: The cafeteria had 23 apples originally. They used 20 to make lunch. So
       they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9.
       The answer is 9.                                          (correct)

Figure 1 of the CoT paper. The original figure highlights the solution parts of the example and output on the right in color and puts wrong and correct marks beside the outputs. The marks in parentheses render those in words.

The only difference between the two inputs is one line in the example answer. The example on the right first writes out the solution that leads to "5 + 6 = 11", and the model follows that format, writing 23 − 20 = 3 and then 3 + 6 = 9 to get the right answer. The model on the left gives only an answer and gets it wrong with 27. The figure shows just one example, but the actual prompt contained eight (§3.1, Appendix Table 20), and the figure does not say which model produced the output. It is a diagram that shows the method, not an experimental result.

The editor's bridge — marginalizing a latent variable

The paper's own analogy is the way people solve problems (§2). When solving a multi-step word problem, people break it into intermediate steps, solve each, and then give the answer: "After Jane gives 2 flowers to her mom she has 10 . . . then after she gives 3 to her dad she will have 7 . . . so the answer is 7." The authors grant that such a solution is no different from a worked solution, but because a worked solution usually comes after the answer, they call it a "chain of thought" in the sense that it imitates the thought process leading to the answer. This insistence on order comes back in the ablations of section 4 of "The paper in detail".

The paper in detail

1. The definition and four properties (§1–2)

It combines two lines of thought (§1). One is research that trains models from scratch (Ling et al. 2017) or fine-tunes them (Cobbe et al. 2021) to generate natural-language rationales that lead to an answer. The other is the in-context learning that GPT-3 demonstrated. The first is expensive because it needs a large amount of data annotated with rationales; the second performs poorly on reasoning tasks and does not improve much as models grow. The paper combines the two and puts in examples that are triples of three elements: ⟨input, chain of thought, output⟩.

The definition is one sentence: "A chain of thought is a series of intermediate natural language reasoning steps that lead to the final output" (§1). It is a series of intermediate reasoning steps in natural language that lead to the answer.

In formulas, it looks like this. Let the set of examples be $D$. The examples of standard prompting are $D=\{(x_i,y_i)\}$, and the model generates $y$ for a new question $x$. The examples of CoT are $D^{\prime }=\{(x_i,c_i,y_i)\}$, and the model generates the chain $c$ first and then $y$.

$$p_{\theta }(c,y\mid x,D^{\prime })=p_{\theta }(c\mid x,D^{\prime })·p_{\theta }(y\mid x,c,D^{\prime })$$

The parameters $\theta$ are fixed. Only the examples in the prompt change. This formula is the editor's rendering of the paper's description; the paper has no formula.

The structure of standard prompting and CoT prompting. In both cases the weights are fixed and only the format of the examples differs. The dashed arrows mean that the examples set the format of generation. The editor drew the diagram from the descriptions in §1–2 and §3.1 of the paper.

Four properties (§2). The paper gives four reasons why CoT is attractive.

1. It can break a multi-step problem into intermediate steps. So more computation can be allocated to problems that need more reasoning steps.
2. It gives a window for looking into how the model arrived at an answer and for finding where the reasoning went wrong. In parentheses, though, it adds the caveat that fully characterizing the computation behind an answer remains an open question.
3. In principle it can be used for any task that people can solve through language, such as arithmetic word problems, commonsense reasoning and symbolic manipulation.
4. In a sufficiently large off-the-shelf model it can be elicited just by putting chains into the examples.

The first property is tested by "variable compute only" in section 4 of "The paper in detail", and the caveat of the second comes up again in "Common misreadings".

2. Arithmetic reasoning — the experimental setup (§3.1)

Five benchmarks. The numbers of items are in Appendix Table 12.

Only the numbers of items are taken from Appendix Table 12 of the paper.

Examples. Most datasets have only an evaluation split, so the authors wrote eight examples with chains themselves (Appendix Table 20) and used the same ones for every arithmetic benchmark except AQuA. For AQuA, which is multiple choice, they used four examples and solutions from the training set (Appendix Table 21). The authors say these examples did not undergo prompt engineering.

Models. There are five families.

• GPT-3: text-ada-001, text-babbage-001, text-curie-001, text-davinci-002. The paper writes that these "presumably" correspond to InstructGPT models of 350M, 1.3B, 6.7B and 175B parameters. The sizes are guesses.
• LaMDA: 422M, 2B, 8B, 68B, 137B. Appendix Table 2 gives the smallest model as 420M.
• PaLM: 8B, 62B, 540B.
• UL2 20B and Codex (code-davinci-002). According to the version record in Appendix E, these were added in v5. That the abstract still says "three large language models" can be read as a trace of the earlier version.

Decoding. Every model samples once with greedy decoding. In parentheses the authors note that follow-up work improves results by sampling several times and taking a majority vote, citing the self-consistency paper (Wang et al. 2022a) (§3.1). That paper is a side paper of this chapter.

3. Results — it appears only with scale (§3.2, Figure 4, Appendix Table 2)

The authors draw three things from the results.

First, CoT is an emergent ability of model scale. Small models gain nothing; gains appear only at around 100 billion parameters. Smaller models are said to have written chains that were fluent but illogical. Figure 4 is a figure with nine panels, GSM8K, SVAMP and MAWPS (rows) by LaMDA, GPT and PaLM (columns), with model size on the horizontal axis. Read the numbers from Appendix Table 2.

Appendix Table 2 of the paper. Each cell gives accuracy (%) as "standard prompting → CoT prompting". The original marks in bold green the cells where CoT is higher. LaMDA values are averages over five orderings of the examples.

Read the GSM8K column family by family and the pattern is clear. The gains (CoT − standard) are as follows.

Values the editor obtained by subtraction from the GSM8K column of Appendix Table 2.

Below 10B the gains are negative in all three families. Appendix A.1 also writes that "chain of thought actually hurts performance for most models smaller than 10B parameters." The AQuA column moves differently: even at 10B and above, CoT is lower for PaLM 62B (25.6 → 22.4), LaMDA 68B (22.3 → 20.2) and LaMDA 137B (25.5 → 20.6). With five choices, guessing alone gives 20%, so the AQuA standard scores of the models below 10B (12.6–23.5%) are close to chance.

Second, the more complex the problem, the larger the gain. On GSM8K, where the baseline score is lowest, the scores of the largest GPT and PaLM more than doubled. On MAWPS SingleOp, which one step solves, the gain was negative or very small (Appendix Table 3). Here is the PaLM 540B row.

The PaLM 540B row of Appendix Table 3 of the paper. The characters of the subsets follow the descriptions in §3.2, Appendix A.3 and the caption of Table 3. The original groups SingleOp, SingleEq and AddSub together as easy problems solved in one or two steps.

Third, it rivals task-specific fine-tuned models. PaLM 540B exceeded the prior state of the art on GSM8K, SVAMP and MAWPS (on SVAMP standard prompting had already exceeded it), and came within 2% on AQuA and ASDiv. The prior state of the art on GSM8K is the 55% of Cobbe et al. (2021), who fine-tuned GPT-3 and added a verifier, and "surpassing even finetuned GPT-3 with a verifier" in the abstract is this comparison. Whether this difference exceeds the margin of chance is calculated in "Rereading with a statistician's eye".

An external calculator. Because there are cases where the solution is right but only the arithmetic is wrong, Appendix B also tried post-processing that recomputes the equations in the solution with Python eval and puts the results back. In the GSM8K column of Appendix Table 1, CoT rose from 14.3 → 17.8 for LaMDA 137B, 46.9 → 49.6 for GPT-3 175B and 56.9 → 58.6 for PaLM 540B.

People read the solutions. The authors randomly chose 50 correct and 50 incorrect solutions written by LaMDA 137B on GSM8K and read them. According to the main text (§3.2), all 50 correct ones had correct logic and arithmetic except two that got the right answer by coincidence. Of the 50 incorrect ones, 46% were almost-correct solutions with only minor mistakes, and 54% had major errors of semantic understanding or coherence. The breakdown in Appendix D.2 is calculator error only 8%, symbol mapping error 16%, one step missing 22% and needing major changes 54% (27 cases); the first three add up to 46%.

Two numbers disagree. The main text says "two" solutions were correct by coincidence, while Appendix D.1 says "only one". D.1 then lists six by adding five with minor imperfections, and the table also shows six examples, yet the caption of Table 9 says there were 7 imperfect cases and 43 others. D.2 also says that with the external calculator LaMDA 137B's GSM8K score went "from 14.3% to 17.3%", pointing to Table 2, but the calculator results are in Table 1 and the value is 17.8. Which is right cannot be settled from the original.

Errors fixed by scale (Appendix A.1). When the 45 items PaLM 62B got wrong were classified by hand, 540B fixed 6 of the 20 semantic understanding errors, 12 of the 18 one-step-missing errors and 4 of the 7 other errors (hallucination, repetition, symbol mapping errors). The authors see this as consistent with the hypothesis that larger models acquire semantic understanding and logical reasoning ability, while adding the caveat that scale is confounded with factors such as training compute.

4. Ablations — where does the effect come from? (§3.3, Figure 5, Appendix Tables 6 and 7)

If CoT has an effect, there could be several causes. The authors test three plausible explanations one at a time. Equation only has the model write only an equation before the answer. Variable compute only has it write dots (. . .) as many as the characters of the equation the solution needs. Reasoning after answer has it give the answer first and write the solution afterward. They target, respectively, the explanations that the effect comes from producing equations, from spending more intermediate tokens (computation), and from activating knowledge acquired in pretraining.

Figure 5 shows bars for five conditions (standard, the three variants and CoT) for LaMDA 137B and PaLM 540B on GSM8K. LaMDA's numbers are in Appendix Table 6. The paper gives no table of PaLM 540B's numbers by variant, so this chapter does not carry them.

The first part of Appendix Table 6 of the paper. The ± values are the standard deviations of runs with five different orderings of the examples (caption).

Equation only does not help on GSM8K. At 5.4 it is even lower than standard (6.5). The authors' interpretation is that the meaning of GSM8K items is too semantically challenging to translate straight into one equation without natural-language reasoning steps. On SVAMP, ASDiv and MAWPS, which have many one- or two-step problems, writing only an equation does raise the score (29.5 → 35.1, 40.1 → 45.9, 43.2 → 50.1). Appendix A.4 gives one example from LaMDA 137B.

QUESTION: Mike plays ping pong for 40 minutes. In the first 20 minutes, he scores
4 points. In the second 20 minutes, he scores 25% more points. How many total
points did he score?

EQUATION ONLY (WRONG ANSWER): (4 + 20 * 0.25) = 6. The answer is 6.

CHAIN OF THOUGHT (CORRECT): Mike played ping pong for 40 minutes. In the first 20
minutes, he scored 4 points. In the second 20 minutes, he scored 25% more points.
So he scored 25% more in the second 20 minutes. 4 x 1.25 = 5. So he scored 5
points in the second 20 minutes. So he scored 9 points in total. The answer is 9.

The equation-only side cannot decide what "25% more" is 25% of, and drags in the number 20. The side that writes a chain first establishes in a sentence the intermediate fact that the second 20 minutes gave 1.25 times 4 points. The authors' reading is that the step written as a sentence does the work of pinning down the meaning.

Increasing computation alone does not help. Writing only dots gives 6.4, about the same as standard. In this setting (LaMDA 137B, GSM8K), increasing the number of intermediate tokens alone did not explain the effect of CoT. The experiment tests the first of the four properties (the problem can be broken up and more computation allocated), and the result fits the authors' reading that natural-language steps matter. But it tested only one variant, dot tokens, so it does not go as far as showing the necessary condition that the computation must be expressed in natural language.

Writing the solution after the answer does not help. At 6.1 it is about the same as standard. If the solution sentences only had the effect of activating knowledge, they should help even when written after the answer. Since they do not, the authors conclude that the sequential reasoning contained in the chain is useful for reasons beyond knowledge activation. This result can be read alongside assumption A0 from Chapter 2 (the reasoning trace $z_t$ is generated before the action $a_t$). Reverse the order and the effect disappears. This connection is this book's reading.

Appendix Table 7 repeats the same ablations on five commonsense and symbolic tasks (Table 7 has no equation-only row). Variable compute only and reasoning after answer both stay near standard on date understanding, sports understanding, last-letter concatenation and coin flip. Only SayCan is an exception: reasoning after answer (83.3) is slightly higher than standard (80.8), and variable compute only (74.2) is lower than standard.

One number disagrees. The LaMDA 137B CoT SVAMP value in Table 6 is 36.7, while the same cell in Appendix Tables 2 and 1 is 37.5. GSM8K (14.3), ASDiv (46.6) and MAWPS (57.9) agree across the three tables, so only the SVAMP cell differs. The paper does not explain the difference.

5. Robustness — does it work whoever writes the examples? (§3.4, Figure 6, Appendix A.2)

Prompting is sensitive to examples. The authors open this section with the earlier finding (Zhao et al. 2021) that on GPT-3, merely changing the order of the examples moved SST-2 accuracy from 54.3% to 93.4%, and then try changing only the chains while keeping the questions of the examples.

• Annotators B and C. The chains in the main experiment were written by annotator A. Two co-authors independently wrote chains for the same examples (Appendix H). They were given no instructions on how to write them.
• Concise style. Annotator A rewrote the chains more briefly, following the style of the solutions in Cobbe et al. (2021).
• Examples α, β and γ from the GSM8K training set. Three sets of eight were randomly drawn from GSM8K training items that came with crowd workers' solutions. To fit the input window, items of 60 tokens or fewer were chosen, and to compare fairly with the examples the authors wrote, items of two steps or fewer (footnote 2).

The second part (Robustness) of Appendix Table 6 of the paper. The standard and annotator A rows are taken from the first part of the same table.

The authors write that every variant beats standard "by a large margin" on the four benchmarks (§3.4, Appendix A.2). Look at Table 6 and some cells show small differences. On SVAMP, variant α is 32.8 against 29.5, a difference of 3.3 percentage points, and annotator B a difference of 5.7. The authors read this to mean that the success of CoT does not depend on a particular style, and that since the GSM8K examples also worked on other benchmarks, the examples need not come from the same distribution as the test items (Appendix A.2).

Prompt engineering still matters (Appendix A.2). This is a caveat the authors add themselves. On coin flip, annotator A's chains scored 99.6% and annotator C's 71.4% (Appendix Table 7). There is also the anecdote that on a task of reversing the order of five items, two co-authors could not write chains that worked, and a third co-author wrote chains that solved it perfectly.

The rest of the checks in A.2 are summarized briefly. Except for coin flip, the standard deviation across orderings of the examples was small. Changing the number of examples generally kept the gain (Figure 11). The number of examples was fixed at 8 because the input length was limited to 1,024 tokens (Appendix D.3). A.2 writes that when the same prompts were used across models, only GPT-3's CSQA and StrategyQA showed no gain, and the authors record this as a limitation. But in Table 1, which A.2 points to as its basis, LaMDA 137B's AQuA also goes down from 25.5 to 20.6 (section 3 of "The paper in detail"). The "only" of A.2 leaves that cell uncounted. A.2 gives the standard prompting value for coin flip as "50.0%", while Table 7, which it points to, gives 49.0 ±2.1.

6. Commonsense reasoning and symbolic reasoning (§4, §5, Figure 3)

Figure 3 gathers ⟨input, chain, output⟩ examples of the nine tasks used in this paper on one page. Only the coin flip panel, which comes back in Exercise 2, is reproduced.

Coin Flip (state tracking)
  Q: A coin is heads up. Maybelle flips the coin. Shalonda does not flip the
     coin. Is the coin still heads up?
  A: The coin was flipped by Maybelle. So the coin was flipped 1 time, which
     is an odd number. The coin started heads up, so after an odd number of
     flips, it will be tails up. So the answer is no.

The Coin Flip panel of Figure 3 of the CoT paper. The original figure highlights the chain in color.

Commonsense reasoning (§4). The tasks are CSQA (commonsense multiple choice), StrategyQA (yes-or-no questions that need a multi-step strategy), date understanding (Date) and sports understanding (Sports) from BIG-bench, and SayCan (mapping natural-language instructions to sequences of robot actions). The two BIG-bench tasks have no training set, so the first ten items of the evaluation set were used as examples and the rest for evaluation. The SayCan evaluation set has 120 items (caption of Appendix Table 7).

Two rows from Appendix Table 4 of the paper. Cells are accuracy (%) as "standard → CoT".

The main text writes that as models grow, standard prompting also rises and CoT adds gains on top, and the gains appear largest for PaLM 540B. PaLM 540B surpassed an unaided sports enthusiast (84%) on sports understanding (95.4%) and the prior state of the art (69.4%) on StrategyQA. The gain on CSQA was small.

Where the text and the table differ. The §4 text gives the StrategyQA score as "75.6% vs 69.4%", but the PaLM 540B CoT value in Appendix Table 4 is 77.8. This can be read as a trace of only the table being updated in a revision, but the paper does not explain it. In presentation materials, say which one you took.

Symbolic reasoning (§5). Two toy tasks. Last-letter concatenation joins the last letters of the words in a name ("Amy Brown" → "yn"), and coin flip asks whether a coin is still heads up after several people have or have not flipped it. The examples are built only with two words and two flips, and evaluation is done at the same length (in-domain) and at longer lengths (OOD: three or four words, three or four flips).

The PaLM rows of Appendix Table 5 of the paper. Cells are accuracy (%) as "standard → CoT".

In-domain, PaLM 540B's CoT is nearly 100%. The authors themselves stress that these are toy tasks in which the structure of the solution is already in the examples and the same steps just have to be repeated with new symbols, and yet they note that small models failed. Out of domain, standard prompting fails on both tasks and CoT rises with scale. This is length generalization to inputs longer than the examples. On coin flip, PaLM 540B's standard prompting alone gets 98.1% in-domain, so that cell is weak evidence for the effect of CoT.

7. Rereading with a statistician's eye

This paper prints the numbers of items. That is the difference from ReAct's Table 1 in Chapter 6 (Appendix Table 12). So a binomial standard error can be attached to each accuracy. On GSM8K (1,319 items), the standard error near an accuracy of 14.3% is $\sqrt{0.143\times 0.857/1319}\approx 0.0096$, or about 1.0 percentage points, and near 56.9% it is about 1.4 percentage points. The ± values in the GSM8K column of Appendix Table 6 (0.2–1.0 percentage points) are a different quantity. They are the fluctuation when only the order of the examples changes on the same items, so they do not include the uncertainty from sampling items. Read these ± values as confidence intervals and you understate the uncertainty.

Reading the "emergence" of scale as a threshold model. Treat the table of gains by scale in section 3 of "The paper in detail" as a regression. With $x$ the base-10 logarithm of the number of parameters, the hinge model $\Delta (x)={\beta }_0+{\beta }_{1}max(0,x-x_0)$ with the constraints ${\beta }_0=0$ and ${\beta }_1>0$ is a rendering of the statement "no effect below the threshold $x_0$ and a growing one above it". Left unconstrained, it is a model that allows a constant effect ${\beta }_0$ below the threshold as well. The problem is the data. Each family has three to five points, and the points above the threshold are two for PaLM, two for LaMDA and one for GPT. With these one cannot tell a broken line from a smoothly bending curve, and the threshold can only be pinned down as an interval such as "between 8B and 62B for PaLM". Appendix A.1 uses emergence in the sense that "its success cannot be predicted only by extrapolating the performance of small scale models," but the failure of an extrapolation depends on which extrapolation model was used. It was later argued that with discontinuous metrics such as exact-match accuracy, even smooth change can look like a jump (Schaeffer et al. 2023). The negative gains of small models are seen with the same eye. LaMDA 420M's −2.2 percentage points is larger than the item sampling error (about 0.5 percentage points), but Appendix A.1 writes that small models often produced output that could not be parsed because they repeated themselves or did not reach a conclusion. Part of the negative gain can also be read as a product of measurement, namely failures of answer extraction.

Checking the numbers in the tables. With the numbers of items you can also run checks. Two things come up. First, the MAWPS column of Appendix Table 2 matches the simple average of the subset accuracies, not the accuracy over the 2,065 items of the four subsets combined. The four Table 3 values for PaLM 540B standard prompting (94.1, 86.5, 93.9, 42.2) have a simple average of 79.2, the same as Table 2, while the average weighted by the numbers of items (562, 508, 395, 600) is 77.1. The six other rows the editor checked (UL2 20B, GPT-3 175B, Codex, PaLM 8B and 62B, LaMDA 137B) also match the simple average within rounding. The paper does not state how it aggregated. Second, the accuracies of models that used only one ordering of the examples (all but LaMDA) should be "integer ÷ number of items" rounded, but four values in the SingleEq (508 items) column (UL2 20B's 18.0 and 20.2, PaLM 8B's 28.2, PaLM 540B's 86.5) cannot arise from 508 items. For example, 439/508 = 86.42% and 440/508 = 86.61%, so no integer gives 86.5. Why this happens cannot be learned from the paper. AQuA (254 items) and the other MAWPS subsets pass this check, and the benchmarks with 1,000 items or more (SVAMP 1,000, GSM8K 1,319, ASDiv 2,096) realize every 0.1-percentage-point value with some integer, so this check cannot catch anything there.

Exercise 1 — measuring differences on the 1,319 GSM8K items. (a) How many items are PaLM 540B's 17.9% (standard) and 56.9% (CoT)? Find the Wilson 95% confidence interval for each. (b) The two methods were compared on the same 1,319 items. Treated as independent samples, what is the standard error of the difference of 39.0 percentage points? Since the item-level results are unknown, also estimate the range the standard error of a paired comparison could take. (c) The abstract writes that PaLM 540B + CoT (56.9%) beat fine-tuned GPT-3 + verifier (55%) to set a new state of the art. Test this difference the same way. What about the 58.6% with the external calculator? (d) In the ablations, may LaMDA 137B's standard 6.5% and "reasoning after answer" 6.1% be read as "about the same"?

Exercise 2 — fluctuation by who wrote the chains. The GSM8K column of the table in section 5 of "The paper in detail" has seven CoT variants (annotators A, B and C, concise style, GSM8K examples α, β and γ). (a) Find the mean and standard deviation of the seven values, and compare them with the ± in the table (the standard deviation over orderings of the examples) and with the binomial standard error (about 1.0 percentage points). (b) If the authors had picked the best of the seven to report, how much higher than the mean would the score be expected to come out? Assume the seven values came independently from the same normal distribution (if the distribution is not normal, the expected value of the maximum changes). The expected maximum of seven standard normal draws can be found with a short simulation (for example, draw 100,000 batches of 7 normal random numbers and average the maxima). Where does annotator A's value, the one actually reported, sit? (c) Annotators A, B and C on coin flip (Appendix Table 7) score 99.6 ±0.3, 77.5 ±7.9 and 71.4 ±11.1. Can "robust to different annotators" be said for this task too? (d) To report the score of "the method called CoT" as one number, what is better to write, and how?

8. Limitations the paper states

• Does the model really "reason"? That CoT imitates the human thought process does not answer whether the neural network actually reasons. It is left as an open question (§6).
• Annotation cost. Adding chains to a few examples costs little, but large-scale annotation for fine-tuning could be expensive; the paper notes that this might be overcome with synthetic data generation or zero-shot generalization (§6). DeepSeek-R1 and Zero-shot CoT among the side papers touch these two directions respectively (this book's reading).
• Correct reasoning paths are not guaranteed. Wrong paths lead both to wrong answers and to answers that happen to be right (§6, Appendix D.2), and on multiple-choice and binary classification tasks it is much more likely to get the right answer through a wrong path (Appendix D.1). The NeurIPS checklist adds that the authors do not suggest using the generated chains in a factual manner or in any real-world scenario.
• It appears only in large models. That makes it expensive to use in real services. Research on eliciting reasoning in small models is left as future work (§6).
• Gains that do not carry over between models. The same prompts produced no gain on GPT-3's CSQA and StrategyQA (Appendix A.2).
• Scale confounded with other factors. Model scale is confounded with factors such as training compute, and the paper writes that what makes reasoning possible must be studied separately through pretraining data, architecture and objective functions (Appendix A.1).

Side papers — how they differ from the main paper

Three side papers (Zero-shot CoT, self-consistency, DeepSeek-R1) are treated only within the scope of their abstracts, and one earlier work the main paper compares itself with directly is added, along with the next two chapters of this book.

Zero-shot CoT (Kojima et al. 2022 · NeurIPS 2022). With no examples, it simply adds the one sentence "Let's think step by step" before each answer. According to the abstract, on text-davinci-002 GSM8K rose from 10.4% to 40.7% and MultiArith from 17.7% to 78.7%, with similar results on PaLM 540B. The main paper gives eight examples with chains written by people for each task, while Kojima et al. get the same thing done with one sentence unrelated to the task. The baselines differ (answering with no examples against standard prompting with examples), so the numbers of the two papers are not set against each other. The "zero-shot generalization" that §6 of the main paper names as a way past annotation cost is this direction.

Self-consistency (Wang et al. 2022 · ICLR 2023 · the CoT-SC of Chapter 6 of this book). The abstract proposes replacing CoT's "naive greedy decoding". It samples several reasoning paths and picks the most consistent answer by "marginalizing out the sampled reasoning paths." The improvements the abstract reports are GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2%, StrategyQA +6.4% and ARC-challenge +3.9% (the abstract does not say for which model). It is the paper that §3.1 of the main paper cites as follow-up work, from the same group with the first and second authors swapped. The difference is the number of paths (one against many), and the cost grows with the number of samples.

DeepSeek-R1 (DeepSeek-AI 2025 · Nature, vol. 645). The abstract starts from the point that the success of CoT is heavily contingent on human-annotated demonstrations, and claims to have elicited reasoning ability through pure reinforcement learning alone, "obviating the need for human-labeled reasoning trajectories." It says that patterns such as self-reflection, verification and strategy switching emerged during training, and that it outperformed supervised models on verifiable tasks such as mathematics and coding. The main paper holds the parameters $\theta$ fixed and elicits chains through examples; R1 changes $\theta$ to build the habit of writing chains into the model. In the language of Chapter 2, it is the $z$ mode against the $\theta$ mode.