Pretraining and generative models (2017–2022)

Pretraining and fine-tuning (2018)

In the first year after the Transformer of chapter 5, the use of language models changed. Howard and Ruder's ULMFiT (18 January 2018) fine-tuned a language model pretrained on a general corpus to target tasks with discriminative fine-tuning, slanted triangular learning rates and gradual unfreezing, cutting error by 18–24% on most of six classification datasets. ELMo (Peters et al.) (15 February 2018) is a contextualized word representation formed as a task-specific combination of all internal layers of a bidirectional LSTM language model. GPT-1 (Radford et al.) (11 June 2018, OpenAI report) trained a multilayer Transformer decoder on unlabeled text and fine-tuned it per task, improving the state of the art on 9 of 12 natural-language-understanding datasets. BERT (Devlin et al.) (11 October 2018) pretrained a Transformer encoder with a masked language model and next-sentence prediction and reported new state-of-the-art results on eleven tasks; its reference list cites GPT-1. The four differ in architecture, two LSTMs, one decoder, one encoder, but the first clause is the same: pretrain on large text first.

T5 (Raffel et al.) (23 October 2019) cast every text problem as text in, text out, compared pretraining objectives, architectures and datasets systematically, and released the C4 corpus. Retrieval-augmented generation (Lewis et al.) (22 May 2020) combined a pretrained generator with a retriever over a dense Wikipedia index and fine-tuned the pair end to end, one of the original texts for what the agents of chapter 7 come to call tools and memory.

Measuring scale

Hestness et al. (1 December 2017, Baidu) measured learning curves in translation, language modeling, images and speech and reported that generalization error falls as a power law in training-set size, with exponents (usually −0.07 to −0.35) that model changes barely move. Kaplan et al., "Scaling Laws for Neural Language Models" (23 January 2020) wrote that Transformer language-model loss follows power laws in model size, dataset size and compute, some trends spanning more than seven orders of magnitude, and derived that under a fixed compute budget it is efficient to train very large models and stop before convergence. Four months later GPT-3 (Brown et al.) (28 May 2020) trained a 175-billion-parameter model and evaluated it in few-shot settings where task and demonstrations are given only as text. Hoffmann et al., "Training Compute-Optimal Large Language Models" (29 March 2022) trained over 400 models and wrote that compute-optimal training scales model size and token count in equal proportion; Chinchilla, a 70-billion model trained on four times the data at the same compute, outperformed larger models. It revised Kaplan's conclusion. The Switch Transformer of Fedus, Zoph and Shazeer (11 January 2021; JMLR 2022) simplified the sparse mixture-of-experts routing of chapter 5 and pretrained a trillion-parameter model. PaLM (5 April 2022) trained a 540-billion-parameter dense model on 6,144 TPU v4 chips, and Wei et al., "Emergent Abilities" (15 June 2022) defined as emergent those abilities absent in smaller models and present in larger ones, which extrapolation cannot predict.

Eliciting reasoning

Chain-of-thought prompting (Wei et al.) (28 January 2022) reported that a few exemplars containing intermediate reasoning steps improve arithmetic, commonsense and symbolic reasoning. Self-consistency (Wang et al.) (21 March 2022) sampled several reasoning paths and chose the most consistent answer, adding 17.9 points on GSM8K, and Kojima et al. (24 May 2022) showed that the single sentence "Let's think step by step" enables zero-shot reasoning. The three papers change the prompt and the decoding, not the model's weights; ReAct in chapter 7 starts from exactly this point.

Alignment

Stiennon et al., "Learning to summarize from human feedback" (2 September 2020; NeurIPS 2020) carried the method of Christiano et al. (chapter 5) to language models, training a reward model on human comparisons of summaries and fine-tuning a summarization policy with PPO. InstructGPT (Ouyang et al.) (4 March 2022), which cites both that paper and Christiano 2017, fine-tuned GPT-3 with supervised learning on labeler demonstrations and then with reinforcement learning from human feedback on labeler rankings. One sentence of the result: labelers preferred the outputs of the 1.3-billion InstructGPT to those of the 175-billion GPT-3. It belongs to the problem Amodei et al. (chapter 5) called "scalable oversight". Bender et al., "On the Dangers of Stochastic Parrots" (3 March 2021, FAccT) asked how big is too big and recommended weighing environmental and financial costs and documenting data first. Bommasani et al., "On the Opportunities and Risks of Foundation Models" (16 August 2021, 214 pages) named models trained on broad data, generally with self-supervision at scale, and adapted to many downstream tasks "foundation models".

Beyond language

The Vision Transformer of Dosovitskiy et al. (22 October 2020; ICLR 2021) applied a standard Transformer directly to sequences of image patches and reported that, pretrained on large data, it matches or exceeds convolutional networks; CLIP (Radford et al.) (26 February 2021; ICML 2021) trained image and text encoders on 400 million internet image–text pairs to predict which caption goes with which image, and showed zero-shot transfer by naming visual concepts in natural language. These are the two original texts of the Transformer leaving language. AlphaFold (Jumper et al.) (15 July 2021, Nature) built knowledge of protein structure and multiple-sequence alignments into the network and reached accuracy competitive with experimental structures in a majority of CASP14 cases; the 2024 Nobel Prize in Chemistry citation points to it. Codex (Chen et al.) (7 July 2021) introduced a GPT fine-tuned on GitHub code and the HumanEval benchmark (28.8%, 70.2% with 100 samples), leading to SWE-bench in chapter 7. In games, AlphaZero (7 December 2018, Science) learned chess, shogi and Go with one algorithm, and MuZero (19 November 2019) planned with a learned model without even being given the rules. GPT-2 (OpenAI blog, 14 February 2019) reported a 1.5-billion model beginning to learn tasks zero-shot from WebText. The GPT-4 technical report (15 March 2023) reported a multimodal model with image and text input while stating that it gives no details of architecture, size, data or training method, and LLaMA (Touvron et al.) (27 February 2023) released 7- to 65-billion models trained on public data only, reporting that the 13-billion model outperforms GPT-3 on most benchmarks. The contrast between the two, a closed report and open weights, is the background of chapter 7.

What was and was not checked

All 29 papers were read. Only Bender et al. is quoted from the abstract as reconstructed by OpenAlex, without a page, because the publisher's page blocked automated access. arXiv papers were read in the latest downloaded version, in some cases a revision long after v1 (the card names the version). Candidates that did not pass the rule include World Models (2018), StyleGAN (2018), "The Bitter Lesson" (2019, a blog post), AlphaStar (2019), Pluribus (2019), DDPM (2020), score-based generation (2020), DALL·E (2021), LoRA (2021), S4 (2021), latent diffusion (2021), Constitutional AI (2022), Whisper (2022), DPO (2023) and Mamba (2023). Generative modeling and efficiency are under-represented, and we record that the two-textbook rule still works narrowly for work after 2020.