Deep learning (2012–2017)

2012

Krizhevsky, Sutskever and Hinton, "ImageNet Classification with Deep Convolutional Neural Networks" was presented at NIPS in December 2012: a network of five convolutional and three fully connected layers, 60 million parameters, trained with non-saturating neurons, a GPU convolution implementation and dropout, reaching 17.0% top-5 error on the LSVRC-2010 test set and, as an ensemble, 15.3% in ILSVRC-2012. A month earlier, in the November IEEE Signal Processing Magazine, "Deep Neural Networks for Acoustic Modeling in Speech Recognition" gave the shared view of four groups (Toronto, MSR, Google, IBM) that DNN acoustic models beat GMM-HMMs on TIMIT and five large-vocabulary tasks. Cireşan et al.'s multi-column deep networks (arXiv v1 13 February 2012; CVPR 2012) and Le et al.'s large-scale unsupervised feature learning (arXiv v1 29 December 2011; ICML 2012) belong to the same months; the latter reported units responding to faces learned from 10 million unlabeled images. It is the year in which chapter 4's conditions — ImageNet, GPU training, rectified units, layer-wise initialization — came together.

Learning representations (2013–2014)

word2vec (Mikolov et al.) (arXiv v1, 16 January 2013) proposed CBOW and skip-gram for learning word vectors from very large corpora at low cost; the follow-up paper of October added negative sampling and phrase vectors. GloVe (Pennington, Socher and Manning) (October 2014, EMNLP) did the same from a co-occurrence matrix. These three carried the distributed representations of Bengio 2003 (chapter 4) into practice. Graves's sequence generation (4 August 2013) showed an LSTM synthesizing handwriting by predicting one data point at a time, and the Neural Turing Machine of Graves, Wayne and Danihelka (20 October 2014) coupled a network to external memory through attention, and five days earlier the memory networks of Weston, Chopra and Bordes (15 October 2014) combined a readable and writable long-term memory with inference components for question answering. The two are the original texts of the memory strand in chapter 7.

In vision, R-CNN (arXiv v1, 11 November 2013; CVPR 2014) applied CNNs to region proposals and changed detection; VGG (4 September 2014) and GoogLeNet (17 September 2014) pushed depth to 16–22 layers. Szegedy et al., "Intriguing properties of neural networks" (21 December 2013) showed that hardly perceptible perturbations can make a network misclassify, opening the study of adversarial examples. On the training side, dropout (June 2014, JMLR), Adam (22 December 2014), batch normalization (11 February 2015) and layer normalization (21 July 2016) became handles that later networks widely use.

Three generative models

The variational autoencoder of Kingma and Welling (20 December 2013), the generative adversarial network of Goodfellow et al. (10 June 2014) and the diffusion model of Sohl-Dickstein et al. (12 March 2015) appeared within two years. They solve the same problem by different routes, training a model that produces new samples from the data distribution by gradient methods: the GAN as a minimax game between generator and discriminator, the VAE by reparameterizing a variational bound, diffusion by learning the reverse of a forward process that slowly destroys structure. DCGAN (19 November 2015) and WaveNet (12 September 2016) continued those routes for images and raw audio.

From translation to attention

The RNN encoder–decoder of Cho et al. (3 June 2014) and seq2seq (Sutskever, Vinyals and Le) (10 September 2014) set up the frame in which one RNN puts a sentence into a vector and another takes it out. Bahdanau, Cho and Bengio (1 September 2014) treated that fixed-length vector as a bottleneck and proposed an extension that, for each target word, softly searches the source for relevant positions, learning "to align and translate jointly": this mechanism is attention. Luong et al. (17 August 2015) refined it into global and local attention; the subword units of Sennrich et al. (31 August 2015) solved the rare-word problem with byte-pair encoding; Google's GNMT (26 September 2016) bundled all of it into an 8-layer LSTM production system. Then Vaswani et al., "Attention Is All You Need" (arXiv v1, 12 June 2017; NIPS 2017) removed recurrence altogether: 28.4 BLEU English–German, 41.8 BLEU English–French. The language models of chapter 6 stand on this architecture.

Deep reinforcement learning

Mnih et al., "Playing Atari with Deep Reinforcement Learning" (19 December 2013) trained a convolutional network with a variant of Q-learning to play seven games from raw pixels; the Nature version (25 February 2015) added a periodically updated target network to the experience replay already in the 2013 paper and reported a level comparable to a professional games tester on most of 49 games. Both cite TD-Gammon from chapter 4; the Nature paper's reference list does not include the 2013 preprint. TRPO (19 February 2015) and DDPG (9 September 2015) extended policy gradients to continuous actions and large networks; A3C (Mnih et al.) (4 February 2016) ran several actor-learners asynchronously instead of using experience replay and exceeded the Atari state of the art on a single multicore CPU. PPO (Schulman et al.) (20 July 2017) simplified TRPO with a surrogate objective of clipped probability ratios, the algorithm later used by reinforcement learning from human feedback in chapter 6. Christiano et al., "Deep reinforcement learning from human preferences" (12 June 2017) learned a reward model from human preferences between pairs of trajectory segments and trained a policy on it, learning Atari and MuJoCo tasks without the true reward; its references cite Amodei et al. 2016 and DQN, and it is the direct forerunner of InstructGPT in chapter 6. AlphaGo (Silver et al.) (Nature online, 27 January 2016) combined a supervised policy network, a reinforcement-learned policy network and a value network with the Monte Carlo tree search of chapter 4 and beat the European champion 5–0; AlphaGo Zero (19 October 2017) beat that AlphaGo 100–0 trained by self-play alone, without human games. The AlphaGo paper cites Sutton 1988 and Sutton 1999.

At the edges

Residual learning (He et al.) (10 December 2015) let layers learn residuals with respect to their inputs, trained 152 layers and won ILSVRC 2015 with an ensemble's 3.57% top-5 ImageNet error; the box on the priority of residual connections follows. Distillation (9 March 2015) trains a small model from the knowledge of a large one. The Nature review of LeCun, Bengio and Hinton (27 May 2015) and Schmidhuber's Neural Networks overview (arXiv v1 30 April 2014; January 2015 issue) wrote the history of the same field differently in the same year; among the latter's 888 references, citation lines to 47 papers of this version were confirmed (41 in OpenAlex, 6 from the reference list). The sparsely gated mixture of experts of Shazeer et al. (23 January 2017; ICLR 2017) revived the mixture of experts of Jacobs 1991 (chapter 4), letting a gating network pick a sparse combination of thousands of experts per example in a 137-billion-parameter language model, the forerunner of the Switch Transformer in chapter 6. Amodei et al., "Concrete Problems in AI Safety" (21 June 2016) set side effects, reward hacking, safe exploration and distributional shift as research problems and leads to the alignment work of chapter 6. Google's TPU paper (arXiv v1 16 April 2017; ISCA 2017) is the chapter's only hardware point: an inference ASIC deployed in Google datacenters since 2015, compared with the CPU and GPU of its time.

What was and was not checked

All 46 papers were read. arXiv papers were read in the latest downloaded version, so quotation pages refer to that version, which the card names. The four Nature papers (DQN, the review, AlphaGo, AlphaGo Zero) were read in publisher-typeset copies or author manuscripts on author, laboratory or repository sites. Candidates that did not pass the rule include Faster R-CNN, YOLO, U-Net, Highway Networks, Show-Attend-Tell, GCN and DeepStack.