The originals of the 1990s ask about guarantees. Support-vector networks (1995), the lasso (1996), random forests (2001) and conditional random fields (2001) tried to answer with statistics why learning generalizes, while Q-learning (1992), REINFORCE (1992) and policy gradients (1999) laid down the mathematics of reinforcement learning.
Neural networks did not vanish from this chapter. Hochreiter's 1991 thesis analyzed vanishing gradients; LSTM and bidirectional RNNs (1997) and LeNet (1998) followed; the two deep-belief-network papers of 2006 and ImageNet (2009) set the conditions for chapter 5.
Statistical machine translation (1990) and the neural language model (2003) head the longest lineage on the site, the one that runs to the language models of chapter 7. In the 1990s that convention calls a winter, papers with "neural network" in the title were increasing (winters page).
Language as probability
Brown et al., "A statistical approach to machine translation" (June 1990, Computational Linguistics) set translation as choosing the source sentence S that maximizes Pr(S)Pr(T|S), a language model times a translation model. That one-line decomposition is the skeleton of thirty years of statistical and neural translation; the paper cites the EM algorithm of chapter 3. At ICML in June 2001, conditional random fields (Lafferty, McCallum and Pereira) modeled the labeling and segmentation of sequences as conditional probabilities, avoiding the label bias of maximum-entropy Markov models; in 2003, in JMLR, latent Dirichlet allocation (Blei, Ng and Jordan) set up a three-level Bayesian model with each document a mixture of topics. The same journal, the same year, carried Bengio et al., "A Neural Probabilistic Language Model" (February 2003), which learned a distributed representation for each word and the probability of word sequences in one network. Its sentence on fighting the curse of dimensionality by learning a distributed representation for words is where word2vec in chapter 5 and the language models of chapter 6 point back.
The statistics of generalization
Boser, Guyon and Vapnik (July 1992, COLT) is the training algorithm for optimal-margin classifiers; the original could not be opened for this version. Its extension to non-separable data, Cortes and Vapnik, "Support-vector networks" (September 1995, Machine Learning), was read: "The support-vector network is a new learning machine for two-group classification problems." Inputs are mapped non-linearly into a very high-dimensional feature space where a linear decision surface is drawn; the reference list includes the 1986 Nature backpropagation paper (under a misprinted title). Tibshirani's lasso (1996, JRSS B) estimates linear models under a bound on the sum of absolute coefficients, so that some coefficients become exactly zero; Breiman's "Random Forests" (October 2001) classifies by the vote of trees grown with independent, identically distributed random vectors and shows that the generalization error converges as trees are added. All three ask in mathematics why learning generalizes: answers of the 1990s and 2001 to Valiant's question in chapter 3.
The mathematics of reinforcement learning
Two papers appeared in one issue of Machine Learning in May 1992. Watkins and Dayan, "Q-learning" completed the convergence proof the 1989 thesis had only outlined: with action values represented discretely and every action sampled repeatedly in every state (under conditions on the learning rate), the action values converge to the optimum with probability 1. Williams's REINFORCE showed that the weight adjustments of a network of stochastic units follow the gradient of expected reinforcement, and cites Barto 1983 from chapter 3. Tesauro's TD-Gammon (March 1995, CACM) reported a network that trained itself by self-play with TD(λ) to play backgammon; both DQN papers in chapter 5 cite it. The ACM printing could not be opened, so it was read in an authorized transcription. Kaelbling, Littman and Cassandra (May 1998, Artificial Intelligence) brought partially observable Markov decision processes to the problem of choosing actions in AI. Sutton and Barto's Reinforcement Learning: An Introduction (1998, MIT Press) gathered the lineage in one framework, as its preface promises "a clear and simple account of the key ideas and algorithms", and is the most-cited book in this version, cited by twelve other papers. The next year the policy-gradient theorem of Sutton et al. (NIPS 12) represented the policy by its own function approximator and showed that the gradient of expected reward can be estimated with an approximate action-value function. AlphaGo in chapter 5 cites both.
Neural networks continued
Hochreiter's diploma thesis at TU Munich (15 June 1991, in German) analyzed why backpropagation in recurrent networks fails to learn from inputs far in the past, by following the backward flow of error signals, and proposed memory units. Bengio, Simard and Frasconi (March 1994, IEEE TNN) showed the same problem by another route: the longer the time span of the dependencies, the harder gradient learning becomes. The answer to both analyses is Hochreiter and Schmidhuber, "Long Short-Term Memory" (1 November 1997, Neural Computation): constant error flow inside units whose access is controlled by multiplicative gates, bridging lags of more than 1000 steps. The bidirectional RNN of Schuster and Paliwal appeared the same month. Jacobs et al., "Adaptive Mixtures of Local Experts" (February 1991) combined several expert networks with a gating network, the original text to which the sparse mixtures of experts in chapter 6 point back.
LeCun, Bottou, Bengio and Haffner, "Gradient-based learning applied to document recognition" (November 1998, Proceedings of the IEEE) wrote that hand-crafted feature extraction can be replaced by learning machines operating directly on pixel images. This paper (LeNet), continuing the 1989 zip-code paper of chapter 3, describes a deployed bank-check reader. Hinton's contrastive divergence (technical report 2000; Neural Computation, August 2002) gave a way to train products of experts, and three papers followed in 2006. The deep belief network of Hinton, Osindero and Teh (July 2006) learned deep networks one layer at a time using complementary priors; Hinton and Salakhutdinov (28 July 2006, Science) showed with the same layer-wise pretraining that deep autoencoders learn low-dimensional codes better than PCA; Bengio et al. (NIPS 2006, December) confirmed experimentally the hypothesis that the greedy layer-wise strategy "mostly helps the optimization". Glorot and Bengio's initialization (May 2010, AISTATS) and Nair and Hinton's rectified linear units (June 2010, ICML) supplied two handles later deep networks use: an initialization scheme and the rectified unit (introduced here in RBMs). Raina, Madhavan and Ng (June 2009, ICML) argued that graphics processors far surpass multicore CPUs and could change the applicability of deep unsupervised learning, reporting deep-belief-network training up to 70 times faster; its references cite both Hinton papers of 2006. Cireşan et al. (arXiv v1 1 March 2010; Neural Computation) reported 0.35% error on MNIST from large, plain multilayer perceptrons trained on GPUs without unsupervised pretraining.
Search, games, data
Campbell, Hoane and Hsu, "Deep Blue" (January 2002) described the search engine, evaluation function and game database of the machine that beat Kasparov in 1997. In 2006 another road opened in Go: Coulom (May, Turin) combined tree search with Monte Carlo evaluation through a backup operator that moves from averaging to minimax, and the UCT of Kocsis and Szepesvári (September, ECML) guided that search with bandit ideas. AlphaGo's search in chapter 5 comes from these two. The recurrent-network language model of Mikolov et al. (26 September 2010, Interspeech) and Collobert et al., "Natural Language Processing (almost) from Scratch" (arXiv 2 March 2011; JMLR) are the two points where feature engineering in language processing began to give way to learned representations. And Deng et al., "ImageNet" (June 2009, CVPR) reported 3.2 million images organized by the WordNet hierarchy and collected through Mechanical Turk. Chapter 5 does not begin without this database.
Boosting, and two events
Schapire's "The strength of weak learnability" (FOCS 1989; Machine Learning, 1990) proved constructively that a weak learner, only slightly better than random guessing, is equivalent to a strong learner of arbitrarily high accuracy; it cites Valiant 1984 from chapter 3. AdaBoost (Freund and Schapire) (EuroCOLT, March 1995; JCSS, 1997) derived from multiplicative weight updates a boosting algorithm that needs no prior knowledge of the weak learner's performance; this version read the authors' manuscript. Conditional random fields in this chapter and the candidate Viola–Jones cite it. Quinlan's C4.5 (1993), the book that followed ID3 of chapter 3, could not be read. Breiman's "Bagging predictors" appeared first as Berkeley Statistics technical report 421 in September 1994 and in Machine Learning in August 1996: predictors built on bootstrap replicates and combined by averaging or plurality vote, with the finding that accuracy improves when the prediction method is unstable, the forerunner of random forests (2001).
Two events stand at the end of the statistical-learning era. "Stanley" (Thrun et al.) (25 September 2006, Journal of Field Robotics) describes the autonomous vehicle that first completed the 2005 DARPA Grand Challenge desert course, with software the paper says was built on machine learning and probabilistic reasoning. "Checkers Is Solved" (Schaeffer et al.) (19 July 2007, Science) announced a computational proof, from endgame databases and proof-tree search run almost continuously since 1989, that checkers is weakly solved: perfect play by both sides from the starting position is a draw. Its references include Shannon 1950: one end of the lineage of measuring intelligence by games that began in chapter 1.
What was and was not checked
Of 41 papers, 35 were read in the original. C4.5 could not be read; Raina 2009 was read in an archived copy of the ICML proceedings, bagging in the journal's open-access PDF. Boser 1992 was not read; Bengio 1994 was read in an author preprint, Russell–Norvig in the 4th-edition preface, Deep Blue in the authors' manuscript of August 2001, and Sutton–Barto 1998 in the authors' HTML of the first edition, which has no page numbers. Candidates of the period that did not pass the rule include PageRank (1999), SIFT (1999), gradient boosting (2001), Viola–Jones (2001), Breiman's "two cultures" (2001), "The Unreasonable Effectiveness of Data" (2009) and Watson (2010). Several papers of statistical learning and retrieval are missing, and we record it as a limit of a reference set weighted toward AI and neural-network textbooks.
Lineage
Solid lines are confirmed citations (OpenAlex or the citing paper's own reference list); dotted lines are influences stated in textbooks without a confirmed citation. Vertical position is the theme group.
Paper list (41)
The strength of weak learnability
Robert E. Schapire · First public 1989 (conference) · Venue: Machine Learning 5(2) · Pages 197–227 · Citations (OpenAlex) 2,453 (checked 2026-11-03)
DOI · FOCS 1989 (30 Oct-1 Nov 1989), pp. 28-33; COLT 1989 proceedings p. 383 (1989, month not established); Machine Learning 5(2):197-227, 1990-06 (Crossref published-print)
In this paper, it is shown that these two notions of learnability are equivalent.p. 197
What changed. Proves that weak learnability (hypotheses only slightly better than random guessing) and strong learnability are equivalent in the distribution-free (PAC) model. Gives a constructive method that converts a weak learning algorithm into one achieving arbitrarily high accuracy by filtering the example distribution.
Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Fredrick Jelinek, John Lafferty et al. · First public 1990-06 (journal issue) · Venue: Computational Linguistics 16(2) · Pages 79–85 · Citations (OpenAlex) 1,690 (checked 2026-11-03)
DOI · Related earlier conference paper: Brown et al., 'A Statistical Approach to Language Translation', COLING Budapest 1988, Vol. 1 (ACL Anthology C88-1016; different title, and Lafferty is not listed)
In this paper, we present a statistical approach to machine translation.79
What changed. Frames translation as choosing the source sentence S that maximizes Pr(S)Pr(T|S), combining a language model with a translation model. Describes applying this approach to translation from French to English and gives preliminary results.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, Geoffrey E. Hinton · First public 1991-02 (journal issue) · Venue: Neural Computation 3(1) · Pages 79–87 · Citations (OpenAlex) 5,339 (checked 2026-11-03)
DOI · Idea first presented by Jacobs and Hinton at the Connectionist Summer School, Pittsburgh, 1988 (footnote 1, p. 79); received 27 Jul 1990, accepted 1 Nov 1990 (not public dates)
We present a new supervised learning procedure for systems composed of many separate networks, each of which learns to handle a subset of the complete set of training cases.79
What changed. The paper proposes a supervised learning procedure for a system of separate 'expert' networks plus a gating network that decides which expert handles each training case, described as a modular version of a multilayer supervised network or an associative version of competitive learning, and shows it dividing a vowel discrimination task into subtasks.
Hochreiter · First public 1991-06-15 (thesis) · Venue: Diplomarbeit, Institut für Informatik, Technische Universität München (Aufgabensteller: W. Brauer; Betreuer: J. Schmidhuber) · Pages 70 numbered pp. (74 PDF pages)
In dieser Arbeit wird untersucht, wie man an das Problem der langen Lernzeit bei Netzeingaben, welche später noch Einfluß auf die gewünschte Ausgabe haben sollen, herangehen kann.1
What changed. The thesis analyzes why backpropagation learning algorithms for recurrent networks take a long time to learn to use inputs lying far in the past, studying the backward flow of error signals, and investigates remedies: shortening input sequences while keeping the relevant information, and an architecture with 'Speicherknoten' (memory units) that can carry information over arbitrarily long periods.
Christopher J. C. H. Watkins, Peter Dayan · First public 1992-05 (journal issue) · Venue: Machine Learning 8(3-4) (Technical Note) · Pages 279–292 · Citations (OpenAlex) 9,084 (checked 2026-11-03)
DOI · Algorithm and proof outline in Watkins (1989) PhD thesis 'Learning from delayed rewards', Cambridge (cited)
We show that Q-learning converges to the optimum action-values with probability 1 so long as all actions are repeatedly sampled in all states and the action-values are represented discretely.279
What changed. This technical note presents in detail the convergence proof for Q-learning outlined in Watkins (1989), showing convergence to the optimal action-values with probability 1 when all actions are repeatedly sampled in all states and action-values are represented discretely, and sketches extensions to non-discounted absorbing environments and to changing many Q values per iteration.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J. Williams · First public 1992-05 (journal issue) · Venue: Machine Learning 8(3-4) · Pages 229–256 · Citations (OpenAlex) 6,997 (checked 2026-11-03)
DOI · Related earlier reports by the author (cited): UCSD ICS TR 8605 (1986); NU-CCS-87-3 (1987); ICNN 1987 'A class of gradient-estimating algorithms for reinforcement learning in neural networks'
This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units.229
What changed. The article presents REINFORCE algorithms, a class of associative reinforcement learning algorithms for networks of stochastic units, and shows that their weight adjustments lie along the gradient of expected reinforcement in immediate-reinforcement and certain delayed-reinforcement tasks without explicitly computing gradient estimates; it also shows how they can be integrated with backpropagation.
A training algorithm for optimal margin classifiers
Bernhard E. Boser, Isabelle Guyon, Vladimir N. Vapnik · First public 1992-07 (conference) · Venue: Proceedings of the Fifth Annual Workshop on Computational Learning Theory (COLT '92), ACM · Pages 144–152 (per Crossref; not seen) · Citations (OpenAlex) 11,991 (checked 2026-11-03)
J. R. Quinlan · First public 1993 (book) · Venue: Morgan Kaufmann Publishers, San Mateo, CA (Morgan Kaufmann Series in Machine Learning) · Citations (OpenAlex) 23,428 (checked 2026-11-03)
URL · Reviewed in Machine Learning 16(3), 1994-09 (Crossref 10.1007/bf00993309)
What changed.
Cites (within this list): none Cited by (within this list): none
Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, P. Simard, Paolo Frasconi · First public 1994-03 (journal issue) · Venue: IEEE Transactions on Neural Networks 5(2) · Pages 157–166 (journal); preprint has 35 pp. with its own pagination · Citations (OpenAlex) 8,543 (checked 2026-11-03)
DOI · Earlier related conference paper: Bengio, Frasconi & Simard, 'The problem of learning long-term dependencies in recurrent networks', IEEE ICNN 1993, San Francisco (preprint ref [3])
We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases.1 (author preprint)
What changed. The paper shows why gradient-based learning in recurrent networks becomes increasingly difficult as the time span of the dependencies to be captured grows, exposing a trade-off between efficient learning by gradient descent and latching on to information for long periods, and considers alternatives to standard gradient descent.
Leo Breiman · First public 1994-09 (tech report) · Venue: Machine Learning 24(2), Kluwer Academic Publishers · Pages 123–140 · Citations (OpenAlex) 16,328 (checked 2026-11-03)
DOI · Machine Learning 24(2) journal issue 1996-08 (Crossref published-print/issued 1996-08; printed header 'Machine Learning, 24, 123-140 (1996)'); journal received 1994-09-19, accepted 1995-01-02, final manuscript 1995-11-21 (printed on p. 140; not public dates)
Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor.p. 123
What changed. Introduces bagging ('bootstrap aggregating'): predictors are built on bootstrap replicates of the learning set and combined by averaging (numerical outcome) or plurality vote (class). Tests with classification and regression trees and subset selection in linear regression show accuracy gains, and the paper identifies instability of the prediction method as the condition under which bagging improves accuracy.
A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
Yoav Freund, Robert E. Schapire · First public 1995-03 (conference) · Venue: Journal of Computer and System Sciences 55(1) · Pages 35 pp. (author manuscript); journal 119–139 (not seen) · Citations (OpenAlex) 20,470 (checked 2026-11-03)
DOI · Journal of Computer and System Sciences 55(1):119-139, 1997-08 (Crossref 10.1006/jcss.1997.1504 published-print); manuscript dated December 19, 1996
In the second part of the paper we apply the multiplicative weight-update technique to derive a new boosting algorithm.p. 1 (author manuscript)
What changed. Adapts the multiplicative weight-update rule of Littlestone and Warmuth to a general decision-theoretic on-line allocation setting (the Hedge algorithm). Applies the technique to derive a boosting algorithm (AdaBoost) that needs no prior knowledge of the weak learner's performance, with extensions to multiclass and real-valued outputs.
Gerald Tesauro · First public 1995-03 (journal issue) · Venue: Communications of the ACM, 38(3) · Pages 58–68 (per Crossref; the transcription has no page numbers) · Citations (OpenAlex) 1,599 (checked 2026-11-03)
This article presents a game-learning program called TD-Gammon.page unknown
What changed. Presented TD-Gammon, a neural network that trains itself as a backgammon evaluation function by playing against itself and learning from the outcome with the TD(lambda) algorithm. The article compares this approach with Neurogammon, the author's earlier network trained by backpropagation on expert games, and discusses why TD learning worked in this domain.
Sepp Hochreiter, Jürgen Schmidhuber · First public 1995 (tech report) · Venue: Neural Computation 9(8) · Pages 1735–1780 · Citations (OpenAlex) 102,398 (checked 2026-11-03)
DOI · Neural Computation 9(8), 1 November 1997 (journal issue); TU Munich technical report FKI-207-95 (1995), known from the authors' publication list
This article presents long short-term memory (LSTM), a novel recurrent network architecture in conjunction with an appropriate gradient-based learning algorithm.1736
What changed. The article introduces LSTM, a recurrent architecture with a gradient-based learning algorithm that enforces constant error flow through constant error carousels within special units whose access is controlled by multiplicative gates, so it can bridge time lags in excess of 1000 steps; experiments compare it with RTRL, BPTT and other recurrent methods.
Stuart Russell, Peter Norvig · First public 1995 (book) · Venue: Prentice Hall (1st ed.); 4th ed. Pearson, Hoboken · Pages 4th-ed. front matter only (preface pp. vii–x) · Citations (OpenAlex) 1,733 (checked 2026-11-03)
URL · 3rd ed. 2010 ('the last edition in 2010', 4th-ed. preface p. vii); 4th ed. Pearson [2021], LCCN 2019047498 (4th-ed. copyright page); Open Library lists a 1994 record among the work's editions (not verified)
The main unifying theme is the idea of an intelligent agent.vii (4th edition, 2021)
What changed. The textbook organises AI around the idea of an intelligent agent that receives percepts from the environment and performs actions, covering different ways to represent the agent function (taken from the 4th-edition preface, not from the 1995 text).
Corinna Cortes, Vladimir N. Vapnik · First public 1995-09 (journal issue) · Venue: Machine Learning 20(3) · Pages 273–297 · Citations (OpenAlex) 59,474 (checked 2026-11-03)
DOI · Received 15 May 1993; accepted 20 Feb 1995; final manuscript 8 Mar 1995 (not public dates)
The support-vector network is a new learning machine for two-group classification problems.273
What changed. The paper describes the support-vector network, which maps input vectors non-linearly into a very high-dimensional feature space and constructs a linear decision surface there, and extends the earlier method for separable training data to non-separable data; it reports results with polynomial input transformations and compares with classical algorithms on an OCR benchmark.
Robert John Tibshirani · First public 1996 (journal issue) · Venue: Journal of the Royal Statistical Society, Series B (Methodological) 58(1) · Pages 267–288 · Citations (OpenAlex) 51,457 (checked 2026-11-03)
DOI · Received January 1994, revised January 1995 (not public dates); Wiley online 2018-12-05 (digitisation; Crossref published-online)
We propose a new method for estimation in linear models.267
What changed. The paper proposes the lasso, which minimizes the residual sum of squares subject to the sum of the absolute values of the coefficients being less than a constant, so that some coefficients become exactly 0, and reports simulation studies comparing it with subset selection and ridge regression.
Mike Schuster, Kuldip K. Paliwal · First public 1997-11 (journal issue) · Venue: IEEE Transactions on Signal Processing 45(11) · Pages 2673–2681 · Citations (OpenAlex) 10,168 (checked 2026-11-03)
DOI · Manuscript received June 5, 1997 (not a public date)
In the first part of this paper, a regular recurrent neural network (RNN) is extended to a bidirectional recurrent neural network (BRNN).2673
What changed. The paper extends a regular RNN to a bidirectional RNN trained simultaneously in positive and negative time directions, so it is not limited to input information up to a preset future frame, reports experiments on artificial data and TIMIT phonemes, and modifies the structure to estimate the conditional posterior probability of complete symbol sequences.
Planning and acting in partially observable stochastic domains
Leslie Pack Kaelbling, Michael L. Littman, Anthony R. Cassandra · First public 1998-05 (journal issue) · Venue: Artificial Intelligence 101(1-2) · Pages 99–134 · Citations (OpenAlex) 4,147 (checked 2026-11-03)
DOI · Received 11 Oct 1995, revised 17 Jan 1998 (not public dates); related earlier conference paper Cassandra, Kaelbling & Littman, AAAI-94, pp. 1023–1028 (ref [8])
We then outline a novel algorithm for solving POMDPs off line and show how, in some cases, a finite-memory controller can be extracted from the solution to a POMDP.99
What changed. The paper brings operations-research techniques for Markov decision processes and partially observable MDPs to the problem of choosing actions in partially observable stochastic domains, outlines an algorithm for solving POMDPs off line, and shows how a finite-memory controller can in some cases be extracted from the solution.
Richard S. Sutton, Andy Barto · First public 1998 (book) · Venue: MIT Press (A Bradford Book), Cambridge, MA · Pages page numbers not shown in HTML edition · Citations (OpenAlex) 27,232 (checked 2026-11-03)
DOI · 2nd edition 2018 (listed on the same author page as the-book-2nd.html)
Our goal in writing this book was to provide a clear and simple account of the key ideas and algorithms of reinforcement learning.Preface (HTML edition, page unknown)
What changed. The book gives an account of the key ideas and algorithms of reinforcement learning, mainly from an artificial-intelligence and engineering point of view. It places temporal-difference learning, dynamic programming and function approximation within one framework for learning from interaction to achieve goals.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner · First public 1998-11 (journal issue) · Venue: Proceedings of the IEEE, 86(11) · Pages 2278–2324 (journal); HAL copy pp. 1–47 · Citations (OpenAlex) 58,786 (checked 2026-11-03)
Using character recognition as a case study, we show that hand-crafted feature extraction can be advantageously replaced by carefully designed learning machines that operate directly on pixel images.p. 1 (HAL re-typeset copy; journal p. 2278 not seen)
What changed. The paper reviews methods for handwritten character recognition and reports that convolutional neural networks outperform the other techniques compared on a standard digit task. It also introduces graph transformer networks, which train multimodule document recognition systems globally with gradient-based methods, and describes a deployed bank-check reader.
Policy Gradient Methods for Reinforcement Learning with Function Approximation
Richard S. Sutton, David McAllester, Satinder Pal Singh, Yishay Mansour · First public 1999 (conference) · Venue: Advances in Neural Information Processing Systems 12 (NIPS 1999) · Pages 1057–1063 · Citations (OpenAlex) 5,142 (checked 2026-11-03)
URL · proceedings volume published by MIT Press (2000; not printed on the copy read)
Our main new result is to show that the gradient can be written in a form suitable for estimation from experience aided by an approximate action-value or advantage function.p. 1057 (first page unnumbered; next page headed 1058)
What changed. The paper represents the policy with its own function approximator, updated along the gradient of expected reward. It shows that this gradient can be estimated using an approximate action-value or advantage function, and proves convergence to a locally optimal policy for a version of policy iteration with differentiable function approximation.
Training Products of Experts by Minimizing Contrastive Divergence
Geoffrey E. Hinton · First public 2000 (tech report) · Venue: Neural Computation, 14(8) · Pages 1771–1800 · Citations (OpenAlex) 4,878 (checked 2026-11-03)
DOI · received 2000-07-28, accepted 2001-12-10 (journal footer); Neural Computation 14(8) 2002-08-01 (Crossref)
Fortunately, a PoE can be trained using a different objective function called “contrastive divergence” whose derivatives with regard to the parameters can be approximated accurately and efficiently.p. 1771
What changed. The paper proposes combining latent-variable models by multiplying their distributions and renormalizing (a product of experts). It shows that such models can be trained by minimizing an objective called contrastive divergence instead of maximizing likelihood, with examples using several types of expert and data.
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
John Lafferty, Andrew McCallum, Fernando C. N. Pereira · First public 2001-06-28 (conference) · Venue: Proceedings of the 18th International Conference on Machine Learning (ICML 2001), Morgan Kaufmann/ACM · Pages 282–289 (proceedings); postprint 8 pp. unnumbered · Citations (OpenAlex) 12,947 (checked 2026-11-03)
We present conditional random fields, a framework for building probabilistic models to segment and label sequence data.p. 1 of postprint (unnumbered; proceedings p. 282 by position)
What changed. The paper introduces conditional random fields for labeling and segmenting sequences, with iterative parameter estimation algorithms. It reports that they avoid the label bias problem of maximum entropy Markov models and compares them with HMMs and MEMMs on synthetic data and part-of-speech tagging.
Random forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest.p. 5
What changed. The paper defines random forests as ensembles of trees grown with independent, identically distributed random vectors that vote for the most popular class. It shows that their generalization error converges as trees are added, relates it to the strength and correlation of the trees, and introduces out-of-bag estimates and variable importance measures. It also covers regression.
Murray S. Campbell, A. Joseph Hoane, Feng-hsiung Hsu · First public 2002-01 (journal issue) · Venue: Artificial Intelligence, 134(1–2) · Pages 57–83 (journal); manuscript pp. 1–31 · Citations (OpenAlex) 1,211 (checked 2026-11-03)
DOI · manuscript dated 2001-08-01 (title page of copy read)
This paper describes the Deep Blue system, and gives some of the rationale that went into the design decisions behind Deep Blue.p. 1 (manuscript)
What changed. The paper describes the chess machine that defeated Garry Kasparov in a six-game match in 1997, including its single-chip search engine, multi-level parallel system, search extensions, complex evaluation function and grandmaster game database. It also covers the predecessor machines and the reasons for its design decisions.
David M. Blei, Andrew Y. Ng, Michael I. Jordan · First public 2003-01 (journal issue) · Venue: Journal of Machine Learning Research, 3 · Pages 993–1022 · Citations (OpenAlex) 32,332 (checked 2026-11-03)
We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora.p. 993
What changed. The paper introduces latent Dirichlet allocation, a three-level hierarchical Bayesian model in which each document is a mixture over topics. It gives variational inference and an EM procedure for parameter estimation, and reports results on document modeling, text classification and collaborative filtering.
Cites (within this list): none Cited by (within this list): Radford et al. 2021
A Neural Probabilistic Language Model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin, Jaz K, Thomas Hofmann et al. · First public 2003-02 (journal issue) · Venue: Journal of Machine Learning Research, 3 · Pages 1137–1155 · Citations (OpenAlex) 3,451 (checked 2026-11-03)
We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences.p. 1137
What changed. The paper proposes a neural network language model that learns a distributed representation for each word together with the probability function for word sequences. On two text corpora it reports improvements over n-gram models and shows that the model can use longer contexts.
Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search
Rémi Coulom · First public 2006-05-29 (conference) · Venue: Computers and Games (CG 2006), Turin; Lecture Notes in Computer Science (Springer, 2007) · Pages 72–83 (LNCS); HAL copy pp. 1–12 · Citations (OpenAlex) 1,286 (checked 2026-11-03)
DOI · HAL deposit 2006-11-29 (HAL cover); LNCS proceedings 'Computers and Games' 2007, pp. 72-83 (Crossref published-print 2007)
This paper presents a new framework to combine tree search with Monte-Carlo evaluation, that does not separate between a min-max phase and a Monte-Carlo phase.p. 1 (HAL copy)
What changed. The paper combines tree search with Monte-Carlo evaluation using a backup operator that moves from averaging to min-max as the number of simulations grows, which allows selectivity at the level of individual simulations. It reports the method in the 9×9 Go program Crazy Stone, which won the 10th KGS computer-Go tournament.
Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.p. 1 (author preprint, unnumbered)
What changed. The paper uses complementary priors to derive a greedy algorithm that learns deep directed belief networks one layer at a time, followed by fine-tuning with a contrastive version of the wake-sleep algorithm. The resulting three-hidden-layer generative model of handwritten digits and labels classifies MNIST digits better than the discriminative methods it was compared with.
We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.p. 504
What changed. The paper trains multilayer autoencoder networks with a small central layer to produce low-dimensional codes. Their weights are first pretrained layer by layer with restricted Boltzmann machines and then fine-tuned by backpropagation, and the codes are reported to work better than PCA on several data sets.
In this paper we introduce a new algorithm, UCT, that applies bandit ideas to guide Monte-Carlo planning.PDF p. 1 (author preprint, no printed page number)
What changed. Introduces UCT, an algorithm that applies bandit ideas to guide Monte-Carlo planning; the algorithm is shown to be consistent in finite-horizon or discounted MDPs, with finite-sample bounds on the estimation error.
Stanley: The robot that won the DARPA Grand Challenge
Sebastian Thrun, Mike Montemerlo, Hendrik Dahlkamp, David Stavens, Andrei Aron, James R. Diebel et al. · First public 2006-09-25 (journal issue) · Venue: Journal of Field Robotics 23(9) · Pages 661–692 · Citations (OpenAlex) 2,239 (checked 2026-11-03)
DOI · Journal of Field Robotics 23(9), September 2006 (print)
This article describes the robot Stanley, which won the 2005 DARPA Grand Challenge.p. 662
What changed. Describes Stanley, the autonomous vehicle that completed the 2005 DARPA Grand Challenge desert course first, and its software system built on machine learning and probabilistic reasoning. Covers long-range terrain perception, real-time collision avoidance and vehicle control, and reports the race results.
Cites (within this list): none Cited by (within this list): none
Greedy Layer-Wise Training of Deep Networks
Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle · First public 2006-12 (conference) · Venue: Advances in Neural Information Processing Systems 19 (NIPS 2006), MIT Press · Pages 153–160 (per Crossref; PDF unpaginated, 8 pp.) · Citations (OpenAlex) 4,841 (checked 2026-11-03)
DOI · MIT Press proceedings (NIPS 19) published-print 2007-09-07 (Crossref 10.7551/mitpress/7503.003.0024)
Our experiments also confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.PDF p. 1 (no printed page number; proceedings p. 153)
What changed. Studies empirically the greedy layer-wise unsupervised training algorithm of Hinton et al. for Deep Belief Networks and extends it to continuous inputs and to cases where the input distribution is not revealing about the target.
Jonathan Schaeffer, NEIL R. BURCH, Yngvi Björnsson, Akihiro Kishimoto, Martin Müller, Robert Lake et al. · First public 2007-07-19 (journal issue) · Venue: Science 317(5844) · Pages 1518–1522 · Citations (OpenAlex) 452 (checked 2026-11-03)
DOI · Science 317(5844):1518-1522, 14 September 2007 (print issue; Crossref published-print 2007-09-14)
This paper announces that checkers is now solved: Perfect play by both sides leads to a draw.p. 1518
What changed. Announces that checkers has been weakly solved: a computational proof shows that from the starting position perfect play by both sides leads to a draw. Describes the combination of endgame databases and proof-tree search used, run almost continuously since 1989.
Cites (within this list): Shannon 1949 Cited by (within this list): none
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei · First public 2009-06 (conference) · Venue: 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE · Pages 248–255 (per Crossref); project-site PDF 8 pp. · Citations (OpenAlex) 65,343 (checked 2026-11-03)
DOI · PDF creation date 2009-04-01 (metadata only, not a public date); IEEE Xplore DOI created 2010-03-01
We introduce here a new database called “ImageNet”, a large-scale ontology of images built upon the backbone of the WordNet structure.PDF p. 1 (no printed page number; proceedings p. 248)
What changed. Introduces ImageNet, an image database organized by the WordNet hierarchy, reporting 12 subtrees with 5247 synsets and 3.2 million images, collected and cleaned with Amazon Mechanical Turk.
Large-scale deep unsupervised learning using graphics processors
Rajat Raina, Anand Madhavan, Andrew Y. Ng · First public 2009-06-14 (conference) · Venue: Proceedings of the 26th Annual International Conference on Machine Learning (ICML 2009), Montreal, ACM · Pages 873–880 (ACM pagination per Crossref; the proceedings PDF itself carries no printed folios, 8 pp.) · Citations (OpenAlex) 742 (checked 2026-11-03)
We argue that modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods.PDF p. 1 (abstract; no printed folio, = p. 873 in ACM pagination)
What changed. Develops general principles for massively parallelizing unsupervised learning on graphics processors and applies them to deep belief networks and sparse coding. Reports DBN learning up to 70 times faster than a dual-core CPU implementation and a 5- to 15-fold speedup for sparse coding with a new parallel algorithm.
Deep, Big, Simple Neural Nets for Handwritten Digit Recognition
Dan Claudiu Ciresan, Ueli Meier, Luca Maria Gambardella, Jürgen Schmidhuber · First public 2010-03-01 (arXiv v1) · Venue: arXiv:1003.0358 [cs.NE]; published in Neural Computation, 22(12), 3207–3220 · Pages 14 pp. (arXiv v1) · Citations (OpenAlex) 1,094 (checked 2026-11-03)
DOI · Neural Computation 22(12), December 2010 (arXiv journal-ref; Crossref published-print 2010-12)
Good old on-line back-propagation for plain multi-layer perceptrons yields a very low 0.35% error rate on the famous MNIST handwritten digits benchmark.p. 1 (arXiv v1)
What changed. Trained large, deep plain multilayer perceptrons (2 to 9 hidden layers, up to 12.11 million weights) with on-line back-propagation on graphics cards, using continually deformed training images, and reported a 0.35% test error on MNIST without unsupervised pre-training.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot, Yoshua Bengio · First public 2010-05 (conference) · Venue: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010), PMLR (JMLR W&CP) 9 · Pages 249–256 · Citations (OpenAlex) 14,126 (checked 2026-11-03)
URL · PMLR RIS record date 2010/03/31 (as reported for the landing page; not independently confirmed)
Based on these considerations, we propose a new initialization scheme that brings substantially faster convergence.p. 249
What changed. Analyses why standard gradient descent from random initialization trains deep feedforward networks poorly, by monitoring activations and gradients across layers, and proposes a new initialization scheme.
Rectified Linear Units Improve Restricted Boltzmann Machines
Vinod B Nair, Geoffrey E. Hinton · First public 2010-06 (conference) · Venue: Proceedings of the 27th International Conference on Machine Learning (ICML 2010), Haifa · Pages 807–814 (per MLAnthology/secondary citations; author copy unpaginated, 8 pp.) · Citations (OpenAlex) 13,987 (checked 2026-11-03)
URL · University of Toronto ML seminar talk, March 2010 (Nair CV); PDF creation 2010-05-08 (metadata only)
Compared with binary units, these units learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset.PDF p. 1 (author copy, no printed page number)
What changed. Generalizes binary RBM hidden units to Stepped Sigmoid Units approximated by noisy rectified linear units, and reports better features for NORB object recognition and LFW face verification than binary units.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jaň Černocký, Sanjeev P. Khudanpur · First public 2010-09-26 (conference) · Venue: Interspeech 2010 (11th Annual Conference of the International Speech Communication Association), ISCA · Pages 1045–1048 · Citations (OpenAlex) 5,386 (checked 2026-11-03)
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented.p. 1045
What changed. Presents a recurrent neural network language model and reports about 50% perplexity reduction with a mixture of RNN LMs versus a backoff model, and word error rate reductions on WSJ and NIST RT05.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel P. Kuksa · First public 2011-03-02 (arXiv v1) · Venue: Journal of Machine Learning Research 12(76) (journal version); arXiv:1103.0398 · Pages 47 pp. (arXiv v1) · Citations (OpenAlex) 6,313 (checked 2026-11-03)
We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling.p. 1 (arXiv v1)
What changed. Proposes a single neural network architecture and learning algorithm for POS tagging, chunking, NER and SRL that learns internal representations from large amounts of mostly unlabeled text instead of task-specific features, released as a tagging system.