Neural network learning
From formal neurons to perceptrons, backpropagation, convolutional and recurrent networks, normalization and residual connections.
| Year | First author | Title | Era |
|---|---|---|---|
| 1943 | McCulloch & Pitts | A logical calculus of the ideas immanent in nervous activity | Ch. 1 Origins 1943–56 |
| 1949 | Hebb | The Organization of Behavior | Ch. 1 Origins 1943–56 |
| 1958 | Rosenblatt | The perceptron: A probabilistic model for information storage and organization in the brain | Ch. 2 Symbols and perceptrons 1956–74 |
| 1960 | Widrow & Hoff | Adaptive switching circuits | Ch. 2 Symbols and perceptrons 1956–74 |
| 1969 | Minsky & Papert | Perceptrons: An Introduction to Computational Geometry | Ch. 2 Symbols and perceptrons 1956–74 |
| 1974 | Werbos | Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences | Ch. 2 Symbols and perceptrons 1956–74 |
| 1982 | Hopfield | Neural networks and physical systems with emergent collective computational abilities | Ch. 3 Knowledge systems and connectionism 1974–93 |
| 1982 | Werbos | Applications of advances in nonlinear sensitivity analysis | Ch. 3 Knowledge systems and connectionism 1974–93 |
| 1984 | Valiant | A theory of the learnable | Ch. 3 Knowledge systems and connectionism 1974–93 |
| 1985 | Ackley et al. | A learning algorithm for Boltzmann machines | Ch. 3 Knowledge systems and connectionism 1974–93 |
| 1986 | Rumelhart et al. | Learning representations by back-propagating errors | Ch. 3 Knowledge systems and connectionism 1974–93 |
| 1988 | Elman | Finding Structure in Time | Ch. 3 Knowledge systems and connectionism 1974–93 |
| 1988 | Hornik et al. | Multilayer feedforward networks are universal approximators | Ch. 3 Knowledge systems and connectionism 1974–93 |
| 1989 | Cybenko | Approximation by superpositions of a sigmoidal function | Ch. 3 Knowledge systems and connectionism 1974–93 |
| 1991 | Jacobs et al. | Adaptive Mixtures of Local Experts | Ch. 4 Statistical learning 1990–2011 |
| 1991 | Hochreiter | Untersuchungen zu dynamischen neuronalen Netzen | Ch. 4 Statistical learning 1990–2011 |
| 1994 | Bengio et al. | Learning long-term dependencies with gradient descent is difficult | Ch. 4 Statistical learning 1990–2011 |
| 1995 | Hochreiter & Schmidhuber | Long Short-Term Memory | Ch. 4 Statistical learning 1990–2011 |
| 1997 | Schuster & Paliwal | Bidirectional recurrent neural networks | Ch. 4 Statistical learning 1990–2011 |
| 2000 | Hinton | Training Products of Experts by Minimizing Contrastive Divergence | Ch. 4 Statistical learning 1990–2011 |
| 2006 | Hinton et al. | A Fast Learning Algorithm for Deep Belief Nets | Ch. 4 Statistical learning 1990–2011 |
| 2006 | Hinton & Salakhutdinov | Reducing the Dimensionality of Data with Neural Networks | Ch. 4 Statistical learning 1990–2011 |
| 2006 | Bengio et al. | Greedy Layer-Wise Training of Deep Networks | Ch. 4 Statistical learning 1990–2011 |
| 2009 | Raina et al. | Large-scale deep unsupervised learning using graphics processors | Ch. 4 Statistical learning 1990–2011 |
| 2010 | Glorot & Bengio | Understanding the difficulty of training deep feedforward neural networks | Ch. 4 Statistical learning 1990–2011 |
| 2010 | Nair & Hinton | Rectified Linear Units Improve Restricted Boltzmann Machines | Ch. 4 Statistical learning 1990–2011 |
| 2011 | Le et al. | Building high-level features using large scale unsupervised learning | Ch. 5 Deep learning 2012–17 |
| 2014 | Srivastava et al. | Dropout: A Simple Way to Prevent Neural Networks from Overfitting | Ch. 5 Deep learning 2012–17 |
| 2014 | Graves et al. | Neural Turing Machines | Ch. 5 Deep learning 2012–17 |
| 2014 | Weston et al. | Memory Networks | Ch. 5 Deep learning 2012–17 |
| 2014 | Kingma & Ba | Adam: A Method for Stochastic Optimization | Ch. 5 Deep learning 2012–17 |
| 2015 | Ioffe & Szegedy | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift | Ch. 5 Deep learning 2012–17 |
| 2015 | Hinton et al. | Distilling the Knowledge in a Neural Network | Ch. 5 Deep learning 2012–17 |
| 2016 | Ba et al. | Layer Normalization | Ch. 5 Deep learning 2012–17 |
| 2017 | Shazeer et al. | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer | Ch. 5 Deep learning 2012–17 |
| 2017 | Jouppi et al. | In-Datacenter Performance Analysis of a Tensor Processing Unit | Ch. 5 Deep learning 2012–17 |