AI History in Papers
한국어

Language and speech

From statistical language models and speech recognition to embeddings, attention, the Transformer and pretrained language models.

Language and speech 1960 1970 1980 1990 2000 2010 2020
Solid lines are confirmed citations (OpenAlex or the citing paper's own reference list); dotted lines are influences stated in textbooks without a confirmed citation. Vertical position is the theme group.
YearFirst authorTitleEra
1956ChomskyThree models for the description of languageCh. 1 Origins 1943–56
1966WeizenbaumELIZA—a computer program for the study of natural language communication between man and machineCh. 2 Symbols and perceptrons 1956–74
1971WinogradUnderstanding natural languageCh. 2 Symbols and perceptrons 1956–74
1987Waibel et al.Phoneme recognition using time-delay neural networksCh. 3 Knowledge systems and connectionism 1974–93
1989RabinerA tutorial on hidden Markov models and selected applications in speech recognitionCh. 3 Knowledge systems and connectionism 1974–93
1990Brown et al.A statistical approach to machine translationCh. 4 Statistical learning 1990–2011
2001Lafferty et al.Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence DataCh. 4 Statistical learning 1990–2011
2003Bengio et al.A Neural Probabilistic Language ModelCh. 4 Statistical learning 1990–2011
2010Mikolov et al.Recurrent neural network based language modelCh. 4 Statistical learning 1990–2011
2011Collobert et al.Natural Language Processing (almost) from ScratchCh. 4 Statistical learning 1990–2011
2012Hinton et al.Deep Neural Networks for Acoustic Modeling in Speech RecognitionCh. 5 Deep learning 2012–17
2013Mikolov et al.Efficient Estimation of Word Representations in Vector SpaceCh. 5 Deep learning 2012–17
2013Mikolov et al.Distributed Representations of Words and Phrases and their CompositionalityCh. 5 Deep learning 2012–17
2014Cho et al.Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine TranslationCh. 5 Deep learning 2012–17
2014Bahdanau et al.Neural Machine Translation by Jointly Learning to Align and TranslateCh. 5 Deep learning 2012–17
2014Sutskever et al.Sequence to Sequence Learning with Neural NetworksCh. 5 Deep learning 2012–17
2014Pennington et al.GloVe: Global Vectors for Word RepresentationCh. 5 Deep learning 2012–17
2015Luong et al.Effective Approaches to Attention-based Neural Machine TranslationCh. 5 Deep learning 2012–17
2015Sennrich et al.Neural Machine Translation of Rare Words with Subword UnitsCh. 5 Deep learning 2012–17
2016Wu et al.Google's Neural Machine Translation System: Bridging the Gap between Human and Machine TranslationCh. 5 Deep learning 2012–17
2017Vaswani et al.Attention Is All You NeedCh. 5 Deep learning 2012–17
2018Howard & RuderUniversal Language Model Fine-tuning for Text ClassificationCh. 6 Pretraining and generation 2017–22
2018Peters et al.Deep contextualized word representationsCh. 6 Pretraining and generation 2017–22
2018Radford et al.Improving Language Understanding by Generative Pre-TrainingCh. 6 Pretraining and generation 2017–22
2018Devlin et al.BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingCh. 6 Pretraining and generation 2017–22
2019Radford et al.Language Models are Unsupervised Multitask LearnersCh. 6 Pretraining and generation 2017–22
2019Raffel et al.Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerCh. 6 Pretraining and generation 2017–22
2020Brown et al.Language Models are Few-Shot LearnersCh. 6 Pretraining and generation 2017–22
2020Lewis et al.Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksCh. 6 Pretraining and generation 2017–22
2021Chen et al.Evaluating Large Language Models Trained on CodeCh. 6 Pretraining and generation 2017–22
2022Wei et al.Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsCh. 6 Pretraining and generation 2017–22
2022Wang et al.Self-Consistency Improves Chain of Thought Reasoning in Language ModelsCh. 6 Pretraining and generation 2017–22
2022Kojima et al.Large Language Models are Zero-Shot ReasonersCh. 6 Pretraining and generation 2017–22
2023Touvron et al.LLaMA: Open and Efficient Foundation Language ModelsCh. 6 Pretraining and generation 2017–22
2023OpenAIGPT-4 Technical ReportCh. 6 Pretraining and generation 2017–22