Sequence to Sequence Learning with Neural Networks
May 1, 2026
This episode explores the 2014 seq2seq paper by Sutskever and colleagues as a turning point in machine translation, asking what problem it actually solved and why it mattered at the time. It explains how deep LSTM encoder-decoder models reframed translation as end-to-end learning of \(p(\text{output}|\text{input})\), replacing hand-built phrase tables, alignment models, and decoding heuristics with a single learned system. The discussion highlights both the breakthrough and the limitation: compressing an entire source sentence into one fixed-length vector was elegant but created a severe bottleneck, which later made attention mechanisms so important. Listeners would find it interesting because the episode separates the paper’s real contribution from its mythology, situating it against phrase-based SMT, earlier encoder-decoder work, and the practical role of LSTMs and beam search in making the approach viable.
Sources:
1. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014
http://arxiv.org/abs/1409.3215
2. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014
https://scholar.google.com/scholar?q=Sequence+to+Sequence+Learning+with+Neural+Networks
3. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, Yoshua Bengio and collaborators, 2014
https://scholar.google.com/scholar?q=Learning+Phrase+Representations+using+RNN+Encoder-Decoder+for+Statistical+Machine+Translation
4. Neural Machine Translation by Jointly Learning to Align and Translate — Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014
https://scholar.google.com/scholar?q=Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate
5. A Survey of the Usages of Deep Learning for Natural Language Processing — Tom Young, Devamanyu Hazarika, Soujanya Poria, Erik Cambria, 2018
https://scholar.google.com/scholar?q=A+Survey+of+the+Usages+of+Deep+Learning+for+Natural+Language+Processing
6. Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber, 1997
https://scholar.google.com/scholar?q=Long+Short-Term+Memory
7. Learning to Forget: Continual Prediction with LSTM — Felix A. Gers, Jürgen Schmidhuber, Fred Cummins, 2000
https://scholar.google.com/scholar?q=Learning+to+Forget%3A+Continual+Prediction+with+LSTM
8. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio, 2014
https://scholar.google.com/scholar?q=Empirical+Evaluation+of+Gated+Recurrent+Neural+Networks+on+Sequence+Modeling
9. An Empirical Exploration of Recurrent Network Architectures — Razvan Pascanu, Tomas Mikolov, Yoshua Bengio, 2013
https://scholar.google.com/scholar?q=An+Empirical+Exploration+of+Recurrent+Network+Architectures
10. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation — Yonghui Wu and many Google Brain collaborators, 2016
https://scholar.google.com/scholar?q=Google%27s+Neural+Machine+Translation+System%3A+Bridging+the+Gap+between+Human+and+Machine+Translation
11. A Call for Clarity in Reporting BLEU Scores — Matt Post, 2018
https://scholar.google.com/scholar?q=A+Call+for+Clarity+in+Reporting+BLEU+Scores
12. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches — Kyunghyun Cho and collaborators, 2014
https://scholar.google.com/scholar?q=On+the+Properties+of+Neural+Machine+Translation%3A+Encoder-Decoder+Approaches
13. Beam Search Strategies for Neural Machine Translation — Markus Freitag, Yaser Al-Onaizan, 2017
https://scholar.google.com/scholar?q=Beam+Search+Strategies+for+Neural+Machine+Translation
14. A Systematic Comparison of Search Algorithms for Neural Machine Translation — Various later comparison studies, e.g. work by Felix Stahlberg and collaborators, 2018
https://scholar.google.com/scholar?q=A+Systematic+Comparison+of+Search+Algorithms+for+Neural+Machine+Translation
15. Skip-Thought Vectors — Ryan Kiros, Yukun Zhu, Russ R. Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, Sanja Fidler, 2015
https://scholar.google.com/scholar?q=Skip-Thought+Vectors
16. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data — Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, Antoine Bordes, 2017
https://scholar.google.com/scholar?q=Supervised+Learning+of+Universal+Sentence+Representations+from+Natural+Language+Inference+Data
17. Universal Sentence Encoder — Daniel Cer, Yinfei Yang, Sheng-yi Kong and collaborators, 2018
https://scholar.google.com/scholar?q=Universal+Sentence+Encoder
18. A Neural Network for Machine Translation, at Production Scale — Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom, 2014
https://scholar.google.com/scholar?q=A+Neural+Network+for+Machine+Translation%2C+at+Production+Scale
19. Sequence Transduction with Recurrent Neural Networks — Alex Graves, 2012
https://scholar.google.com/scholar?q=Sequence+Transduction+with+Recurrent+Neural+Networks
20. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks — Alex Graves, Santiago Fernandez, Faustino Gomez, Jürgen Schmidhuber, 2006
https://scholar.google.com/scholar?q=Connectionist+Temporal+Classification%3A+Labelling+Unsegmented+Sequence+Data+with+Recurrent+Neural+Networks
21. Statistical Machine Translation: The Mathematics of SMT — Philipp Koehn, 2010
https://scholar.google.com/scholar?q=Statistical+Machine+Translation%3A+The+Mathematics+of+SMT
22. Recurrent Neural Network Based Language Model — Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, Sanjeev Khudanpur, 2010
https://scholar.google.com/scholar?q=Recurrent+Neural+Network+Based+Language+Model
23. Effective Approaches to Attention-based Neural Machine Translation — Luong, Pham, Manning, 2015
https://scholar.google.com/scholar?q=Effective+Approaches+to+Attention-based+Neural+Machine+Translation
24. Learning Transductions and Alignments with RNN Seq2seq Models — approx. Britz, Goldie, Luong, Le, or related Google authors, 2017
https://scholar.google.com/scholar?q=Learning+Transductions+and+Alignments+with+RNN+Seq2seq+Models
25. Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow, Birch, 2016
https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units
26. On Using Very Large Target Vocabulary for Neural Machine Translation — Jean, Cho, Memisevic, Bengio, 2015
https://scholar.google.com/scholar?q=On+Using+Very+Large+Target+Vocabulary+for+Neural+Machine+Translation
27. Attention Is All You Need — Vaswani et al., 2017
https://scholar.google.com/scholar?q=Attention+Is+All+You+Need
28. AI Post Transformers: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp3
29. AI Post Transformers: Experimental Comparison of Agentic and Enhanced RAG — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-14-experimental-comparison-of-agentic-and-e-37d8bc.mp3
1. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014
http://arxiv.org/abs/1409.3215
2. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014
https://scholar.google.com/scholar?q=Sequence+to+Sequence+Learning+with+Neural+Networks
3. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, Yoshua Bengio and collaborators, 2014
https://scholar.google.com/scholar?q=Learning+Phrase+Representations+using+RNN+Encoder-Decoder+for+Statistical+Machine+Translation
4. Neural Machine Translation by Jointly Learning to Align and Translate — Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014
https://scholar.google.com/scholar?q=Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate
5. A Survey of the Usages of Deep Learning for Natural Language Processing — Tom Young, Devamanyu Hazarika, Soujanya Poria, Erik Cambria, 2018
https://scholar.google.com/scholar?q=A+Survey+of+the+Usages+of+Deep+Learning+for+Natural+Language+Processing
6. Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber, 1997
https://scholar.google.com/scholar?q=Long+Short-Term+Memory
7. Learning to Forget: Continual Prediction with LSTM — Felix A. Gers, Jürgen Schmidhuber, Fred Cummins, 2000
https://scholar.google.com/scholar?q=Learning+to+Forget%3A+Continual+Prediction+with+LSTM
8. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio, 2014
https://scholar.google.com/scholar?q=Empirical+Evaluation+of+Gated+Recurrent+Neural+Networks+on+Sequence+Modeling
9. An Empirical Exploration of Recurrent Network Architectures — Razvan Pascanu, Tomas Mikolov, Yoshua Bengio, 2013
https://scholar.google.com/scholar?q=An+Empirical+Exploration+of+Recurrent+Network+Architectures
10. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation — Yonghui Wu and many Google Brain collaborators, 2016
https://scholar.google.com/scholar?q=Google%27s+Neural+Machine+Translation+System%3A+Bridging+the+Gap+between+Human+and+Machine+Translation
11. A Call for Clarity in Reporting BLEU Scores — Matt Post, 2018
https://scholar.google.com/scholar?q=A+Call+for+Clarity+in+Reporting+BLEU+Scores
12. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches — Kyunghyun Cho and collaborators, 2014
https://scholar.google.com/scholar?q=On+the+Properties+of+Neural+Machine+Translation%3A+Encoder-Decoder+Approaches
13. Beam Search Strategies for Neural Machine Translation — Markus Freitag, Yaser Al-Onaizan, 2017
https://scholar.google.com/scholar?q=Beam+Search+Strategies+for+Neural+Machine+Translation
14. A Systematic Comparison of Search Algorithms for Neural Machine Translation — Various later comparison studies, e.g. work by Felix Stahlberg and collaborators, 2018
https://scholar.google.com/scholar?q=A+Systematic+Comparison+of+Search+Algorithms+for+Neural+Machine+Translation
15. Skip-Thought Vectors — Ryan Kiros, Yukun Zhu, Russ R. Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, Sanja Fidler, 2015
https://scholar.google.com/scholar?q=Skip-Thought+Vectors
16. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data — Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, Antoine Bordes, 2017
https://scholar.google.com/scholar?q=Supervised+Learning+of+Universal+Sentence+Representations+from+Natural+Language+Inference+Data
17. Universal Sentence Encoder — Daniel Cer, Yinfei Yang, Sheng-yi Kong and collaborators, 2018
https://scholar.google.com/scholar?q=Universal+Sentence+Encoder
18. A Neural Network for Machine Translation, at Production Scale — Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom, 2014
https://scholar.google.com/scholar?q=A+Neural+Network+for+Machine+Translation%2C+at+Production+Scale
19. Sequence Transduction with Recurrent Neural Networks — Alex Graves, 2012
https://scholar.google.com/scholar?q=Sequence+Transduction+with+Recurrent+Neural+Networks
20. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks — Alex Graves, Santiago Fernandez, Faustino Gomez, Jürgen Schmidhuber, 2006
https://scholar.google.com/scholar?q=Connectionist+Temporal+Classification%3A+Labelling+Unsegmented+Sequence+Data+with+Recurrent+Neural+Networks
21. Statistical Machine Translation: The Mathematics of SMT — Philipp Koehn, 2010
https://scholar.google.com/scholar?q=Statistical+Machine+Translation%3A+The+Mathematics+of+SMT
22. Recurrent Neural Network Based Language Model — Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, Sanjeev Khudanpur, 2010
https://scholar.google.com/scholar?q=Recurrent+Neural+Network+Based+Language+Model
23. Effective Approaches to Attention-based Neural Machine Translation — Luong, Pham, Manning, 2015
https://scholar.google.com/scholar?q=Effective+Approaches+to+Attention-based+Neural+Machine+Translation
24. Learning Transductions and Alignments with RNN Seq2seq Models — approx. Britz, Goldie, Luong, Le, or related Google authors, 2017
https://scholar.google.com/scholar?q=Learning+Transductions+and+Alignments+with+RNN+Seq2seq+Models
25. Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow, Birch, 2016
https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units
26. On Using Very Large Target Vocabulary for Neural Machine Translation — Jean, Cho, Memisevic, Bengio, 2015
https://scholar.google.com/scholar?q=On+Using+Very+Large+Target+Vocabulary+for+Neural+Machine+Translation
27. Attention Is All You Need — Vaswani et al., 2017
https://scholar.google.com/scholar?q=Attention+Is+All+You+Need
28. AI Post Transformers: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp3
29. AI Post Transformers: Experimental Comparison of Agentic and Enhanced RAG — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-14-experimental-comparison-of-agentic-and-e-37d8bc.mp3