Convolutional Sequence to Sequence Learning for Translation

This episode explores the 2017 ConvS2S paper from Facebook AI Research, which argued that sequence-to-sequence models for machine translation did not need recurrence and could instead use fully convolutional encoder-decoder networks with attention. It explains the seq2seq and neural machine translation setup, clarifies that the paper replaces recurrent computation rather than attention, and breaks down the architecture’s core ideas: stacked convolutions, expanding receptive fields, positional embeddings, residual connections, and gated linear units. The discussion highlights why the approach was provocative at the time: it challenged the LSTM-based consensus, reported stronger BLEU scores, and promised much faster, more parallelizable decoding on GPUs. Listeners would find it interesting as a key pre-transformer moment that shows how researchers were already rethinking sequential modeling, while also surfacing the tradeoff between efficiency and limited context windows for long-range dependencies.

Sources:
1. Convolutional Sequence to Sequence Learning — Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017
http://arxiv.org/abs/1705.03122
2. Convolutional Sequence to Sequence Learning — Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017
https://scholar.google.com/scholar?q=Convolutional+Sequence+to+Sequence+Learning
3. ByteNet: Generating High-Resolution Discrete Sequences with Multiscale Dilation — Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, Koray Kavukcuoglu, 2016
https://scholar.google.com/scholar?q=ByteNet%3A+Generating+High-Resolution+Discrete+Sequences+with+Multiscale+Dilation
4. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2017
https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks
5. WaveNet: A Generative Model for Raw Audio — Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, et al., 2016
https://scholar.google.com/scholar?q=WaveNet%3A+A+Generative+Model+for+Raw+Audio
6. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014
https://scholar.google.com/scholar?q=Sequence+to+Sequence+Learning+with+Neural+Networks
7. Neural Machine Translation by Jointly Learning to Align and Translate — Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014
https://scholar.google.com/scholar?q=Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate
8. Effective Approaches to Attention-based Neural Machine Translation — Minh-Thang Luong, Hieu Pham, Christopher D. Manning, 2015
https://scholar.google.com/scholar?q=Effective+Approaches+to+Attention-based+Neural+Machine+Translation
9. A Survey on Deep Learning for Neural Machine Translation — Antonio Toral, Víctor M. Sánchez-Cartagena, et al. (representative survey literature varies by edition), 2018
https://scholar.google.com/scholar?q=A+Survey+on+Deep+Learning+for+Neural+Machine+Translation
10. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation — Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, et al., 2016
https://scholar.google.com/scholar?q=Google%27s+Neural+Machine+Translation+System%3A+Bridging+the+Gap+between+Human+and+Machine+Translation
11. Neural Machine Translation of Rare Words with Subword Units — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016
https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units
12. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017
https://scholar.google.com/scholar?q=Attention+Is+All+You+Need
13. Neural Machine Translation and Sequence-to-sequence Models: A Tutorial — Graham Neubig, 2017
https://scholar.google.com/scholar?q=Neural+Machine+Translation+and+Sequence-to-sequence+Models%3A+A+Tutorial
14. Neural Machine Translation in Linear Time — Jonas Gehring, Michael Auli, David Grangier, Yann N. Dauphin, 2016
https://scholar.google.com/scholar?q=Neural+Machine+Translation+in+Linear+Time
15. ByteNet: Neural Machine Translation in Linear Time — Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, Koray Kavukcuoglu, 2016
https://scholar.google.com/scholar?q=ByteNet%3A+Neural+Machine+Translation+in+Linear+Time
16. Quasi-Recurrent Neural Networks — James Bradbury, Stephen Merity, Caiming Xiong, Richard Socher, 2016
https://scholar.google.com/scholar?q=Quasi-Recurrent+Neural+Networks
17. Convolutional Neural Network for Machine Translation — Fandong Meng, Zhengdong Lu, Hang Li, Qun Liu, 2015
https://scholar.google.com/scholar?q=Convolutional+Neural+Network+for+Machine+Translation
18. Resurrecting Recurrent Neural Networks for Long Sequences — approx. Orvieto et al., 2023
https://scholar.google.com/scholar?q=Resurrecting+Recurrent+Neural+Networks+for+Long+Sequences
19. A Comprehensive Survey on Long Context Language Modeling — approx. recent survey authors, exact authorship to verify, 2024
https://scholar.google.com/scholar?q=A+Comprehensive+Survey+on+Long+Context+Language+Modeling
20. Linear Recurrent Models for Robust and Interpretable Long-context Modeling — approx. recent long-context modeling authors, exact authorship to verify, 2024
https://scholar.google.com/scholar?q=Linear+Recurrent+Models+for+Robust+and+Interpretable+Long-context+Modeling
21. Cross-layer Attention Sharing for Pre-trained Large Language Models — approx. recent LLM systems authors, exact authorship to verify, 2024
https://scholar.google.com/scholar?q=Cross-layer+Attention+Sharing+for+Pre-trained+Large+Language+Models
22. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — approx. recent efficiency-focused LLM authors, exact authorship to verify, 2024
https://scholar.google.com/scholar?q=Reducing+Transformer+Key-Value+Cache+Size+with+Cross-Layer+Attention
23. AI Post Transformers: RoPE — Hal Turing & Dr. Ada Shannon, Thu,
https://podcast.do-not-panic.com/episodes/rope/
24. AI Post Transformers: Apple's Speculative Streaming: Fast LLM Inference without Auxiliary Models — Hal Turing & Dr. Ada Shannon, Sat,
https://podcast.do-not-panic.com/episodes/apples-speculative-streaming-fast-llm-inference-without-auxiliary-models/
25. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3
26. AI Post Transformers: FlatAttention for Tile-Based Accelerator Inference — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-04-flatattention-for-tile-based-accelerator-56e6ca.mp3