Gated Convolutions Challenge RNNs in Language Modeling

This episode examines "Language Modeling with Gated Convolutional Networks" (Dauphin et al., Facebook AI Research, ICML 2017), which replaces recurrent architectures with stacked causal convolutions for language modeling. The discussion covers why LSTMs dominated the field due to vanishing-gradient mitigation through gating, and contrasts that with the paper's Gated Linear Unit — a sigmoid-gated linear projection that preserves an undiminished gradient path, unlike the tanh-based gating in DeepMind's PixelCNN. A central debate weighs the RNN's theoretically unbounded context against the convolutional model's finite but better-conditioned receptive field, testing whether structured, hierarchical context outperforms raw sequential memory. The hosts walk through the model's pipeline — embeddings, bottlenecked causal convolution blocks with residual connections, and an adaptive softmax — and preview results on the Google Billion Word benchmark and the long-range WikiText-103 test. Listeners interested in the shift toward parallelizable, non-recurrent sequence models will find the tension between theoretical and practical memory capacity especially compelling.