Alternating Updates: Decoupling Width for Efficient Transformers

This episode examines "Alternating Updates for Efficient Transformers" (Baykal et al., Google Research and UC Berkeley, NeurIPS 2023), a technique that reports up to an 87% inference speedup on SQuAD at matched accuracy without changing attention mechanisms or reducing parameter count. The discussion contrasts this approach, called AltUp, with Mixture-of-Experts methods like Switch Transformer and GShard, clarifying that while MoE scales total parameter count and activates only a subset per token, AltUp instead widens the token embedding itself while keeping each transformer layer's compute cost fixed. The hosts unpack why widening representations normally taxes every layer quadratically through attention, and how AltUp sidesteps this by running only one slice of the widened vector through full computation per layer while cheaply predicting and correcting the rest. A key thread running through the conversation questions whether the paper's framing of AltUp as "conditional computation" alongside MoE is fully justified, teeing up a deeper look at whether its slice-selection mechanism is truly input-dependent like MoE routing or something simpler. Listeners interested in inference efficiency, transformer architecture tradeoffs, or the shift from parameter-count to FLOP-based cost thinking will find the mechanics of the predict-compute-correct cycle a genuinely novel angle on model scaling.