Attention with a bias

We review why some transformer models use a bias in attention and how ALiBi helps with long context. The provided sources focus on significant advancements in computational biology, specifically the evolution of the AlphaFold series for predicting 3D biomolecular structures. AlphaFold 2 revolutionized the field by using the Evoformer and attention mechanisms to interpret evolutionary and geometric data with near-experimental accuracy. Building on this, AlphaFold 3 expanded capabilities to include complexes with ligands and nucleic acids using an atom-level diffusion module. To further refine these models, HelixFold-S1 introduces a contact-guided sampling strategy that prioritizes likely binding sites to improve structural diversity and accuracy. Additionally, technical papers describe architectural components like ALiBi for handling long sequences and Swin Transformer's shifted windows. Together, these texts illustrate a shift toward more efficient, targeted sampling and integrated deep learning frameworks for complex molecular modeling.