Full-Bandwidth Transformer: Feeding Hidden States Back Into the Stack
Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford · JHU / Princeton / Microsoft · 2026
arXiv:2608.08888
model 1B params
depth 24 layers
width D = 1,536
vocab 100,352 (tied)
context 8K
data Phi-4 mix
batch 300K tok (1.2M for 1T baseline)