Query-Key Normalization Improves Low-Resource Transformer Translation

Henry, Dachapally, Pawar, Chen (Cyndx Technologies) · 2020 · small Transformers, ~31–48M params, 5 low-resource pairs
arXiv:2010.04245

The change in one picture

Dot-product attention vs QKNorm: normalize Q and K, then multiply by a learnable g.

Headline numbers (reported)

+0.928avg BLEU, 5 pairs
~10k–214ksentence pairs
72–79L (97.5th pct length)

Illustrative log-scale position of corpus sizes; English-French is described as tens of millions of pairs.

Five language pairs

Softmax only sees differences

The paper's example: softmax(760, 752, 750) = softmax(12, 4, 2).

Cosine scores are bounded: why a scale is needed

Illustrative cosines 0.9, 0.3, 0.1, −0.2 pushed through softmax(g·cos). Drag g.

Initial g from sequence length

g₀ = log₂(L² − L). Curve: weight on a best match (cos 1) against L−1 entries at cos 0 (computed from the formula; the paper's rough claim is >99.9% near L≈75).

QKNorm step by step

Attention = softmax(g · Q̂ K̂ᵀ) V, with Q̂, K̂ L2-normalized along the head dimension. V is left alone.

Attention weights: dot product vs QKNorm

Illustrative hand-made 6-token vectors (dim 4), not from the paper. Hover cells.

Hover a cell.

Reported results

Reading the ablation

Dropping g costs about 8.7 BLEU: scores in [−1,1] without a temperature leave softmax nearly flat. It shows a scale is needed, not that cosine beats a learnable-temperature dot product.

Confound: two things changed at once

The 2×2 the paper does not run

Galician→English share

≈2.2 BLEU of the 4.64 total (5 × 0.928), ~47%. Other four average ~0.6.

Sensitivity (Vietnamese)

Reported approximate drops: percentile 97.5→99 ≈ 0.6; heads 8→16 ≈ 0.8.

What is shown vs not shown

References

  1. Query-Key Normalization for Transformers — Henry et al., 2020 · arXiv:2010.04245
  2. Transformers without Tears — Nguyen, Salazar, 2019
  3. Scaling Vision Transformers to 22 Billion Parameters — Dehghani et al., 2023
  4. Small-scale proxies for large-scale Transformer training instabilities — Wortsman et al., 2023
  5. Revisiting Low-Resource NMT — Sennrich, Zhang, 2019
  6. Massively Multilingual NMT — Aharoni et al., 2019
  7. Pre-trained Word Embeddings for NMT — Qi et al., 2018
  8. No Language Left Behind — NLLB Team, 2022
  9. Swin Transformer V2 cosine attention — Liu et al., 2022
  10. On Layer Normalization in the Transformer Architecture — Xiong et al., 2020
  11. Normalized Attention Without Probability Cage — Richter, Wattenhofer, 2020
  12. Preventing Attention Entropy Collapse — Zhai et al., 2023
  13. RMSNorm — Zhang, Sennrich, 2019
  14. BPE-Dropout — Provilkov et al., 2020