ICML 2023 · PMLR v202 · Phang, Mao, He, Chen · NYU / EleutherAI / Microsoft Azure AI

HyperTuning: Adapting Language Models Without Back-propagation Using Hypernetworks

arXiv 2211.12485 PMLR PDF Interactive page

One forward pass instead of a training run

Toggle between using the hypermodel on a new task and training the hypermodel. The "without back-propagation" claim only covers the first.

Passes over the examples counted, not timed

HyperT5 encodes the 16 examples once. A few-shot baseline re-reads them with every query. The paper reports no latency or memory number.

queries per task: 10

What the hypermodel emits

Prefix: extra key/value rows per attention layer. LoRA: a low-rank product added to the query and value projections.Illustrative values

prefix length (virtual tokens): 3
Hover a cell to read its toy value.

Hypermodel anatomy

Encoder reads the examples; decoder gets fixed learned embeddings with causal masking removed; MLP heads emit parameters.

Stage 1 · CACLM hyperpretraining

Step through one training example. Only B's length (32 tokens) and the 512-token total are given; other segment lengths are drawn illustratively.

Two stages, to scale

Step counts from the paper. Bars drawn proportional to steps.

Reported results

Numbers as stated in the episode. The paper reports no seeds or confidence intervals on these tables.

Claim 1 · matches full fine-tuningRoughly tied on P3 (+0.5 at XL, −0.2 at Large). Clearly behind on Super-NaturalInstructions.
Claim 2 · beats shared PEFTAbout 9 points over a shared multi-task LoRA at XL. Baselines are shared sets, not per-task LoRA.
Claim 3 · good PEFT initializationSturdiest result: averaged over 3 seeds at Large, no spread reported. Still back-propagates.

Where did the cost go?

The hypermodel is about the size of the model it steers. Parameters in millions; backbone figures are derived as total minus heads.

Cost ledger

What the paper gives versus what a break-even argument needs.

Break-even schematic Illustrative · no paper data

Cumulative cost of per-task LoRA versus a one-time hypermodel bill. Move the sliders to see where lines cross.

one-time training bill: 8 hypermodel per-task cost ÷ LoRA per-task cost: 0.20

P3 XL per-task scoreboard Illustrative · no magnitudes

Eight held-out tasks: CB and HellaSwag carry the gain; three others go the other way.

References

  1. HyperTuning — Phang, Mao, He, Chen, 2023
  2. T0: Multitask Prompted Training Enables Zero-Shot Task Generalization — Sanh et al.
  3. FLAN: Finetuned Language Models Are Zero-Shot Learners — Wei et al.
  4. Super-NaturalInstructions — Wang et al.
  5. MetaICL: Learning to Learn In Context — Min et al.
  6. HyperNetworks — Ha, Dai, Le
  7. HINT: Hypernetwork Instruction Tuning — Ivison et al.
  8. Boosting NLG from Instructions with Meta-Learning — Deb, Zheng, Awadallah
  9. HyperFormer: Shared Hypernetworks for Multi-task Fine-tuning — Karimi Mahabadi et al.
  10. SPoT: Soft Prompt Transfer — Vu et al.
  11. Reducing Retraining by Recycling Parameter-Efficient Prompts — Lester et al.
  12. MAML — Finn, Abbeel, Levine
  13. Text-to-LoRA (Sakana AI) · Drag-and-Drop LLMs (Liang et al.)
  14. Transformers generalize differently from information stored in context vs in weights — Chan et al.