1
00:00:01,000 --> 00:00:07,965
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,965 --> 00:00:46,139
[Hal Turing] Today we're looking at Model-Aware Tokenizer Transfer, by Mykola Haltiuk and Aleksander Smywiński-Pohl at AGH University of Krakow. The arXiv ID is 2510.21954, first posted in 2025, and we're reading the version two update from May 2026. The hook is a concrete one. In Gemma 3, Ukrainian text gets about 2.98 characters per token. Extend the vocabulary with a Ukrainian-focused tokenizer and that rises to 4.44, which the authors say is almost a 50% inference speedup.

3
00:00:46,139 --> 00:01:04,436
[Dr. Ada Shannon] And the cost is more than 80,000 new tokens whose embeddings the model has never seen. Swap the tokenizer carelessly and the model breaks. The paper asks whether you can fix that by using the model's own internal behavior, rather than similarity heuristics, and whether attention is really what makes it work.

4
00:01:04,436 --> 00:01:11,217
[Hal Turing] Start with why the model breaks. A tokenizer is just a text-to-integer mapping, so why is the model welded to it?

5
00:01:11,217 --> 00:01:33,183
[Dr. Ada Shannon] Because those integers index two matrices: the input embeddings and the output head. Every attention and feed-forward layer above learned to work on the vectors those specific IDs produce. Tokenizer transfer keeps that transformer body, discards the old vocabulary rows, and builds new ones. The hard part is making the new vectors land where the frozen body can read them.

6
00:01:33,183 --> 00:01:36,108
[Hal Turing] Why does the vector matter so much for attention specifically?

7
00:01:36,108 --> 00:01:59,514
[Dr. Ada Shannon] Attention is where tokens talk to each other. Each token's embedding becomes a query, a key, and a value. Change the tokenization and every position sees different neighbors, in different pieces. The authors assume that if the input vector is right, the feed-forward layers will handle it. That's their stated intuition, an assumption and not a tested fact.

8
00:01:59,514 --> 00:02:02,393
[Hal Turing] So how did people fill in those rows before this paper?

9
00:02:02,393 --> 00:02:44,886
[Dr. Ada Shannon] The oldest recipe is de Vries and Nissim, 2020, "As Good as New," recycling English GPT-2 for Dutch and Italian. They trained a new tokenizer, re-learned only the lexical embeddings, and kept the body. Then came heuristics that build each new vector as a similarity-weighted average of old ones. WECHSEL, from Minixhofer, Paischer and Rekabsaz in 2022, gets its similarities from bilingual word vectors and a dictionary. Transtokenizers uses a dictionary learned from parallel text. TokAlign uses co-occurrence alignment. FOCUS, by Dobler and de Melo in 2023, trains vectors on target-language text and averages over tokens the two vocabularies share. MATT starts from FOCUS.

10
00:02:44,886 --> 00:02:46,604
[Hal Turing] What do all of those have in common?

11
00:02:46,604 --> 00:02:55,103
[Dr. Ada Shannon] None of them look at the transformer's higher layers. They judge a new vector by whether it resembles old ones, not by whether the layers above treat it the same way.

12
00:02:55,103 --> 00:03:02,115
[Hal Turing] And the straightforward alternative is ordinary training, right? Freeze everything except the embeddings and run next-token prediction.

13
00:03:02,115 --> 00:03:29,375
[Dr. Ada Shannon] Yes, and it works but is expensive, since every step needs a full forward and backward pass through the whole model. The other model-aware option is Zero-Shot Tokenizer Transfer, from Minixhofer, Ponti and Vulić in 2024. It trains a hypernetwork with a language-modeling loss over many tokenizers, so it can predict embeddings for a new one. The MATT authors call that compute-heavy. They never run it, so remember that.

14
00:03:29,375 --> 00:03:30,769
[Hal Turing] Then the closest relative?

15
00:03:30,769 --> 00:03:45,444
[Dr. Ada Shannon] Token Distillation, by Dobler, Elliott and de Melo, ICLR 2026. It tunes each new token's embedding so the model's hidden state matches what the original produced for the equivalent multi-token sequence. That's the nearest idea to this paper's.

16
00:03:45,444 --> 00:03:48,462
[Hal Turing] Now the setup. How does the teacher and student framing work?

17
00:03:48,462 --> 00:04:07,131
[Dr. Ada Shannon] The teacher is the original model with the old tokenizer, frozen. The student is the same weights with the new tokenizer, where only the new-token embeddings can train. Tokens both vocabularies share are copied and frozen. The student isn't asked to reproduce the teacher's text. It's asked to reproduce an internal quantity.

18
00:04:07,131 --> 00:04:09,267
[Hal Turing] Which quantity? The title says attention.

19
00:04:09,267 --> 00:04:33,230
[Dr. Ada Shannon] The attention output. For a given token, it's the sum over earlier tokens of an attention weight times that token's value vector. Each term is one weighted value state. Take a query token near the end of a phrase: it pulls in a bit of the first word, more of the second, little of the third. That mix is what the head hands to the next stage, before the output projection.

20
00:04:33,230 --> 00:04:36,806
[Hal Turing] But if the two tokenizers cut the text differently, what do you compare?

21
00:04:36,806 --> 00:04:54,639
[Dr. Ada Shannon] That's the catch. One might produce "_for", "m", "ula" and the other "_form", "ula". There's no token-by-token match. The paper groups tokens into the smallest character spans that neither tokenizer ever splits, and compares at that level. Part of the algorithm comes later.

22
00:04:54,639 --> 00:04:57,704
[Hal Turing] So Attention Influence Modeling, or AIM, in one sentence?

23
00:04:57,704 --> 00:05:11,590
[Dr. Ada Shannon] Train the new embeddings so the new-tokenizer model's attention output matches the old model's. One more term: Gemma ties its input and output embeddings, so training the input side also moves the output side. Not every model does, and that will matter.

24
00:05:11,590 --> 00:05:12,890
[Hal Turing] So what's the plan from here?

25
00:05:12,890 --> 00:05:22,178
[Dr. Ada Shannon] Next we walk through how the method works and what the results show. After that, we test whether the attention story holds up, or whether something simpler is doing the work.

26
00:05:22,178 --> 00:05:26,404
[Hal Turing] Let's start with what actually runs in a training step. What does the loop look like?

27
00:05:26,404 --> 00:05:36,853
[Dr. Ada Shannon] A batch of text goes through both models, each with its own tokenizer. Both stop at a chosen layer n, and at that layer the paper pulls out the attention outputs before the output projection.

28
00:05:36,853 --> 00:05:38,850
[Hal Turing] And the segments tell you what to compare.

29
00:05:38,850 --> 00:06:09,268
[Dr. Ada Shannon] Right. Take "formula". The teacher sees "_for", "m", "ula". The student sees "_form", "ula". The smallest spans neither tokenizer crosses are "_for" plus "m" against "_form", then "ula" against "ula". So the boundary can fall mid-word, and a word-level split would throw that alignment away. For each segment, the query is the last token's query state, because that token's output is what predicts the next segment.

30
00:06:09,268 --> 00:06:11,218
[Hal Turing] Then the loss. What gets matched?

31
00:06:11,218 --> 00:06:38,479
[Dr. Ada Shannon] There are two versions. The full one matches, for every query segment, the weighted value sum from each earlier source segment. That's a lot of vector pairs per step, hundreds of millions in their four-documents-by-256-tokens batches. The simplified one only matches each query segment's total output. Either one can use MSE or cosine loss. Here's what the paper doesn't say: which variant produced the headline results. There's no ablation between them either.

32
00:06:38,479 --> 00:06:41,451
[Hal Turing] That's a fairly basic gap. Where do the gradients go?

33
00:06:41,451 --> 00:07:10,104
[Dr. Ada Shannon] Backward through the frozen layers below n, into the new embedding rows. Depth is the main knob. Cost and memory grow roughly linearly with n, and the gains climb, plateau, then dip slightly near the top. The authors use layer 12 of 34 for Gemma and 16 of 36 for Qwen. Training is FOCUS init, AdamW, constant 1e-4, about 240 million Ukrainian tokens, and by their own admission no real tuning.

34
00:07:10,104 --> 00:07:14,005
[Hal Turing] One more teacher forward pass per step, but only a third of the way up the network.

35
00:07:14,005 --> 00:07:27,798
[Dr. Ada Shannon] Yes, and that's the efficiency argument. For models that don't tie their embeddings, an NTP loss is added. On Qwen 3 8B, that combined version reached the FOCUS-plus-NTP quality in 1 hour 40 minutes instead of 4 hours 10.

36
00:07:27,798 --> 00:07:34,392
[Hal Turing] Okay, results. The headline is Gemma 12B into Ukrainian. Before the numbers, what are we measuring?

37
00:07:34,392 --> 00:08:05,739
[Dr. Ada Shannon] Two groups, higher is better in both. Belebele and Global MMLU are multiple choice in Ukrainian, scored as accuracy in percent. Belebele has four options, so chance is 25. Then generation: translation and summarization scored by BLEU, a word-overlap score where even the original model is low. It gets 14.36 writing Ukrainian from English. Both directions of translation matter, because into-Ukrainian uses the new rows as output and out-of-Ukrainian mostly uses them as input.

38
00:08:05,739 --> 00:08:07,365
[Hal Turing] So how close does MATT get?

39
00:08:07,365 --> 00:08:27,241
[Dr. Ada Shannon] On multiple choice, within about one point of the original average, after seven hours. On generation, set the original at 100. MATT keeps about 93. The best embedding-only language-model training, given the same time or longer, keeps about 69. FOCUS before any training keeps about 16.

40
00:08:27,241 --> 00:08:30,352
[Hal Turing] Ninety-three is suspiciously good. Which direction carries it?

41
00:08:30,352 --> 00:08:55,012
[Dr. Ada Shannon] That's the right question. Ukrainian into English is carrying it. MATT scores 27.89 there, slightly above the original's 26.32. Writing Ukrainian is the hard direction: 8.70 against 14.36, about 60%. That's still roughly double the NTP runs, which sit near 4. But a Ukrainian user mostly wants output in Ukrainian, so the average flatters the result.

42
00:08:55,012 --> 00:08:56,962
[Hal Turing] And why would the output side lag?

43
00:08:56,962 --> 00:09:20,136
[Dr. Ada Shannon] One plausible reason is that nothing supervises the output distribution directly. Tying means the input rows drag the output rows along, but only indirectly. That's a hypothesis, and the paper doesn't test it. Token Distillation is a useful contrast. It lands at 0.00 writing Ukrainian while reading Ukrainian is reasonable, so internal-signal methods aren't interchangeable.

44
00:09:20,136 --> 00:09:22,690
[Hal Turing] So does it hold on smaller models and other scripts?

45
00:09:22,690 --> 00:09:41,080
[Dr. Ada Shannon] Mostly, but weaker. They ran Arabic, German, Japanese and Swahili on Gemma 4B and Qwen 4B. On Gemma, MATT answers about 23 more Belebele questions out of a hundred than FOCUS, and about 9 more than Token Distillation. It's still about four and a half below the untouched model.

46
00:09:41,080 --> 00:09:44,099
[Hal Turing] So understanding is mostly restored. What about generation?

47
00:09:44,099 --> 00:10:12,520
[Dr. Ada Shannon] That's where it's different. MATT recovers roughly 57% on Gemma and 67% on Qwen, not 93. It wins the discriminative average in seven of eight model-language pairs, and the generative average in six. Token Distillation is ahead on Qwen Japanese for the discriminative average, and on Gemma Japanese and Qwen German for generation. The Japanese generation scores are near zero, so those differences are mostly noise. Nothing here has variance reported.

48
00:10:12,520 --> 00:10:13,821
[Hal Turing] Where does it fail outright?

49
00:10:13,821 --> 00:10:34,858
[Dr. Ada Shannon] Qwen on Swahili. The original model scores 44.89 on Belebele, barely above chance. Token Distillation drops below FOCUS, 25.89 against 33.78. MATT improves to 41.67, but it can't beat its teacher. It's self-distillation, so it recovers what the model already knows.

50
00:10:34,858 --> 00:10:37,226
[Hal Turing] And the cost claim, a few GPU hours?

51
00:10:37,226 --> 00:11:01,793
[Dr. Ada Shannon] The convergence curve supports that. Gemma 4B on Arabic goes from 32.89 on Belebele at zero steps to 48.00 after 24 million tokens, then 62.44 at 250 million. The paper says over half the gain arrives in the first tenth of training. They also say it saturates, and that closing the generative gap would need continual pretraining.

52
00:11:01,793 --> 00:11:11,174
[Hal Turing] Before we wrap, there's an ablation I can't let go of. Extra supervision from lower layers should help, yet the paper's own appendix says it hurts. Why?

53
00:11:11,174 --> 00:11:38,852
[Dr. Ada Shannon] The paper doesn't say. Here is what the table shows. On a 4B Gemma, applying the loss at every layer up to depth five gives about 35 on Belebele. Applying it only at the fifth layer gives about 60. That's the difference between barely moving and recovering most of the model, and the all-layers version also costs about two and a half times the memory. At depth three, the two variants are close, about 33 against 37.

54
00:11:38,852 --> 00:11:41,035
[Hal Turing] So what are the candidate explanations?

55
00:11:41,035 --> 00:11:58,775
[Dr. Ada Shannon] Three, and each has a test. One is a scale conflict. If the per-layer mean squared errors are summed without normalization, layers with larger activations dominate the gradient. The paper doesn't say how the losses were combined. Weighting each layer, or adding layers one at a time, would settle that.

56
00:11:58,775 --> 00:12:30,540
[Dr. Ada Shannon] Two is an ill-posed early target. In the first few layers, token states are still tied to the tokenization. The inner-lexicon paper by Guy Kaplan and colleagues, 2024, argues that detokenization into word-level representations happens in early and middle layers. A student with different tokens may simply be unable to reproduce a sum of teacher token states there. That leaves an irreducible error whose gradients swamp everything else. It would also fit the jump between layers three and five.

57
00:12:30,540 --> 00:12:31,422
[Hal Turing] And the third?

58
00:12:31,422 --> 00:12:51,159
[Dr. Ada Shannon] A bug in how the per-layer loops interact. I can't exclude it, because each cell is a single run. There's also no run without AIM in this setting, and the layer-three, last-layer-only result is nearly identical to the FOCUS row. So we don't know whether layer three did anything. And nobody tested all-layers at depth twelve, which is what the main method uses.

59
00:12:51,159 --> 00:12:56,128
[Hal Turing] So in the main method, the objective is effectively one mid-depth layer.

60
00:12:56,128 --> 00:13:17,398
[Dr. Ada Shannon] On the paper's evidence, yes. The target is the head output before the output projection, built from the residual stream after the frozen layers below, so gradients still flow through all of them. But the contribution wording about aligning attention dynamics overstates it. The attention weights are never matched directly, only weights times values.

61
00:13:17,398 --> 00:13:21,949
[Hal Turing] Then how is that different from Token Distillation, which matches a hidden state mid-network?

62
00:13:21,949 --> 00:13:59,565
[Dr. Ada Shannon] Both match a mid-network quantity with the teacher frozen. The paper itself reports the same rise, plateau and slight fall across layers. AIM supervises every position in a context and, in the full version, breaks the signal into source-segment contributions. Token Distillation matches one state per new token on mined examples. Whether that matters is untested. The experiment is the same segmented loss on the residual stream at the same layer, plus the full loss versus the simplified one. Until then, I'd say mid-network matching works, and attention specifically is a hypothesis.

63
00:13:59,565 --> 00:14:05,649
[Hal Turing] Token Distillation scored zero BLEU writing Ukrainian. That looks like a broken setup, not a weak method.

64
00:14:05,649 --> 00:14:36,949
[Dr. Ada Shannon] It does, particularly since the same tied-embedding Gemma lets MATT write Ukrainian. What's known: wall-clock matched, the official package, examples sized to MATT's budget. Not reported: its layer, learning rate, any search, and whether it got a language-modeling loss. The authors admit they barely tuned their own method, so tuning could cut either way. MATT beating it on average is plausible. The size of the gap is uncertain, and no run has seeds or intervals.

65
00:14:36,949 --> 00:14:38,574
[Hal Turing] What other controls are missing?

66
00:14:38,574 --> 00:15:04,859
[Dr. Ada Shannon] ZeTT, the hypernetwork approach, is cited as compute-heavy but never run. MATT also starts from FOCUS, so it complements the heuristics rather than replacing them. The initialization ablation shows WECHSEL, FOCUS and Transtokenizers landing within about a point of each other after AIM at depth five. So the starting point washes out. But there's no random-initialization control, so we can't say how much AIM alone repairs.

67
00:15:04,859 --> 00:15:06,671
[Hal Turing] What about the boundaries of the claim?

68
00:15:06,671 --> 00:15:42,244
[Dr. Ada Shannon] It's self-distillation, so the teacher is the ceiling. Only embeddings train, and no continual pretraining is run. Evaluation is target-language only, so English regression is mostly invisible. The one place English was checked, the short untied Qwen 8B run, dropped to 84.33 on Belebele from 92.56 before recovering with longer training. There are no truly low-resource languages and no encoder-only tests. And the speedup is asserted from compression. A vocabulary that grows by tens of thousands of tokens enlarges the output head, and no latency or memory is measured.

69
00:15:42,244 --> 00:15:43,590
[Hal Turing] So who should use this?

70
00:15:43,590 --> 00:16:08,714
[Dr. Ada Shannon] Someone adapting a base model to a mid-resource language the tokenizer serves poorly, with one GPU and a few hours. Apply the loss around a quarter to a third of the depth, and expect most of the understanding back. Budget for a longer generation fix if writing the language matters. If your model has untied embeddings, add a language-modeling loss. If you need guaranteed latency gains, wait for measured evidence.

71
00:16:08,714 --> 00:16:10,015
[Hal Turing] And the next experiments?

72
00:16:10,015 --> 00:16:26,129
[Dr. Ada Shannon] Hidden-state versus AIM versus simplified AIM at the same layer. Normalized multi-layer AIM. A tuned Token Distillation sweep, a ZeTT baseline, a random-initialization control, seeds, and continual pretraining after MATT, counting tokens to parity.

73
00:16:26,129 --> 00:16:50,928
[Hal Turing] So, a short embedding-only warm-up, guided by a mid-network target from the original model, restores most of a swapped tokenizer's understanding, well beyond what similarity heuristics manage alone. What isn't established is that attention dynamics are the reason, or that the margin over the closest competitor survives tuned baselines. Thanks for listening, everyone.

74
00:16:50,928 --> 00:16:52,182
[Dr. Ada Shannon] Thanks for listening.