1
00:00:01,000 --> 00:00:07,640
[Hal Turing] Alrighty! Thanks for tuning in! Hello AI world! I am your host, Hal Turing, and my co-host is Dr. Ada Shannon.

2
00:00:07,640 --> 00:00:37,408
[Hal Turing] Today we're looking at "MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies." It's by Shengding Hu and colleagues at Tsinghua University and Modelbest Inc. The version we have is arXiv v3, dated June 3rd, 2024. The claim that jumped out at me: the authors say the compute-optimal ratio is about 192 training tokens per parameter. Chinchilla said about 20.

3
00:00:37,408 --> 00:00:59,467
[Dr. Ada Shannon] Roughly ten times off, from a single fit. That's the tension for the whole episode. The paper's practical result is solid: a 2.4B model that competes with 7B-class models, built with a learning-rate schedule you can pause and resume. The 192 number is much shakier than the models themselves. We'll get to that.

4
00:00:59,467 --> 00:01:03,136
[Hal Turing] Start with the basics, then. What counts as a small language model here?

5
00:01:03,136 --> 00:01:33,276
[Dr. Ada Shannon] The paper's working definition is about 7B parameters or fewer, small enough to run on a laptop or phone without a GPU. Architecturally it's the same decoder-only transformer as GPT or Llama. MiniCPM comes in 1.2B and 2.4B, counted in non-embedding parameters. At this scale the vocabulary table is a big share of the total, and it costs memory but adds little compute per token. So non-embedding count is the fairer yardstick.

6
00:01:33,276 --> 00:01:35,505
[Hal Turing] And the pitch for going small is deployment?

7
00:01:35,505 --> 00:02:03,090
[Dr. Ada Shannon] Deployment is half of it. Most of a deployed model's lifetime cost is inference, so a smaller model trained longer shifts cost from every query to one training bill. The other half is that small models are cheap laboratories. Few groups can run experiments at 70B, so if a method validated on small models transfers upward, outside groups can do real scaling research. Phi, TinyLlama, Gemma and MobileLLM are the neighbors in this space.

8
00:02:03,090 --> 00:02:06,109
[Hal Turing] Which brings us to scaling laws. Give me the setup.

9
00:02:06,109 --> 00:02:28,586
[Dr. Ada Shannon] Loss is modeled as L of N and D equals C_N times N to the minus alpha, plus C_D times D to the minus beta, plus L0. N is parameters, D is training tokens, and L0 is the irreducible loss. Alpha and beta say how fast loss falls as you add parameters or data. If alpha is bigger than beta, parameters pay off faster.

10
00:02:28,586 --> 00:02:30,304
[Hal Turing] Then compute-optimal means what?

11
00:02:30,304 --> 00:02:56,217
[Dr. Ada Shannon] Fix a compute budget C, roughly six times N times D in FLOPs. Pick the N and D that minimize loss under that budget. Kaplan and colleagues at OpenAI, 2020, pushed toward bigger models. Then Hoffmann and colleagues at DeepMind, the Chinchilla paper from 2022, found about 20 tokens per parameter, using three different estimation methods that agreed. That's why people say Chinchilla is well supported.

12
00:02:56,217 --> 00:02:58,400
[Hal Turing] So why would anyone want to train past 20?

13
00:02:58,400 --> 00:03:25,660
[Dr. Ada Shannon] Because Chinchilla optimizes training compute only. If you serve the model billions of times, a smaller model trained longer can reach the same quality with cheaper inference. TinyLlama, Peiyuan Zhang and colleagues in 2024, trained a 1.1B model on about 3 trillion tokens, thousands of tokens per parameter. MiniCPM argues the optimum itself is far higher than 20, which is a stronger claim than just choosing to over-train.

14
00:03:25,660 --> 00:03:30,676
[Hal Turing] How do you even measure that cheaply? Every point on a scaling curve seems like a full training run.

15
00:03:30,676 --> 00:03:55,521
[Dr. Ada Shannon] That's where the learning-rate schedule matters. The learning rate is the step size of each update. The default for LLM pretraining is cosine decay: a short warmup, then a smooth curve down to near zero exactly at the end of a training length fixed in advance. The paper reproduces the known result that cosine works best when its period matches the total training length, tested at 20 to 80 tokens per parameter.

16
00:03:55,521 --> 00:03:58,075
[Hal Turing] So if I stop cosine halfway, the model is bad.

17
00:03:58,075 --> 00:04:14,190
[Dr. Ada Shannon] Right, you're stopped at a high learning rate. And if you decide to train longer, you restart from scratch. For scaling laws that means a separate full run for every pair of model size and token count. The cost grows with the square of how many sizes and token counts you want.

18
00:04:14,190 --> 00:04:15,676
[Hal Turing] What does their schedule change?

19
00:04:15,676 --> 00:04:41,032
[Dr. Ada Shannon] Warmup-Stable-Decay, WSD. Three phases: a short warmup, a long stable phase at constant rate, and a final decay over about 10 percent of the tokens. The stable phase doesn't depend on the final budget. So at any stable checkpoint you can branch off a short decay and get a finished model, or keep going at the high rate. The loss drops sharply once decay starts.

20
00:04:41,032 --> 00:04:43,958
[Hal Turing] That also sounds like continual pre-training for free.

21
00:04:43,958 --> 00:05:10,243
[Dr. Ada Shannon] That's the idea. Continual pre-training means resuming from a checkpoint with more data or a different mixture instead of restarting. Under cosine, the checkpoint you'd resume from has already annealed. Under WSD, the high-rate checkpoint stays reusable. Related horizon-free ideas exist, like the reciprocal-square-root schedule with cooldown in Zhai and colleagues' Scaling Vision Transformers from 2022.

22
00:05:10,243 --> 00:05:13,029
[Hal Turing] There's also the wind tunnel thing. I saw that in the abstract.

23
00:05:13,029 --> 00:05:37,735
[Dr. Ada Shannon] It's the authors' name for cheap tests on tiny models, 0.009B up to 0.5B, of batch size, learning rate and stability. The aim is to find settings that carry over to the final model, like testing a scale model of an aircraft. That only works if the best learning rate stops drifting as width grows. Under standard parameterization it does drift, which is why big models need smaller rates.

24
00:05:37,735 --> 00:05:39,825
[Hal Turing] And the fix is the Tensor Program stuff?

25
00:05:39,825 --> 00:06:09,872
[Dr. Ada Shannon] Yes. Greg Yang and colleagues at Microsoft, Tensor Programs V in 2022, introduced muP, a parameterization that rescales initialization and per-layer rates so the optimal hyperparameters stay roughly constant across width. You tune on a small proxy and copy the values up. The later Tensor Programs VI handles depth with a residual scaling rule. Transfer is better validated for width than for depth or longer training, so it lowers risk without removing the need to check at full scale.

26
00:06:09,872 --> 00:06:17,813
[Hal Turing] So we have a cheap-experiment toolkit, a restartable schedule, and a claim that the data-to-model ratio is ten times Chinchilla's.

27
00:06:17,813 --> 00:06:24,454
[Dr. Ada Shannon] And the question is whether that claim holds up. The fit depends on the experiments underneath it, so those come first.

28
00:06:24,454 --> 00:06:28,633
[Hal Turing] Then start with batch size. How do you choose one without training the big model?

29
00:06:28,633 --> 00:06:57,565
[Dr. Ada Shannon] They train 0.009B, 0.03B and 0.17B models at several batch sizes and fit a parabola to find the best one for each. That best batch size follows the loss the model reaches. On C4 the fit is batch size equals 1.21e9 divided by loss to the 6.24th power. Lower loss calls for a bigger batch. The paper itself notes the catch: the rule needs a loss estimate, and you only have a trustworthy one after training.

30
00:06:57,565 --> 00:06:58,587
[Hal Turing] And learning rate?

31
00:06:58,587 --> 00:07:15,956
[Dr. Ada Shannon] With the width and depth scaling rules applied, the best learning rate stays near 0.01 from 0.04B to 0.5B. They also checked it at 2.1B. The transfer is empirical. Nothing guarantees it for deep models, so the paper shows it holds in the range they tested.

32
00:07:15,956 --> 00:07:19,160
[Hal Turing] Now the schedule. Why does a short decay matter so much?

33
00:07:19,160 --> 00:07:42,798
[Dr. Ada Shannon] Hold the learning rate high during the long stable phase, then decay it, and the loss drops sharply during the decay. Because the stable phase never decays, you can branch a decay off any checkpoint, or keep training at the high rate. On a 0.036B model, a decay over 10% of the tokens matched a full cosine run at the same token count. A 2.5% decay fell short.

34
00:07:42,798 --> 00:07:44,191
[Hal Turing] So what do you get from that?

35
00:07:44,191 --> 00:08:14,098
[Dr. Ada Shannon] One stable run yields many cosine-quality data points. A scaling study then costs a number of runs proportional to the number of model sizes, not its square. They also trained that 0.036B model far past the usual token count and it kept improving. It reached the quality of a 0.17B model at about four times the training compute, and each later query costs roughly a fifth as much. That is one dataset, one small model, a single seed and no variance.

36
00:08:14,098 --> 00:08:16,885
[Hal Turing] Do they explain why the loss drops during decay?

37
00:08:16,885 --> 00:08:34,578
[Dr. Ada Shannon] They describe it and stop there. Weight updates track the learning rate. Yet the loss falls even as updates shrink. The gradient cosine between steps turns positive and the curvature rises. The paper calls this suggestive and leaves the explanation for future work, so it is a description and not a theory.

38
00:08:34,578 --> 00:08:37,272
[Hal Turing] Back to the scaling fit. What did they actually run?

39
00:08:37,272 --> 00:09:05,043
[Dr. Ada Shannon] Six model sizes, roughly 0.04B to 2B, each with six decay checkpoints from 10 to 60 tokens per parameter. They measured loss in bytes on five held-out corpora and fit with scipy curve_fit. The exponent on parameters is α and the exponent on data is β. Define η as β minus α, over α plus β. When β is below α, η is negative, and the optimal tokens-per-parameter ratio grows with compute.

40
00:09:05,043 --> 00:09:06,065
[Hal Turing] And the numbers?

41
00:09:06,065 --> 00:09:32,628
[Dr. Ada Shannon] On the average fit, α is 0.29, β is 0.23 and η is minus 0.10. At 1e21 FLOPs that gives about 192 tokens per parameter. Per corpus, the ratio runs from about 96 to about 834. The paper also reanalyzed Llama-2 as a cross-check and got roughly 70 to 100. That method reads curve intersections off a plot, and the paper itself calls it only a rough check.

42
00:09:32,628 --> 00:09:36,436
[Hal Turing] Moving on to what happens to the model itself. Two-stage training?

43
00:09:36,436 --> 00:10:03,372
[Dr. Ada Shannon] The idea is to mix high-quality and SFT-style data into the decay phase and not only into the later fine-tuning. On the 2.4B model, same SFT data, different timing: C-Eval, a Chinese knowledge exam, went from 40.0 to 52.6. On the 1.2B model, doubling SFT tokens from 6B to 12B gained almost nothing, while moving data into decay gained about 8 C-Eval points.

44
00:10:03,372 --> 00:10:04,207
[Hal Turing] Any losses?

45
00:10:04,207 --> 00:10:23,991
[Dr. Ada Shannon] GSM8K, grade-school math, dropped for the 1.2B decay-mix model, from 34.4 to 31.8. The comparison also changes total token exposure, and the data is proprietary. Each variant is a single run with no error bars. Part of the gain is probably the Chinese-heavy data mix.

46
00:10:23,991 --> 00:10:25,431
[Hal Turing] Then the models themselves.

47
00:10:25,431 --> 00:11:02,536
[Dr. Ada Shannon] MiniCPM comes in 1.2B and 2.4B non-embedding parameters. They are deep and thin, with shared embeddings, and the 1.2B uses grouped-query attention. Both trained on about 1.1 trillion tokens, with an exponential decay over 20B tokens and then roughly 6B SFT tokens. The family also has a DPO version, where MTBench rose from 6.89 to 7.25 at a small benchmark cost. It also has a 128K-context version and a mixture-of-experts model with 13.6B total and about 4B active parameters.

48
00:11:02,536 --> 00:11:05,183
[Hal Turing] What does the 2.4B actually do on benchmarks?

49
00:11:05,183 --> 00:11:29,378
[Dr. Ada Shannon] On the paper's average, it lands level with 7B-class models like Mistral-7B and Gemma-7B, and ahead of Llama2-13B. The 1.2B beats Llama2-7B on average. The edge shrinks or reverses on English knowledge: it trails Mistral-7B by about nine points on MMLU. The paper reads BBH as a sign that reasoning depends more on size than knowledge does.

50
00:11:29,378 --> 00:11:31,375
[Hal Turing] So the average is not a uniform win.

51
00:11:31,375 --> 00:11:43,496
[Dr. Ada Shannon] Right. The average mixes Chinese and English benchmarks, and the paper reports each model's better score from two scoring methods. It's a strong result for the size, but not a clean apples-to-apples comparison.

52
00:11:43,496 --> 00:11:49,487
[Hal Turing] Let's test the headline number. The ratio is about 192 tokens per parameter. How well pinned down is it?

53
00:11:49,487 --> 00:12:13,264
[Dr. Ada Shannon] Not well, and the paper doesn't say how well. The fit uses six model sizes, six decay checkpoints each, and five free parameters. There are no confidence intervals on the 0.29 and 0.23 exponents. There's no check on sensitivity to initialization, no leave-one-size-out test, and no robust loss like Huber. So the uncertainty is unknown, not small.

54
00:12:13,264 --> 00:12:15,632
[Hal Turing] But the per-dataset spread is already a warning.

55
00:12:15,632 --> 00:12:40,385
[Dr. Ada Shannon] Yes. The ratio runs from about 96 on Ultratext to about 834 on Chinese Wikihow, nearly nine times apart. Chinese Wikihow also has the largest negative η, so it pulls any arithmetic average upward. And an average of per-dataset ratios isn't the ratio of an averaged fit. So 192 summarizes unstable estimates. It isn't a measurement of one constant.

56
00:12:40,385 --> 00:12:43,961
[Hal Turing] Then there's the compute point. They quote it at 1e21 FLOPs.

57
00:12:43,961 --> 00:13:06,763
[Dr. Ada Shannon] At that budget the optimal model is about 0.93B parameters, which is inside the range they trained. But the data is about 190 tokens per parameter, and their largest checkpoint is 60. So the data side is extrapolated roughly threefold. Since η is negative, the ratio also changes with compute. The paper's 192 is one point on a curve.

58
00:13:06,763 --> 00:13:08,760
[Hal Turing] Chinchilla had the same kind of fit, though.

59
00:13:08,760 --> 00:13:23,946
[Dr. Ada Shannon] Chinchilla reached about 20 with three different methods that agreed. Its third method is the parametric fit used here. MiniCPM has only that one method, and reports no intervals for it, so there is nothing to compare against Chinchilla's reported uncertainty.

60
00:13:23,946 --> 00:13:26,175
[Hal Turing] Does their Llama-2 cross-check rescue it?

61
00:13:26,175 --> 00:13:47,026
[Dr. Ada Shannon] Partly. It gives about 70 to 100, not 192. They built it by eye, reading curve intersections off cosine-schedule plots and straightening a concave segment to mimic WSD. The paper itself calls it rough. It supports 'well above Chinchilla' more than it supports 192. I'd say the direction is better supported than the number.

62
00:13:47,026 --> 00:13:49,627
[Hal Turing] Could the gap come from the recipe, not a different law?

63
00:13:49,627 --> 00:14:18,094
[Dr. Ada Shannon] That's plausible, and the paper can't rule it out. The data mix, tokenizer, byte-normalized loss, architecture and decay-based losses all differ from Chinchilla's setup. They never rerun the Chinchilla protocol on their own data. Also note the qualitative trend matches: α above β, as in Chinchilla. It's the level that differs tenfold. And the text says β is below α for every corpus, but Ultratext shows both at 0.30.

64
00:14:18,094 --> 00:14:23,342
[Hal Turing] What about the schedule itself? Is anyone else seeing the one-run, many-decays trick work?

65
00:14:23,342 --> 00:14:59,333
[Dr. Ada Shannon] Yes. Hägele and colleagues, 2024, in 'Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations', found that constant learning rate plus cooldown can reproduce cosine results and cut scaling-law compute. That backs the method. On why the loss drops during decay, Wen and colleagues, 2024, in 'Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective', give a theory. Roughly, the high rate moves fast along a valley floor and decay settles into it. MiniCPM only offers the descriptive gradient and curvature analysis.

66
00:14:59,333 --> 00:15:01,609
[Hal Turing] Where does the paper's own evidence stay thin?

67
00:15:01,609 --> 00:15:26,129
[Dr. Ada Shannon] WSD matching cosine is shown on one 0.036B model, one dataset, no seeds. The 10% decay rule comes from that model. The real MiniCPM runs use a decay of about 20B tokens out of 1.1T, under 2%, and the paper doesn't reconcile that. And the proprietary data makes it hard to check for benchmark-shaped gains in the Table 1 results.

68
00:15:26,129 --> 00:15:27,429
[Hal Turing] And the 7B comparison?

69
00:15:27,429 --> 00:15:47,956
[Dr. Ada Shannon] The better-of-two-scoring-methods rule and the English-Chinese average make it hard to reproduce. The BBH reading, that reasoning depends more on size than knowledge, is plausible, not proven. And nothing tests quantization, on-device latency, long-tail facts or safety, even though on-device use is the stated motivation.

70
00:15:47,956 --> 00:15:48,977
[Hal Turing] So what holds up?

71
00:15:48,977 --> 00:16:13,637
[Dr. Ada Shannon] A restartable schedule, with the caveat that its parity with cosine is shown at tiny scale. Cheap scaling studies from branching decays. Hyperparameters transferring across the range they tested. And small models trained far past 20 tokens per parameter being strong. For a deployment-bound model, train well beyond 20. Then measure your own optimum on your own corpus, with bootstrap intervals.

72
00:16:13,637 --> 00:16:14,891
[Hal Turing] What would settle the ratio?

73
00:16:14,891 --> 00:16:33,792
[Dr. Ada Shannon] Bootstrap intervals, refits dropping each model size, and different decay shapes. Chinchilla-style approaches one and two on the same data would give independent checks, and so would a direct replication of the Chinchilla protocol. The authors also list explaining the decay drop and scaling both model and data.

74
00:16:33,792 --> 00:16:55,758
[Hal Turing] So here's where I land. WSD works as a restartable schedule, decay-stage data matters, and small models trained on about a trillion tokens are competitive. The boundary is the 192-to-1 figure. It's a single fit at an extrapolated compute point, and it moves about ninefold by corpus. Thanks for listening, everyone.