Self-Evolving Curriculum: Bandit-Selected Problems for LLM Reasoning RL

The SEC loop

A tiny bandit table sits beside the LLM and picks the next data category. Step through it.

P(category c) ∝ exp(Q_c / τ) · Q_c ← α·r_c + (1−α)·Q_c · r_c = mean |advantage| over c's rollouts

Where rollouts goIllustrative

DAPO-style dynamic sampling generates then discards flat groups. SEC reuses advantages GRPO already computes. Tile counts are schematic; the paper reports no rollout counts.

GRPO: why flat groups teach nothing

8 rollouts on one problem, binary reward. Advantage = (r − mean) / std. Drag k = number correct.

correct k = 3/8

Mean |advantage| is a pass-rate detector

Binary case, appendix result: E|A| = 2·√(p(1−p)). Peaks at p = ½, zero at 0 and 1. Click a dot.

Puzzle tasks add 0.1 partial credit for correct format, so this clean shape is a guide there, not exact.

Toy non-stationary banditIllustrative toy, not paper data

5 difficulty arms. Pass rate = sigmoid(1.6·(competence − difficulty)); competence grows with the reward collected. Compare SEC vs Random vs Ordered in this toy.

α 0.5 τ 0.30 step 60

Reported comparisons

Accuracy, higher is better. Single training run per method; values marked ≈ are read from the episode discussion.

Win / tie / loss map (qualitative)

Direction of SEC vs Random as described in the episode; hover a cell.

Hover a cell for the detail.

How big is "one problem"?

AIME24 has 30 problems. Each square is one problem; filled part = solved fraction.

Sampling error on a held-out setIllustrative arithmetic

Host's back-of-envelope: SE = √(p(1−p)/n). Test-set size is not stated in the paper. Excludes run-to-run training variance. Scores ≈0.48 vs ≈0.54.

n problems 200

Evidence ledger

What the episode says the paper does and does not report.