Upload README.md with huggingface_hub

README.md

@@ -9,6 +9,20 @@ GRPO RL fine-tune of the Marin **Delphi-25B** base model (`marin-community/delph
 
 This is the **step-160** checkpoint (20 epochs). Training reward on the hard-math distribution rose from ~4% to ~29%.
 
+## Training dynamics & generalization
+
+![RL scaling](rl_scaling.png)
+
+Held-out generalization (greedy, few-shot prompt) tracked against training step:
+
+| step | MATH-500 (few-shot) | MATH-500 (plain) | AIME-24 |
+|---|---|---|---|
+| 0 (base) | 5.2% | 1.4% | 0% |
+| 70 | **17.4%** (peak) | 0.6% | 0% |
+| 160 (this ckpt) | 13.6% | 2.4% | 0% |
+
+Read: RL produced **real held-out gains** (MATH-500 5.2% -> ~17%), but held-out **peaks around step 70 and then overfits** while train reward keeps climbing. The ability is **few-shot-format-locked** (plain-prompt held-out stays ~1-2.5%) and AIME-24 stays at **0%** throughout. The best-generalizing checkpoint is ~step 70, not the final step.
+
 ## Recipe
 - GRPO, rule-based boxed-answer reward (`math_verify`), `kl_loss_coef=1e-3`, `rollout.n=8`, lr 5e-7, top_p 1.0.
 - **Rollout temperature 0.5** (not the recipe default of 1.0 — a sweep showed the reward signal collapses above ~0.5 for this weak base on hard math).
@@ -44,8 +58,6 @@ Please reason step by step, and put your final answer within \boxed{}.<|im_end|>
 ```
 
 ## Example problems (illustrative of the reliable type; not guaranteed)
-Append each to the 3-shot prefix above. Sampling: greedy, stop=`<|im_end|>`.
-
 | Problem | Answer |
 |---|---|
 | What is the remainder when $2^{10}$ is divided by 7? | 2 |
@@ -55,4 +67,4 @@ Append each to the 3-shot prefix above. Sampling: greedy, stop=`<|im_end|>`.
 | A rectangle has length 8 and width 5. What is its area? | 40 |
 
 ## Capability notes (honest)
-RL gave a weak base **genuine but narrow** competence, and reasoning in the correct shape (steps -> single \boxed{} answer -> clean stop). It is reliable on problems solvable by **a single operation or one known formula with small/obvious inputs** (e.g. a Pythagorean step, a short modular cycle, a percentage). It is **unreliable** whenever it must (a) **track/combine several quantities** (multi-term word problems) or (b) **discover** something, such as a non-trivial factorization — in these cases it tends to confabulate a relation or factorization and not check it. ~29% on hard (level 3-5) MATH reflects this. It is a demonstration that the SimpleRL recipe works mechanically on a weak base; it is **not** a strong general math model.
+RL gave a weak base **genuine but narrow** competence, and reasoning in the correct shape (steps -> single \boxed{} answer -> clean stop). It is reliable on problems solvable by **a single operation or one known formula with small/obvious inputs**. It is **unreliable** whenever it must (a) **track/combine several quantities** or (b) **discover** something like a non-trivial factorization. It is a demonstration that the SimpleRL recipe works mechanically on a weak base; it is **not** a strong general math model.