Case study · post-training

Post-Training Lab

SFT vs DPO vs GRPO for code repair on Qwen2.5-Coder-1.5B, measured under the OctoPack paper's own frozen HumanEvalFix protocol, with no benchmark-derived training data. The data won, the RL didn't, and a random-reward control is how I know.

38.6%pass@1, 3-seed mean (target: OctoCoder's 30.4%)
34.5%worst seed, still above the target
$0.34to generate 2,551 certified training bugs
+0.61pass@1, real minus random-reward GRPO: no RL effect
RepoLab notebookResults explorer

The target, fixed before training

The untrained 1.5B model scored 17.6% pass@1 on HumanEvalFix under the paper's protocol. I locked the target before any training: beat OctoCoder, which reports 30.4%. OctoCoder is StarCoder instruction-tuned, a 15.5B-parameter model. The claim standard was strict, too: the mean and every seed above the line.

  1. Frozen rulerThe paper's HumanEvalFix protocol, locked and committed before training
  2. Data factoryAn LLM breaks clean functions; each bug must compile and fail a real test
  3. Three armsSFT, DPO and GRPO at matched compute, with paired seeds
  4. ControlsA random-reward GRPO run, and three seeds for the headline
  5. Scaffold testBest-of-n with test execution and self-repair, on both base and trained models

The result

Qwen2.5-Coder-1.5B, untrained17.6%
OctoCoder (target)30.4%
Ours, worst seed34.5%
Ours, 3-seed mean38.6%
Ours, best seed42.65%

Bars run from 0 to 50%. Every value links to the line it comes from.

What drove it: data, not RL

The winning change was data. For $0.34 in API calls, an LLM broke clean functions into 2,551 bugs written in the benchmark's docstring style, covering all 374 source functions. Every bug had to compile and fail a real test before it counted.

The RL arms tied or nudged. To check the nudge, I ran GRPO with a random reward. It reproduced the gain: real minus random came to +0.61 pass@1, which is zero within noise. So I credit the data, not the RL signal, at this scale.

Fine-tune or scaffold? The retry ladder

Next I wrapped both the base and the trained model in the same verifier-guided scaffold: sample several fixes, run each problem's tests in a sandbox, repair on failure. Training's value shrinks as the scaffold gets more tries:

1 try (base 26.3%)+21.3 pts
3 tries (base 44.7%)+14.3 pts
5 tries (base 53.0%)+9.6 pts
10 tries (base 62.2%)+3.7 pts
~14–18 tries (base 65.2%)+2.5 pts
With enough verified retries, the untrained model plus the scaffold reaches 65.2%, and training adds only +2.5 points, within noise. Fine-tune when you can't verify; scaffold when you can.

Limits

  • One benchmark (164 problems), one model family for the headline.
  • The comparison sets a 2023 model against a 2024 base model; newer bases start stronger.
  • The target is OctoCoder's published number, not a re-run of OctoCoder under my harness.