A 1.5B model beat OctoCoder on HumanEvalFix. A random-reward control told me why.
I wanted a small model to beat a much larger one on code repair, and I wanted to know why if it did.
The target
The model was Qwen2.5-Coder-1.5B-Instruct. Under the OctoPack paper’s own HumanEvalFix protocol, the untrained model scored 17.6% pass@1. The target was OctoCoder, which reports 30.4%; it’s StarCoder instruction-tuned, a 15.5B-parameter model. I locked the protocol and the target before training, used no benchmark-derived training data, and set a strict bar: the mean and every seed above the line.
The result
Supervised fine-tuning reached 38.6% pass@1 as a 3-seed mean. The worst seed scored 34.5% and the best 42.65%.
Why: the data
The change that mattered was the training data. For $0.34, an LLM broke clean functions into 2,551 bugs in the benchmark’s docstring style, covering all 374 source functions. A bug only counted if it compiled and failed a real test.
Why not: the RL
DPO and GRPO tied SFT or nudged it slightly. The nudge looked like a result, so I ran GRPO with a random reward. It reproduced the gain: real minus random was +0.61 pass@1, zero within noise. At this scale, the RL signal wasn’t doing the work.
When does fine-tuning stop mattering?
Then I wrapped both models in the same scaffold: sample several fixes, run each problem’s tests in a sandbox, repair on failure. Training’s lift falls as the scaffold gets more tries: +21.3 points at one try, +14.3 at three, and +2.5 with repair at the largest budget. With enough verified retries the untrained model reaches 65.2% on its own.
So: fine-tune when you can’t verify; scaffold when you can.
Caveats
One benchmark, one model family for the headline, and the comparison sets a 2023 model against a 2024 base model. The target is OctoCoder’s published number, not a re-run under my harness.
Code, protocol and the full lab notebook: github.com/nidhi1603/post-training-lab. The case study has the charts.
Questions or a role where this fits? nidhi.rajani.ds@gmail.com ·LinkedIn