HumanEvalFix pass@1 from a fine-tuned 1.5B model (3-seed mean). OctoCoder, about ten times larger, reports 30.4%.
AI Engineer · Forward Deployed Engineer · ML Engineer
I build LLM systems, and the evaluations that tell me whether they work.
M.S. in Data Science from the University at Buffalo (May 2026). Before that, about a year as a customer-facing Solutions Engineer at Flipkart (Walmart). Now fine-tuning and evaluating LLMs at Cardio AI.
dev tasks in which a public agent benchmark leaked answer-key data to the agent. Found by my answer-independence check; reported as tau2-bench #574.
into LangChain: support for OpenAI's apply_patch tool in langchain-openai, +401/−18 across 9 files, merged 9 Jun 2026.
Two deep dives
Work I can defend line by line
Agent Eval Platform
Can a harness make the same model complete more banking tasks than the benchmark's standard agent? I built the evaluation machinery to answer that honestly, on Sierra's tau2-bench.
- False "transfer under way" replies: 26 of 27 → 2 of 27 in a replay test, after one feedback rewrite
- Task completion: 3 vs 3 of 30. No gain, and reported that way
- 13 measurement and harness defects found and fixed, each with a regression test
Post-Training Lab
SFT vs DPO vs GRPO for code repair on Qwen2.5-Coder-1.5B, measured under the OctoPack paper's frozen HumanEvalFix protocol, with no benchmark-derived training data.
More work
Fine-tuning, agents, open source
Code-review LoRA
Reasoning-trace SFT on Qwen2.5-Coder-7B that wins 86.0% of head-to-head reviews against the base model (CI [81.5%, 91%]) and cuts hallucinations 14.5% → 7.0%. Three rounds of RL on top didn't beat it, and the repo says so.
ProofCart
Hackathon agent across Slack, Notion and Stripe (test mode): reads supplier threads, waits for the owner's approval in Slack, pays, records. Exactly one charge under crash or duplicate (2/2 vs 0/2 against an ablation); 14/14 deterministic regression cases.
Sokratic tutor
Multi-agent Socratic tutor (UB). I built the first RAG pipeline (MRR 0.493 → 0.616), K-of-N partial reach in the reach gate, Haiku behaviour classifiers (29/30 on a check set) and an end-to-end regression harness. Reach rate: 38.9% → 66.1% across 59 conversations.
LangChain · Apache Airflow
LangChain: OpenAI's apply_patch built-in tool for langchain-openai, with unit tests (+401/−18 across 9 files, merged 9 Jun 2026). Airflow: a dependency fix in the AWS provider (+1/−1 across 1 file, merged 1 Apr 2026).
How I work
Three habits the evidence shows
Freeze the ruler before looking
Every paid experiment had a written plan and decision rule before any spend. At one run per task, an identical agent comes out ahead about 42% of the time, so gates sit above the noise.
Distrust my own numbers
13 defects found and fixed in the agent-eval pipeline: one in the public benchmark, the rest in my own code. Blind reviewers once agreed on three verifications that a later code check showed were invalid.
Report what didn't work
My harness fixed the behaviours it targeted but didn't raise completion (3 vs 3 of 30). In the lab, a random-reward control matched RL's gain, so I credit the data, not the RL.
Results explorer
Every experiment, defect and score in one place
Filter and sort the full record: ten full-conversation batches, fifteen targeted tests, thirteen defects, the post-training retry ladder and the code-review evals. Each row links to the line it was read from.