A coding agent can top the leaderboard and still fail in your repo
Benchmarks show broad ability, not whether an agent understands your code, tests, and conventions. A small trial on real work tells you more.
Public benchmarks are useful for shortlisting, not for predicting performance in your repository. Dependencies, test speed, conventions, and permissions all change real success rates.
Sample real tickets
Sample completed bugs, refactors, and small features. Hide the final patch and provide only information available at the time. Cover different directories, task lengths, and risk levels instead of demo-friendly cases.
Passing tests is not enough
Passing tests is only the first gate. Check requirements, unnecessary edits, and maintainability. Record review time and rework because faster generation can simply move effort into review.
Repeat in a fixed environment
Fix the repository snapshot, dependencies, permissions, and time limit, then repeat each task. Report success rate, cost, median duration, and high-risk failures rather than one lucky run.
Pre-release checks
- Tasks come from real history, not invented puzzles
- Reviewers are blind to model identity
- Failed patches are reproducible and categorized
Sources
Related
Writing, extraction, long reasoning, and high-volume support do not need the same model. Cost, latency, privacy, and reliability rarely point to one name.
GPUs, operations, inference tuning, and downtime all land on the same ledger. Self-hosting only makes sense after you have done the math on volume and people.