Coding
SWE-bench Verified
500 human-validated GitHub issues from real Python repositories. The model must produce a patch that makes the project's own tests pass.
SWE-bench Verified is scored as a percentage of questions answered correctly. 33 of the 159 models in this index report a score for it. The highest published figure here is 80.9%, from Claude Opus 4.5.
Reported scores
Top 20 on SWE-bench Verified
Ordered by the figure each maker published. Models that have not reported this benchmark are not listed — an absent score is not a low score.
| # | Model | Published score |
|---|---|---|
| 1 | Claude Opus 4.5Anthropic | |
| 2 | Claude Sonnet 4.5Anthropic | |
| 3 | Gemini 3 ProGoogle DeepMind | |
| 4 | GPT-5OpenAI | |
| 5 | Claude Opus 4.1Anthropic | |
| 6 | Claude Haiku 4.5Anthropic | |
| 7 | Claude Sonnet 4Anthropic | |
| 8 | Grok 4xAI | |
| 9 | Kimi K2 ThinkingMoonshot AI | |
| 10 | GPT-5 miniOpenAI | |
| 11 | Grok Code Fast 1xAI | |
| 12 | Qwen3-MaxAlibaba Qwen | |
| 13 | Qwen3-Coder-480B-A35BAlibaba Qwen | |
| 14 | MiniMax-M2MiniMax | |
| 15 | o3OpenAI | |
| 16 | o4-miniOpenAI | |
| 17 | GLM-4.6Z.ai (Zhipu) | |
| 18 | DeepSeek-V3.1DeepSeek | |
| 19 | Kimi K2 InstructMoonshot AI | |
| 20 | GLM-4.5Z.ai (Zhipu) |
Where these numbers come from
Every score on this page is a published figure, taken from the model's own card, system card, technical report or release post, or from a public leaderboard. CorX Labs did not run these evaluations. Most are self-reported by the lab that built the model, which means they were produced under that lab's own choice of prompt, scaffold and number of attempts — so treat them as a starting point for a shortlist, not as a settled ranking.
A score someone other than the model's maker measured is marked Independent and names its measurer. Those are the stronger numbers on this page — an outside harness has no reason to flatter anyone — and there are not many of them.
Where a figure has not been published, the cell reads Not reported rather than an estimate. Nothing here is inferred, interpolated or guessed. Each model records the month its row was last checked. Full method and caveats.
How to read this score
SWE-bench Verified is the most decision-relevant benchmark here for anyone using a model to write code. Each task is a real GitHub issue, and the patch has to make the project's own test suite pass — there is no partial credit and no judge model.
Where it is weak
It measures an entire agent, not a model. Retrieval, retries and test execution are all part of the harness, and two labs reporting the same number may be running very different scaffolds. It is also Python-only, on twelve repositories.