Coding
HumanEval
164 short Python functions written from a docstring. Saturated at the frontier — kept here for continuity with older models.
HumanEval is scored as a percentage of questions answered correctly. 46 of the 159 models in this index report a score for it. The highest published figure here is 93.7%, from Claude 3.5 Sonnet.
Reported scores
Top 20 on HumanEval
Ordered by the figure each maker published. Models that have not reported this benchmark are not listed — an absent score is not a low score.
| # | Model | Published score |
|---|---|---|
| 1 | Claude 3.5 SonnetAnthropic | |
| 2 | Mistral Small 3.2 24BMistral AI | |
| 3 | Qwen2.5-Coder-32BAlibaba Qwen | |
| 4 | o1-miniOpenAI | |
| 5 | Mistral Medium 3Mistral AI | |
| 6 | Mistral Large 2Mistral AI | |
| 7 | GPT-4oOpenAI | |
| 8 | DeepSeek-Coder-V2DeepSeek | |
| 9 | Mercury CoderInception Labs | |
| 10 | Llama 3.1 405BMeta AI | |
| 11 | Amazon Nova ProAmazon | |
| 12 | Llama 3.3 70BMeta AI | |
| 13 | Grok 2xAI | |
| 14 | Qwen2.5-32BAlibaba Qwen | |
| 15 | Claude 3.5 HaikuAnthropic | |
| 16 | Gemma 3 27BGoogle DeepMind | |
| 17 | GPT-4o miniOpenAI | |
| 18 | GPT-4 TurboOpenAI | |
| 19 | Codestral 25.08Mistral AI | |
| 20 | Qwen2.5-72BAlibaba Qwen |
Where these numbers come from
Every score on this page is a published figure, taken from the model's own card, system card, technical report or release post, or from a public leaderboard. CorX Labs did not run these evaluations. Most are self-reported by the lab that built the model, which means they were produced under that lab's own choice of prompt, scaffold and number of attempts — so treat them as a starting point for a shortlist, not as a settled ranking.
A score someone other than the model's maker measured is marked Independent and names its measurer. Those are the stronger numbers on this page — an outside harness has no reason to flatter anyone — and there are not many of them.
Where a figure has not been published, the cell reads Not reported rather than an estimate. Nothing here is inferred, interpolated or guessed. Each model records the month its row was last checked. Full method and caveats.
How to read this score
HumanEval is included for continuity with older models rather than for its discriminating power. It is 164 short functions written from a docstring.
Where it is weak
Saturated and contaminated. Frontier models cluster above 90%, and the problems have been in training data for years. Treat any difference under about five points as meaningless.