How Reliable Are AI Coding Tools? A New Test Focuses on Each Model’s Limits
Jonathan Damery
Jonathan is Toptal’s Senior Editor for Engineering, where he develops technical articles and thought leadership content. He previously edited a magazine at Harvard University and holds a master’s degree in literary nonfiction writing from the University of Minnesota.
When AI companies release new models, they often tout their scores on standardized tests called benchmarks. But traditional benchmark methodologies have recently faced questions about reliability from AI researchers as well as the companies that make the models.
One reason for those doubts is that existing benchmarks can sometimes conceal big differences between models. For example, on one popular coding benchmark, DeepSWE, models from Anthropic, OpenAI, and DeepSeek all score within a point of each other. But when the same models were tested on a new benchmark, the gap between them grew to nearly 35 points, separating models that had looked evenly matched.
The new benchmark, ExpertCrowd-SWE, tested seven leading AI models on newly written software engineering tasks designed to reflect realistic, multistep work. The results showed that Anthropic’s Claude Opus 5.5 had the highest average score, 67.3%, followed closely by OpenAI’s GPT-6 Astra at 63.7%, a gap within the margin of error. Claude Fable 5.1 scored 46.5%, Gemini 3.8 Flash scored 40%, and Qwen 3.8 Max scored 33.5%. DeepSeek’s model, which scored within a point of Claude Opus 5.5 and GPT-6 Astra on DeepSWE, finished sixth at 32.5%. SpaceXSI’s Grok 4.6 came in seventh at 32.2%.
The benchmarking tasks were chosen for areas where models tend to struggle, so scores are lower than they would be on a broader range of work. The tasks were designed to represent real software engineering assignments and were not artificially complicated problems. The result was a map of each model’s limits, which can help teams decide which models are best for which kinds of tasks.
ExpertCrowd-SWE is a collaboration between ExpertCrowd, a platform for training and improving AI models and systems developed by AI services company Toptal AI, and HUD, a Y Combinator-backed company that builds reinforcement learning environments.
Methodology: How the Test Works
For ExpertCrowd-SWE, software engineers from Toptal’s talent network created the benchmark tasks based on 87 open-source projects in 11 programming languages, along with the criteria for grading each one. The engineers wrote each task the way real work typically arrives, as an issue report describing what needs to be fixed or added. Each report was paired with a full copy of each project’s code and a sandbox where the models could read files, run commands, and edit code. The models had to investigate the codebase, decide what to change, make adjustments, and check their work within a set number of actions per attempt. HUD provided the infrastructure to host task environments and run evaluations.
ExpertCrowd ran all seven models with the same tools and the same step limit. Each model attempted every task five times with its advanced reasoning settings enabled. The attempts were scored against a rubric written by the engineer who built each task. Models that got partway to a correct solution received partial credit. After the tests were run, members of Toptal AI’s research team reviewed the methodology and results.