Introducing ExpertCrowd-SWE: An AI Benchmark for Coding Agents

ExpertCrowd-SWE is a new benchmark that tests how well AI coding agents handle realistic software engineering work. It arrives at a time when frontier models are saturating many established coding benchmarks, and some of those benchmarks have proven unreliable. OpenAI, for instance, stopped reporting results for SWE-bench Verified in February 2026 after determining that models were performing better, in part, because they had encountered the benchmark’s problems or solutions in their training data. Five months later, the company retracted its endorsement of the successor benchmark, SWE-bench Pro, after an audit found roughly 30% of its tasks were broken.

ExpertCrowd-SWE is designed to address these challenges. Drawing on Toptal’s network of more than 30,000 specialized domain experts, ExpertCrowd enlisted experienced engineers to write 100 new agentic tasks based on real issues in large codebases. The tasks were checked against existing public benchmarks and validated against a human-written reference solution, with grading designed to accept any valid answer. Only tasks that today’s frontier models can’t yet complete reliably made it into the benchmark.

We tested seven frontier models, two from Anthropic and one each from OpenAI, Google, Alibaba, DeepSeek, and SpaceXSI. Scores ranged from 32.2% to 67.3%, with Claude Opus 5.5 at the top. Every task tripped up at least one model, leaving plenty of room to measure future progress. Below, we explain how the benchmark was built and what the results reveal about each model’s limits.

Qualifying AI Agents

ExpertCrowd’s benchmarks qualify AI agents by identifying the point at which they stop being reliable, showing teams where an agent can be trusted and where it can’t. This reverses the usual way systems are vetted. Software and equipment are typically checked against predefined requirements before they’re put into service, either through acceptance testing or, in regulated industries, formal qualification. AI agents break that pattern: Their capabilities emerge from training rather than design, so no one can specify in advance where an agent’s competence ends.

To locate those limits, domain experts from Toptal’s network build realistic task environments, run each agent repeatedly, and record where and how it fails. The result is a map of each agent’s limits, with every failure documented in enough detail to reproduce it and to point toward targeted training that could expand what the agent can reliably accomplish.

Our approach can be tailored to a specific AI system, scaling to the scope of the work. A frontier lab preparing its next model might need several thousand cross-functional specialists to probe for weaknesses across a range of capabilities, while a smaller company adapting an open-weight model for a specialized application might need a more targeted group to identify where fine-tuning or expert data is needed. In both cases, ExpertCrowd brings together professionals with hands-on experience in the relevant fields, who effectively provide external qualification for AI.

Methodology

The ExpertCrowd-SWE benchmark tests whether AI coding agents can do the core work of professional software engineering: resolving issues in large codebases they haven’t seen before. Its 100 tasks span 87 open-source projects in 11 languages.

Engineers from Toptal’s talent network led the development of this benchmark and authored every task as well as the grading rubric. Researchers, including several PhDs in various areas of artificial intelligence from Toptal’s AI team, reviewed and assessed the benchmark’s methodology and results.

The sections below explain how the tasks were designed and graded, how the final 100 were chosen, and how the models were evaluated.

Task Design

Our engineers wrote each task the way real work arrives, as an issue report. They paired it with a complete checkout of the repository and a sandbox where the agent can read files, run commands and tests, and edit code. The agent then has to investigate the codebase, decide what to change, make the change, and check its own work, within a limit of 1,000 steps per attempt.

The issues themselves came from three sources:

1. Issues in public repositories, identified through hands-on experience with each codebase
2. Long-standing unresolved issues in those same repositories
3. New problems designed for the benchmark and added to forks of public repositories

The projects include databases and query tools (SQLAlchemy, SQLGlot, immudb), distributed systems (containerd, HashiCorp Raft, MinIO), authentication and proxy tools (OAuth2 Proxy, Authelia, Ory Kratos), and background job systems (Celery, Asynq, Oban).

Like real issue reports, the tasks don’t spell out every detail. A task may leave out anything a developer could find in the code, its tests, or the project’s conventions, but never anything the rubric checks for. That distinction matters. SWE-bench Verified was created after researchers found that some tasks in the original SWE-bench were missing details the tests checked for, so a model could solve the problem correctly and still fail. ExpertCrowd-SWE tests whether an agent can investigate a problem rather than guess what the grader expects.

Grading

Each ExpertCrowd-SWE task has its own rubric, written by the engineer who built it. The rubric combines automated tests, which check that the code behaves correctly, with an LLM judge for qualities that tests can’t capture, and returns a score from 0% to 100%, so an agent that gets partway to a correct solution receives partial credit. Every task uses the same judge, Claude Sonnet 4.6. Before a task is accepted, the engineer’s reference solution must score at or near 100%, and the unmodified repository is checked against a 0% target, which confirms that the environment works and that the rubric rewards a correct fix.

Task Selection

Our engineers built more tasks than the benchmark needed. Tasks on which agents consistently ran into the step limit were dropped.

We then checked the tasks against public software engineering benchmarks. First, we compared our tasks with 499,647 task records from 93 public datasets, looking for matching commits or task IDs. For the 46 repositories that our tasks share with those datasets, we then compared 7,726 records more closely, on both the task descriptions and the code changes. Finally, an LLM judge reviewed the 231 closest matches in full. None of them was the same task as ours or tested the same thing, though 38 involved the same part of a codebase with a different change.

We kept the 100 tasks that six of the models found hardest, based on their average scores. (Gemini 3.8 Flash was left out of the selection because it lacked a full set of valid runs on some tasks.) As a result, the models score lower than they would on similar tasks chosen without regard to their performance. What the benchmark is designed to show is not a single number but the shape of each model’s limits, including which tasks it handles reliably, which it handles only sometimes, and which it can’t yet do at all.

Evaluation Setup

We ran seven models (Claude Opus 5.5, GPT-6 Astra, Claude Fable 5.1, Gemini 3.8 Flash, Qwen 3.8 Max, DeepSeek V4.1 Flash, and Grok 4.6) as agents in the same harness, with the same tools and the same 1,000-step limit. Each model attempted every task five times with extended reasoning enabled.

A model’s score on a task is the average across its five valid runs, and its benchmark score is the average across all 100 tasks, reported with a 95% confidence interval. In total, the evaluation produced 3,495 valid runs. Runs that hit the step limit or the output-token limit were excluded from scoring. Gemini 3.8 Flash had no valid runs on one task, so its score covers 99 tasks.

Results

On ExpertCrowd-SWE, two models lead the other five by a wide margin. Claude Opus 5.5 scored 67.3% ± 4.7 and GPT-6 Astra 63.7% ± 5.5, though the gap between them isn’t statistically significant. Claude Fable 5.1 followed at 46.5% ± 3.7 and Gemini 3.8 Flash at 40.0% ± 4.3. The scores of Qwen 3.8 Max (33.5% ± 3.3), DeepSeek V4.1 Flash (32.5% ± 3.7), and Grok 4.6 (32.2% ± 3.5) couldn’t be separated statistically. Across all seven models, the average score was 45.1%.

The performance lead of the top two models holds up task by task, not only on average. Opus 5.5, for instance, outscored Fable 5.1 on 82 of the 100 tasks. Even so, the leaders are far from saturating the benchmark. No task was fully solved by all seven models, and on 80 of the 100 tasks, no model earned full credit on every run. On only four tasks did both Opus 5.5 and GPT-6 Astra score a perfect 100%, and on 20, both scored below 60%.

Repeated runs show how uneven that performance is. Of the 699 combinations of model and task, only 26 earned full credit every time, and 14 earned no credit on any run. The rest earned partial credit, varied from run to run, or both. Each model also has areas where it struggles. On 15 tasks, at least one of the five lower-ranked models outscored both Opus 5.5 and GPT-6 Astra.

Despite having similar scores, Opus 5.5 and GPT-6 Astra reached them in very different ways. GPT-6 Astra averaged 37 agent steps per run, compared with 171 for Opus 5.5. Step counts alone don’t measure efficiency, but for teams running agents at scale, differences like this will factor into cost calculations.

Example Failures

Two tasks illustrate the kinds of failure ExpertCrowd-SWE exposes. In the first, an agent must fix how OAuth2 Proxy, a widely used authentication proxy, accounts for session and CSRF cookies. No model averaged more than 22%. GPT-6 Astra reached 21.8%, Opus 5.5 and Fable 5.1 both reached 19.4%, and the other four models scored 8% or less. It’s a gap every model shares.

The second exposes a blind spot in individual models. In Asynq, a task queue library for Go, the agent must make sure a task records the correct outcome when it’s canceled or interrupted. Opus 5.5, the top-ranked model overall, earned no credit on any run, and neither did Grok 4.6. Meanwhile, DeepSeek V4.1 Flash, which ranked sixth overall, averaged 75.6%, and GPT-6 Astra, Gemini 3.8 Flash, and Qwen 3.8 Max each averaged 60%. A single benchmark score hides failures like these, but task-level results reveal them.

Comparison With DeepSWE

To see how ExpertCrowd-SWE differs from established benchmarks, we compared its results with DeepSWE v1.1, a public benchmark of long-horizon software engineering tasks. It’s the only public leaderboard we found that reports all seven models under a single evaluation setup. On DeepSWE, four of the seven models (Opus 5.5, GPT-6 Astra, DeepSeek V4.1 Flash, and Gemini 3.8 Flash) score within 0.9 points of one another. On ExpertCrowd-SWE, the same four span 34.8 points.

The two benchmarks agree on the top two models but diverge below them. DeepSeek V4.1 Flash, for example, ranks third on DeepSWE but sixth on ExpertCrowd-SWE. ExpertCrowd-SWE also spreads the models nearly twice as widely, separating models that DeepSWE can’t tell apart. Some of that spread reflects how the tasks were selected, but the shift in rankings shows the two benchmarks measure different things. The point isn’t that lower scores are better. It’s that ExpertCrowd-SWE reveals differences that established benchmarks don’t capture.

What’s Included

For each public task in ExpertCrowd-SWE, the release includes everything needed to see why a model scored the way it did, reproduce its failures, and measure whether a fix worked:

1. The task: The issue report the agent received.
2. The rubric: The tests and criteria used to grade each attempt.
3. The environment: The repository and sandbox the agent worked in, so runs can be reproduced.
4. The evaluation setup: The model versions, reasoning settings, and number of runs per task.
5. The results: Each model’s outputs and scores across all of its runs.

Where available, we also provide each run’s trace. The trace records the agent’s interactions with the environment, including the tools it called, whether those calls succeeded, and other observable steps leading to the final result, so teams can pinpoint where the model’s approach went off course.

Benchmarks typically keep some tasks private so models can’t be trained on them directly. For ExpertCrowd-SWE, 24 tasks are public, and 76 are private. The public tasks let anyone reproduce and inspect the evaluation, while the private tasks let developers confirm that improvements reflect real capability rather than familiarity with the published set.

Upcoming Benchmarks

ExpertCrowd is also applying the same approach to other domains, starting with management consulting. The consulting benchmark’s expert-designed tasks ask models to work through a case the way a team of consultants would, synthesizing documents, financial data, and other evidence to reach recommendations that may not have a single correct answer.

Improving Agent Performance

As AI agents take on more complex work, knowing where they can be relied on matters as much as knowing what they can do. Once a benchmark exposes a capability gap, ExpertCrowd helps close it. Our domain specialists create targeted training data on our multimodal annotation platform or design tasks and scoring criteria for reinforcement learning environments that target the identified weakness. Teams can handle retraining in-house or choose a fully managed engagement, then rerun the benchmark to see how far the model’s boundaries have expanded.

Whether you’re preparing a frontier model for release or adapting an open-weight model for your business, finding where it falls short is the first step to making it better, and ExpertCrowd can help.