🔍 Read the full analysis: FrontierCode: What Could Keep A Maintainer From Merging AI Code? on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, a benchmark designed to test whether code changes meet maintainers’ review standards, not just whether tests pass. Claude Opus 5.5 led with a 52.2% score and cleared every blocking check in 57.7% of attempts; the results show substantial variation in quality and cost, while leaving some major models untested.
FrontierCode’s central test is whether a patch would meet repository maintainers’ standards. A submission receives a zero if it fails any blocking check, regardless of how many other tests it passes. Attempts that pass all blocking checks receive scores from 70% to 100%, with additional deductions or credit tied to quality criteria. Artificial Analysis said it added the benchmark to its evaluations on October 12 and ran each model through the same open-source agent harness, Stirrup.
GPT-6.1 Sol ranked second with a 49.1% score and a 54.7% pass rate. Claude Fable 5.1 followed at 49.0%, while GPT-6 Astra scored 47.0%. Nine models from six labs were grouped between 43% and 52%; results fell below that range for models including Grok 4.7, at 33.8%, and MiniMax-M3, at 23.0%. The benchmark’s 100-task Main set is drawn from a larger collection of 150 tasks.
The scores do not mean that every patch from a model with a particular result will be accepted or rejected in everyday development. They describe performance under FrontierCode’s task set, checks and agent setup. The reported pass rate is the share of attempts that cleared all blocking checks, while the overall score also reflects other grading criteria.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Maintainer Checks Change the Score
Many coding evaluations emphasize whether a program produces the expected output. FrontierCode adds checks aimed at problems that can make a pull request difficult or risky to merge: regressions, weak tests, oversized changes, edits outside the requested scope and code-review concerns. Its results therefore offer a different signal from a benchmark that counts only passing tests.
For software teams, the distinction matters because human review remains part of deciding whether a change is safe and maintainable. FrontierCode’s top result—57.7% of attempts passing all blocking checks—shows that success was not consistent even for the leading tested model. That is a benchmark finding, not a measurement of how often AI-generated code is accepted across real-world repositories.
Cost also varied widely in Artificial Analysis’s results. It listed GPT-6 Luna at $0.09 per task and Claude Opus 5.5 at $10.15. Opus scored 3.1 percentage points above GPT-6.1 Sol, whose reported cost was $0.81 per task. Those figures describe the tested setup and do not establish what a team would spend in production, where task mix and review requirements differ.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Tests Repository Work
Cognition developed FrontierCode with open-source maintainers, and announced it in June 2026. Version 1.1 followed in July with rules against agents looking up solutions online. According to the source report, more than 20 maintainers created tasks based on 36 established repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task. The tasks are private to reduce the chance that models encountered their solutions during training.
The benchmark checks patches across six areas: hidden tests for correctness; command checks for regressions and cleanliness; reverse tests that verify an agent’s tests fail against the original code; adaptive tests for complex behavior; limits on files changed and diff size; and an LLM-based code review against maintainers’ criteria. Artificial Analysis used up to 250 agent turns per attempt and allowed three attempts per task. Its review judge was GPT-6 Luna in medium mode.
As the source report quotes from Cognition’s announcement, Celery maintainer Tomer Nosrati described the benchmark this way: “Where others grade like a CI, FrontierCode grades like a tech lead.” The comparison captures the benchmark’s aim, but its score remains specific to the rules, tasks and judging system used.
““Where others grade like a CI, FrontierCode grades like a tech lead.””
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
Models and Real-World Acceptance Still Unmeasured
Several models had no FrontierCode result as of October 13, according to the source report, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published leaderboard does not cover every prominent system, and the report does not establish how those models would rank.
It is also unclear how closely benchmark performance predicts acceptance rates across live projects. FrontierCode uses a defined task collection, agent harness, turn limit and review judge; maintainers may apply different standards, and real repositories vary in complexity. The source material does not provide evidence that its rates directly translate into production outcomes. Nine of the 100 Main-set tasks were reportedly not solved by any model on any attempt, but the report does not identify those tasks or explain which factors made them unsolved.
More Results Could Shift the Rankings
The next development to watch is whether additional models receive FrontierCode evaluations, particularly those missing from the October 13 results. Results for those systems could change comparisons across providers and clarify whether the current spread holds as the leaderboard expands.
Further reporting would also help readers judge how representative benchmark outcomes are: task-level results, repeat runs and comparisons with real maintainer decisions could show where models fail and how closely FrontierCode scores track acceptance in practice. Until then, the published figures should be read as performance on this benchmark, not as a guarantee that an AI-generated patch is ready to merge.
Key Questions
What does FrontierCode measure?
It evaluates whether an AI-generated code change meets a range of repository and maintainer-oriented checks, including correctness, regression controls, test quality, change scope and an LLM-based review. Failing a blocking check gives the attempt a zero.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led Artificial Analysis’s results with a 52.2% score. It cleared all blocking checks in 57.7% of attempts, according to the report.
Does a 57.7% pass rate mean the model is accepted in 57.7% of real pull requests?
No. The figure is the share of attempts that cleared FrontierCode’s blocking checks under its benchmark setup. The source material does not establish that this rate predicts acceptance across real repositories.
Which model was the least expensive per task among the reported cost-frontier models?
Artificial Analysis listed GPT-6 Luna (max) at $0.09 per task with a 41.9% score. Costs and scores reflect the tested setup; they are not estimates of a team’s production spending.
Were all major AI models included?
No. The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13, 2026.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
