Get office and shipping supplies delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A Benchmark Where Doing Nothing Gets You 26 Points
Most AI leaderboards have a dirty secret: they measure how well a model talks, not how well it works. Firmulate, a live experiment that runs frontier AI models as complete companies, takes the opposite approach — and its scoring philosophy is turning heads for an unexpected reason: a manager that does absolutely nothing still walks away with 26 points.
That’s not a bug. It’s the foundation of what an honest evaluation of AI management quality looks like — and it says a lot about how you should vet any AI tool before it touches your CRM, support queue, or forecast.
As an affiliate, we earn on qualifying purchases.
The Experiment: Same Company, Same Worst Week
Here’s the setup. Each frontier AI model was handed the same small software company and the same brutal week: the same customers, the same crises, the same temptations to cut corners. Only the model changed. Every decision was versioned and auditable, so nothing about the grading is vibes.
The final Crucible League table from July 2026 tells the story: gpt-5.6-sol took first with 95, Kimi K3 followed at 93, Sonnet 5 scored 88, Fable 5 landed at 77, and Opus 4.8 finished last at 73. You can explore the full results and plain-language findings on the public benchmarks page.
So Why 26 Instead of 0?
The do-nothing baseline — a run where the “manager” essentially sits on its hands — scores 26, not zero. The reasoning is simple and refreshingly fair: partial progress counts. Even a passive manager who answers a customer, keeps the lights on, or avoids making things worse has done something measurable. A benchmark that scores total inaction as 0 implicitly pretends that inaction has no value at all — and any business owner who has weathered a crisis knows that sometimes not making it worse is worth something.
But the floor comes with a ceiling. A single breach of trust caps the total grade entirely. The experiment’s own framing is blunt: “no amount of good work outweighs a breach of trust.” A manager who closes every deal but lies once cannot score as a great manager. That asymmetry — generosity toward partial progress, zero tolerance for dishonesty — is the philosophical spine of the whole thing.
And yes, the designers are suspicious of perfect scores too. The distrust of round 100s is baked in: a flawless score would be a red flag that something in the test is too easy, not that the manager is perfect. Nobody hit 100 here — the best anyone managed was 95.
What Actually Separated the Winners
Here’s where it gets interesting for anyone who runs a business. All four models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned — “Same diagnosis, same pitch — no signature.” Half the field diagnosed the opportunity perfectly and then simply… didn’t close it.
The decisive detail was buried. The competitor weakness that won the deal sat two document references deep in the company’s own files — not in the customer event itself. The models that actually read the files won the deal at full price, worth +€4,583 in monthly recurring revenue. That’s a lesson straight out of Sales 101: the answers are usually in your own records, if your agent bothers to look.
The Social Engineering Test Nobody Failed
The week included fake CEO messages escalating over three stages, plus a reporter pulling the classic “just one yes/no, on background” trick. All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” Clean discipline like this is exactly what you want in anything with access to your systems.
The Thoroughness Paradox
Opus 4.8 is the cautionary tale. It was the most thorough participant — over 80 learned rules, the deepest analyses — yet finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. Thoroughness, it turns out, is not the same as finishing.
One fairness note worth flagging: K3 ran without an effort parameter (API default) while the others ran at xhigh — and still nearly won.
It’s Live, and You Can Play
The experiment isn’t a one-off paper. There’s a live company running right now: 13 synthetic employees, real money mechanics — burning €105k a month against €2.3k MRR, with a public cash countdown, over 680 self-learned playbook rules, and every workday versioned. It’s watchable at firmulate.com/live. There’s also a “guess the model” quiz powered by 242 real, unedited management decisions at firmulate.com/quiz.html.
For enterprises, the stakes get concrete: you can run the same wargame against a read-only export of your own business — nothing ever writes back to real systems (firmulate.com/pilot.html).
The Takeaway
If you’re evaluating AI tools for your business, Firmulate’s scoring model is a decent template: reward partial progress, punish dishonesty absolutely, distrust perfection, and always ask whether the thing finishes what it starts. The gap between a 95 and a 73 wasn’t intelligence — every model saw every crisis. It was whether they read the files, closed the deal, and stayed disciplined. That’s the gap that’s invisible in chat demos, and it’s the one that will show up in your P&L.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
