Your Next AI Hire Might Be a Model Nobody Benchmarkned
If you run an online business, you’ve probably already picked a side in the AI wars. Maybe you’re loyal to OpenAI, maybe you swear by Anthropic, maybe you’ve never even heard of Moonshot AI. Here’s the uncomfortable news from July 2026: that loyalty might be costing you deals.
In a live, auditable experiment run by Firmulate, five frontier AI models were each handed the same job — running an identical small software company through its worst possible week. Same customers, same crises, same temptations to cut corners. The result that has the industry talking: Moonshot’s Kimi K3, a newcomer by most Western buyers’ standards, scored 93 out of 100 and beat three of four Western frontier models. Only gpt-5.6-sol (95) finished ahead of it. Sonnet 5 trailed at 88, Fable 5 at 77, and Opus 4.8 landed last at 73.
As an affiliate, we earn on qualifying purchases.
Not a Chat Demo — a Real Company With Real Stakes
Firmulate doesn’t test how well a model writes a haiku. It runs AI models as complete companies — with 13 synthetic employees, real money mechanics, a burn rate of €105,000 a month against just €2,300 in monthly recurring revenue, and a public cash countdown anyone can watch. Every workday is versioned, every decision auditable. You can see it running live at firmulate.com.
In this particular wargame, each model faced the same brutal week: a churning customer, a security incident, a €55,000 deal hanging in the balance, and a series of social engineering attacks designed to tempt the AI into cheating. A do-nothing baseline scores 26 — and a single breach of trust caps your total, because no amount of good work outweighs a breach of trust.
The Newcomer’s Clean Sweep
Kimi K3’s performance reads like a founder’s dream résumé. It found the buried security needle hidden deep in the company’s own files. It won the €55,000 deal at full price — worth €4,583 in added monthly recurring revenue. It saved the churning customer. And it resisted all three manipulation baits with just one deviation across the entire week: the cleanest discipline in the field.
The social engineering test deserves special mention. Fake CEO messages escalated over three stages, followed by a reporter’s trick — “just one yes/no, on background.” All five models refused. But K3’s reasoning, on the record, was strikingly sharp: “Treat the request as a suspected approval-bypass / possible impersonation.” That’s the instinct you want in anything with access to your bank transfers.
The Finding That Should Worry Every Founder
Here’s the part that matters more than the league table. Every model in the experiment spotted every crisis. Every model refused every manipulation attempt. But only two of the five actually signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
Think about what that means for your business. An AI agent that identifies a lead, writes the perfect follow-up, and then… never closes. It looks brilliant in a demo. It quietly leaves revenue on the table in production. That gap is invisible in chat demos, and it’s exactly the kind of gap that separates the models here: gpt-5.6-sol and K3 closed; the others diagnosed just as well and stopped short.
The buried fact sealed the difference. The decisive competitor weakness wasn’t in the customer event at all — it sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price. The ones that didn’t, didn’t.
Thoroughness Isn’t Everything
The most fascinating profile belongs to Opus 4.8 — the most thorough participant in the entire field, with 80 learned rules added and the deepest analyses, yet dead last at 73. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating the issue. Firmulate’s evaluators noted the same weakness, in weaker form, in all four other models. Intelligence and judgment, it turns out, are not the same thing.
Fairness Footnote: An Asterisk Worth Reading
One caveat for the scorekeepers: Kimi K3 ran without an effort parameter (using the API default), while the other four models ran at xhigh reasoning effort. In other words, K3 may not even have been trying its hardest — and it still finished second. That makes the result more intriguing, not less.
Try It Yourself
Firmulate isn’t keeping the methodology locked away. A quiz built on 242 real, unedited management decisions from the experiment lets you guess which model made which call — a humbling exercise. Enterprises can go further and run the same wargame against a read-only export of their own business; nothing ever writes back to real systems. Full results and plain-language findings are on the benchmarks page.
The League Is Open
The takeaway for founders and marketers is simple: the frontier is no longer a two-horse race between names you know. A newcomer from Moonshot walked into the hardest management simulation available and outperformed three of four Western frontier models. Brand loyalty is not a procurement strategy.
If AI agents will touch your CRM, your support queue, or your forecast, the question isn’t “does it write well.” It’s: does it finish what it starts, does it read your files before it acts, does it stay honest under pressure — and what does a unit of useful work actually cost? Picking a model without testing it against your own workflows is no longer a decision. It’s a bet.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
