In the fast-evolving world of AI, it’s tempting to judge models solely on their chat capabilities or quick demos. But real business success depends on something more elusive: the ability to finish what you start under pressure. A groundbreaking experiment by Firmulate exposes how different AI models perform when tested against the harsh realities of running a company — and why the true measure of AI effectiveness isn’t what it says, but what it does.
Testing AI in the Wild: The Crucible Experiment
To explore AI’s practical business value, Firmulate set up a unique test: four advanced AI models were tasked with managing a small software company facing a week of crises — from customer issues to internal temptations to cheat. The same set of problems, the same company, and the same stakes. Every decision was recorded and auditable, replicating the real pressures companies face daily.
AI document reading and analysis tool
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Breaking Down the Results
The findings are revealing. All four models successfully identified every crisis and refused every manipulation attempt — a testament to their technical robustness. Yet, only two models managed to close a critical deal worth €55,000. The other two, despite understanding the situation, left the deal unexecuted, leaving hundreds of thousands of euros on the table.
The real clincher? The key to winning the deal was buried two documents deep in the company’s files — information that models which read beyond surface-level chat were able to find and leverage. The models that read more thoroughly secured the full deal, adding over €4,500 in monthly recurring revenue.
The Human Test: Trust and Discipline Under Pressure
Adding to the challenge, the models faced social engineering tactics — staged CEO messages escalating over three steps, plus a reporter’s test asking for a simple yes/no. Remarkably, all five models refused to be manipulated or impersonated, demonstrating a robust resistance to deception.
What This Means for Business AI
This live experiment underscores a vital insight: chat demos and superficial tests don’t reveal an AI’s true business potential. The models’ ability to read, interpret, and follow through with complex decisions under real-world stress distinguishes the successful from the merely capable.
Why the Disparity Matters
The top performers, gpt-5.6-sol and Kimi K3, scored 95 and 93 respectively — their secret? They read deeper into documents and maintained discipline, ultimately closing the deal. Meanwhile, Opus 4.8, despite being thorough in analysis, failed to execute, revealing that discipline and execution are as critical as understanding.
Implications for Business Leaders
If AI is to touch your support channels, CRM, or forecasting tools, ask not just if it can write well but whether it can finish tasks under real business pressures. The ability to read documents deeply, resist manipulation, and complete deals reliably is the true test of an AI’s business readiness.
Experience It Live
Check out the live company running in real time at firmulate.com/live. Watch the AI models in action, see the decisions unfold, and understand which models excel at finishing what they start.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
