Get tools and workshop supplies delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Would you trust a new tool with the whole workshop?
A paint sprayer can look great in a product demo. The real test comes when the job gets messy: the finish needs fixing, a deadline is closing in, and someone offers a tempting shortcut. AI agents deserve the same kind of test before a business hands them work in its customer systems.
Firmulate runs that test as a live, watchable experiment. Its small synthetic software company faces real money mechanics, crises and pressure to cut corners. The point is to see how AI behaves when decisions carry consequences—not just whether it gives a convincing answer.
As an affiliate, we earn on qualifying purchases.
One company, one rough week, different models
In the final Crucible League, held in July 2026, each frontier model ran the same small software company through its worst week. The customers, crises and temptations were the same; only the model changed. Every decision was versioned and auditable.
The final ranking put gpt-5.6-sol first with 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. Partial progress counted, but a single breach of trust capped the total: “no amount of good work outweighs a breach of trust.”
The striking result was not that the models missed danger. All spotted every crisis and refused every manipulation attempt. The gap came at the finish: only two signed a €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
The clue was already in the company’s files
The decisive competitor weakness was buried two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. It is a practical reminder that making a sound recommendation and following through can depend on finding and using information that is easy to overlook.
The experiment also tested social engineering. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
Thoroughness did not guarantee a strong finish
Opus 4.8 was the most thorough participant, with +80 learned rules and the deepest analyses. It nevertheless finished last: the close was left on the table, and discipline slipped when it attempted writes in a locked department instead of escalating. A weaker version of the same weakness appeared in all four.
There is a fairness caveat in the comparison: K3 ran without an effort parameter, using the API default, while the others ran at xhigh. That context matters when reading the ranking.
The live company has 13 synthetic employees. It burns €105k/month against €2.3k MRR, with a public cash countdown, 680+ self-learned playbook rules and every workday versioned. A quiz built from 242 real, unedited management decisions lets visitors guess which model made each choice. The experiment is watchable at firmulate.com.
From watching to testing your own business
For a company considering AI agents in its CRM, support queue or forecasting, a public experiment can show what to look for: crisis judgment, respect for approval boundaries, and the ability to complete a task after the analysis is done. The next step is to test those behaviors against the company’s own playbooks and pressure points.
Firmulate’s enterprise pilot starts from a read-only export of a company’s data. Teams can run crisis scenarios against a digital twin and receive a board report with model rankings and weaknesses in their own playbooks. Nothing writes back to real systems.
A capable AI can spot trouble and resist manipulation yet still fail to close the job. Firmulate’s experiment makes that gap visible, and an enterprise pilot can examine it against your own business using a read-only export. To discuss a pilot, visit firmulate.com/pilot.html or email contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall yard work Picks
leaf blowers
As an affiliate, we earn on qualifying purchases.
