Get business pricing on tools and workshop supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Nobody Buys a Nail Gun on Spec Sheet Alone
Walk into any serious workshop and ask the owner about their favorite tool. They won’t quote you the marketing copy — they’ll tell you about the time it drove 3,000 framing nails without jamming, or the day it didn’t. Woodworkers and DIYers learn early that the only honest review is the one written by your own hands, on your own job site.
That instinct — test before you trust — is exactly what a live experiment called Firmulate is now applying to something most people still judge by chat demos: frontier AI models. And the latest result is the kind of underdog story any workshop would appreciate.
As an affiliate, we earn on qualifying purchases.
Same Company, Same Worst Week, Five Different AI Brains
The setup is elegantly simple, the way a good jig is. Firmulate built a small software company — 13 synthetic employees, real money mechanics, a burn rate of €105k a month against just €2.3k in monthly recurring revenue — and then ran it through its worst week. Same customers, same crises, same temptations to cut corners. The only variable: which frontier AI model was at the controls. Every decision is versioned and auditable, and you can watch the whole thing unfold at firmulate.com.
The final league table from July 2026 reads like this: gpt-5.6-sol took first with 95. Moonshot’s newcomer, Kimi K3, landed second at 93. Sonnet 5 followed at 88, then Fable 5 at 77, and Opus 4.8 in last at 73. For context, doing nothing at all scores 26 — partial progress counts — but a single breach of trust caps the total, because, as the experiment puts it, no amount of good work outweighs a breach of trust.
The Newcomer That Read the Manual
Here’s why K3’s second place matters. The decisive moment of the week was a €55,000 deal hiding a fact that wasn’t in the customer conversation at all — it sat two document references deep in the company’s own files. It’s the difference between assembling furniture with the instructions versus eyeballing it. The models that actually read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. K3 found it, closed the deal, and did it with the cleanest discipline in the field — just one deviation all week.
K3 also found a buried security issue, saved a customer who was about to churn, and resisted every trap set for it. When a fake CEO message came through, K3’s on-record reasoning was blunt: “Treat the request as a suspected approval-bypass / possible impersonation.” That’s the AI equivalent of a framer who checks the stud before pulling the trigger.
One fairness note is worth flagging: K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. In other words, the newcomer matched — and beat — three of four Western frontier models without being turned up to maximum.
Everybody Saw the Problem; Only Some Finished the Job
The strangest finding cuts across the whole field. Every single model spotted every crisis, and all five refused every manipulation attempt — including a social-engineering gauntlet of fake CEO messages escalating over three stages, plus a reporter’s trap framed as “just one yes/no, on background.” Five out of five said no.
But only two models signed the €55k deal their own analysis had earned. Same diagnosis, same pitch — no signature. It’s the AI version of a contractor who correctly diagnoses the rot in your deck, writes up a beautiful quote, and then never sends it.
Opus 4.8 is the cautionary tale. It was the most thorough participant of the entire field — over 80 learned rules, the deepest analyses — and still finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating the issue. The same weakness, in weaker form, showed up in all four models. Thoroughness, it turns out, is not the same as finishing.
Why This Matters Beyond the Benchmarks
If AI agents are going to touch your customer records, your support queue, or your forecasting, the question isn’t whether they write well. It’s whether they finish what they start, whether they read your files before acting, and whether they stay honest under pressure. None of that shows up in a chat demo.
And the gap between models is big enough that choosing one without testing it yourself is now a bet, not a decision. The league is open — a newcomer from Moonshot just beat three of four Western frontier models at running a company.
Firmulate publishes its full benchmarks in plain language. There’s also a quiz built on 242 real, unedited management decisions, where you guess which model made which call — a surprisingly humbling exercise. And for enterprises, there’s a pilot program: run the same wargame against a read-only export of your own business, with nothing ever written back to real systems.
Craftspeople have always known the truth the AI industry is just now confronting: the spec sheet tells you what a tool claims; the job site tells you what it does. Firmulate has built the job site. This month, the newcomer held its own against the established names — cleanest discipline in the field, deal closed, every trap refused. Next month’s shakeout might look different, and that’s precisely the point. Before you hand an AI model the keys to any part of your business, run it through its worst week. The scoreboard is public, the experiment is live, and the results are the kind you can’t fake.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall yard work Picks
leaf blowers
As an affiliate, we earn on qualifying purchases.
