Do you believe that no matter how your team uses AI, it makes them faster?
It would be reasonable if thought this to be true – I’m sure you’ve watched a deck get drafted in four minutes that would have taken half a day (thank you, Gamma!). And from there it’s a short walk to an assumption that feels obvious: more AI, applied more broadly, equals more speed.
Yes, but.
A 2025 study from METR, a Berkeley-based AI research group, tested that assumption with experienced software developers. These were not juniors. The developers using AI coding assistants were 19% slower than those working without them. And when asked, they estimated the AI had made them 20% faster.
A nearly 40-point gap between how productive they felt and how productive they were. Invisible to the people inside it. Something was eating the gains, and nobody could see it. And before you start telling me that the study was oh-so-2025, which I will rebut by pointing out that newer studies show the same signal, I want to explain instead by using my ex-friend, Bob.
Meet Bob
Yes, my ex-friend, really, but at one time my best friend. Owing to a mostly successful fight with substances, Bob became unreliable. Sometimes he’d show up for lunch, sometimes he wouldn’t. Sometimes he’d remember the favor he owed you, sometimes he’d act like the conversation never happened. When I finally broke up with him (yes, you can break up with a friend), Bob protested: “Yeah, but I’m not unreliable all the time.”
“Yes, and that’s worse! You’re unreliably unreliable!” I said. “I never know when to trust you.”
Your AI may seem like your new friend, but everybody's AI is a Bob.
If AI were wrong all the time, you’d never trust it, and you’d be fine. If it were right all the time, you’d trust it completely, and you’d also be fine. But it’s brilliant at some things, sloppy at others, and dangerously confident in a third category.
Bob is Costly in Subtle Ways
Think about what Bob was costing me: I didn’t just lose the things Bob failed to deliver. I lost all the time I spent trying to make him deliver: the follow-up calls, the second chances, the rearranged plans, the conversations where I tried to get Bob to be the Bob he was supposed to be. It told myself it was the cost of “maintaining a friendship” but it was really the cost of an unreliable one.
Your team is doing the same thing with AI right now. Reprompting, adjusting, checking output against their own instincts…rewording what the machine said until it sounds like what they meant. Iterating because the first three versions were just wrong enough in ways that took judgment to catch. That time doesn’t show up in anyone’s productivity tracking. That’s the cost of the AI’s unreliability.
AI's Unreliability: Where the Cost Comes From
Every word any AI produces is a bet. When the data behind that bet is deep and consistent, the bet is good. A chemical formula has one answer. Standing rib roast at 500°F (for how long?) has mostly the same answer. These are codified, deeply represented, or agreed upon. Here, the machine performs well, the output needs minimal checking, and the productivity gains are real. The judgment cost of using the tool is near zero.
Now ask “Should I put Splenda in my coffee?” The peer-reviewed toxicology is clear: it’s one of the most thoroughly studied food additives in history and it’s fine. But the volume of text on Splenda is dominated by health-anxiety blog posts and wellness influencers. The machine’s bet drifts toward the loudest answer rather than the rightest one.
The second zone is where your agency actually does its valuable judgment work. Diagnosing what the client actually wants, not what they wrote in the brief. Knowing that this particular CMO says yes to everything in the meeting and changes her mind by Friday. Recognizing that the strategy your team just recommended is the same one three competitors recommended last quarter. It’s not in the AI the training data, or what’s there is a composite of what other clients have ever said…a set of clichés.
I’m calling it the AI Judgment Tax. Every time your team uses AI in the zone where the training data is thin (or dubious), the output arrives fast and the tax follows: evaluating, correcting, reworking, chasing down whether the confident-sounding claim is actually true. And that’s the best case scenario – it assumes that people can see the judgment failure.
As Problem-solvers, We're a Bit Blind to Judgment Tax
You have a story like mine, I’m sure:
I spent an hour once trying to get an AI to say what I wanted to say. One hour. I knew it wasn’t right yet…I kept prompting, reprompting, adjusting, correcting. At the end of the hour, I had something close to what I would have written in twenty minutes if I’d just opened a blank document.
But I felt productive the whole time...all that activity felt like progress! That was the judgment tax in its purest personal form: an hour of paying it, disguised as an hour of working.
Judgment Tax also can be invisible in the moment because the output looks done. You’ll pay eventually in hours nobody tracks, against a productivity gain everybody already counted. The METR developers paid it and didn’t know. Your team is paying it right now.
Workday’s 2026 research put a number on it: 37 to 40 percent of the time AI supposedly saves gets consumed by reviewing, correcting, and verifying the output. You save an hour on a first draft and spend twenty-five minutes fact-checking it because you can’t trust the machine didn’t hallucinate a statistic or flatten a nuance that mattered.
How much time does your team spend evaluating, correcting, reworking, and second-guessing something that polished and confident yet incorrect AI output? Is it taking longer than starting from their own judgment, because they’re reverse-engineering the machine’s mistakes into something they could have just written on their own?
And AI’s (false) Confidence Also Blinds Us
If AI sounded uncertain when it was guessing and confident when it was right, we could just listen for the wobble. But the machine speaks with the same confidence whether it’s giving you the boiling point of water or interpreting your client’s emotional state.
AI has no internal signal that says “I don’t know this one.” It predicts the next word, then the next, and the rhetorical posture is set by the style of question you asked, not by whether the answer is true.
Karmarkar and Tormala showed that expressed certainty increases persuasion, whether or not the source has any expertise. They titled the paper “Believe Me, I Have No Idea What I’m Talking About.” That’s a literal description of your AI’s operating mode: it has no idea what it’s talking about, it expresses total certainty, and the certainty alone makes you more likely to accept the answer.
A Wharton study (Shaw and Nave, 1,372 participants) found that people accepted faulty AI reasoning 73% of the time when it was delivered confidently. We’re wired that way – your people are – we evolved to find the village elder and believe them. The machine knows how to talk like the village elder.
Judgment Tax Charges Compound Interest
Here’s an examples of where it gets expensive, why this is not a one-time or individual productivity issue.
AI-generated meeting summaries are everywhere now. The Zoom call ends, a tidy recap arrives in everyone’s inbox, action items bulleted, decisions captured (thanks, Granola!). It feels like a productivity win.
The meeting summary is an interpretation. The machine decided which exchanges mattered, which silences meant assent and which meant resistance. It will treat a polite “we should look into that” as a commitment when it was actually a deflection. It will miss the fact that the CMO went quiet for the last fifteen minutes because she was furious, not because she agreed.
That’s a local tax, because maybe everyone knows better. But what if the summaries get stored in shared drives, fed into knowledge bases, piped into the next AI tool your firm uses (yes, many of you are doing this). Six months later, when someone asks the firm’s AI assistant what this client wanted last quarter, the answer comes from those summaries. Not from the meetings. From the AI’s prior approximations of the meetings. Splenda. Bob.
The half-truth went unquestioned long enough to harden into fact. Now it’s in the corpus, shaping the next round of bets. And the next person who relies on that corpus will get a confident answer built on a previous confident answer that was built on a misread of a meeting six months ago.
And you've just built a knowledge base that requires more judgment to use than its knowledge justifies. The tax isn’t just on individual outputs anymore. It’s baked into the infrastructure, compounding with every generation of AI-touched content, and the system never tells you it’s happening. Each round costs your team more time, not less, to get to something they can trust.
Where the Tax is Low, Let AI Think. Where it’s High, Stop.
The METR developers were right about AI, as far as it went. Where they got it wrong was where AI helps.
In the zone where the training data is deep and the answers are codified, the judgment tax is near zero and AI earned its speed. In the zone where the data is thin and the work requires judgment, the tax climbs until it swallows the gain. The tool cost them time they couldn’t see, because it kept sounding like it was helping.
Your firm is the same split. Some of your work sits in the zone where the tax is near zero. Some sits in the zone where every piece of output carries a tax your team is paying in hours nobody tracks. The line between those zones isn’t where you think it is, and the tool will never tell you when you’ve crossed it.
The answer is to stop letting AI do the thinking in the zone where its truth drops. Use it there as assembly: formatting, retrieval, first drafts you plan to rewrite. Keep the judgment with the human. The firms slowed by the Judgment Tax are the ones who hand the machine the same level of trust across both zones, because the machine invited them to.
The hard question is where that line actually falls in your firm, across your clients, through your specific work. And that’s a question the tool can’t help you answer.
This is part of The Talking Machine, a series on what AI actually does to organizations that sell thinking for a living.
I run a workshop where your leadership team maps your actual client work against this framework and finds where the line is. Three hours, fixed price, real clients. Details and booking at bettercompany.co.
Jack Skeels is the author of Unmanaged and the forthcoming When the Machine Talks. More at jackskeels.com. Subscribe here for more like this.
