You Have Already Run This Experiment

Think back to the last time you caught one out

You asked an AI something. It answered immediately, fluently, and with complete confidence. And on one occasion, at least, it told you something you happened to know was wrong.

Not obviously wrong. Wrong in a way that would have sailed past almost anybody. Wrong about a detail in your own field, or your own town, or the thing you have been doing for twenty years.

You caught it. Most people file that away as a funny story about the technology. It is actually a result, and the interesting part is not about the machine. It is about you.

Two groups, one tool

Researchers gave students an AI assistant to research a scientific question, and compared the reasoning they produced against what they produced without it.

Both groups were given the same task. Only one group had been trained in the subject.

Stadler, Bannert and Sailer, 2024, with a 2026 preprint adding domain expertise as the moderating variable. The follow-up is a preprint and has not yet been peer reviewed. The study reports which way each group moved rather than the size of the effect, so neither does this.

And it felt easier

The earlier study in the same line of work found something more uncomfortable than a drop in quality. The students using the assistant reported a lower mental workload while producing worse reasoning.

It felt easier at exactly the moment it was going wrong. Which is why nobody catches this from the inside.

So what was the one group carrying?

Not more intelligence. Not more effort. They already knew the subject well enough to argue with what came back. They could tell which claims to be suspicious of, which numbers smelled wrong, what had quietly been left out.

That capacity has a name. It is judgement, and it is built the only way it can be built: by making calls with something riding on them, and then finding out. The wrong ones show you where you personally tend to go wrong. The right ones tell you what your judgement is actually worth. Neither lesson arrives if nothing was at stake, which is why it takes a career and never quite finishes.

The debugging at eleven at night. The hundredth presentation of the same condition. The case that collapsed on the one witness you had not prepared.

Nothing a model does costs it anything

A model has read every account of being right and every account of being wrong that anyone has written down. That is not a small thing. It is why the answers are as good as they are.

What it has not done is been through any of it. When it is wrong, nothing happens to it. No consequence arrives, so nothing in it updates.

That is not a complaint about the technology. It is a description of what it is. And it has one straightforward implication.

The judging has to stay with people.

Which would be fine, except for two things

Nobody can see whether it happened. Two people review the same document and reach the same conclusion. One thought about it. The other agreed with it. Every record of those two decisions is identical, and nothing any organisation currently inspects can tell them apart. Not because anyone hid anything. Because there is no field for it.

A great deal of money assumes the problem solves itself. Hundreds of billions of dollars are committed to an argument whose first step is that this gets handled by scaling. If it does not, the cost does not stay with the people who made the bet. It arrives in retirement accounts nobody chose the contents of, in towns that borrowed against the building, and in jobs that are quietly not refilled.

So what would you build instead?

Here is the distinction that changes the question. A tool that replaces you has to do the whole job. Every part of it, at least as well as you did, or what you end up with is worse than what you had.

A tool that amplifies you only has to do one part: the part where you were actually stuck. The rest stays with you, because you are still there doing it.

Six questions

Everything above is asserted. None of it is made here. Each claim gets made properly in one of six short pieces, with the evidence, the sources, and the objections that have not been answered.

1. Why Does It All Feel Like Too Much? Language got cheap to make. Attention did not. This has happened twice before and both times it ended badly.
2. Why Can't You Just Be Told? The study above, and why judgement cannot be transferred in text.
3. What Holds a Claim Up? Four checks for whether a claim can bear weight, run against the Federal Reserve in 2006.
4. How Would You Know? The two reviewers, and why no inspection anybody currently runs can separate them.
5. Who Pays If It's Wrong? The four step argument carrying the money, and where the cost lands if step one fails.
6. What Would You Build Instead? The constructive half: five principles, four commitments, and a working prototype.

Each one is fifteen to twenty sheets and stands entirely on its own. They are ordered so that each ends by raising the question the next one is called, but you can start anywhere. Nothing in them requires a technical background.

The slides for this introduction: PDF · PPTX

One question to take away

Whatever you do next, this is the one worth asking about any tool you are using seriously:

After a year of this, what can I do that I could not do before?

If the honest answer is "produce more of what I used to produce", the tool widened your throughput. If it is "notice things I used to miss", it widened you.

Those are different products. Nearly everything on the market is sold on the first. That is a choice somebody made, not a fact about the technology.