Architecture & AttentionPlain-language edition
Why Can't You Just Be Told?
Somebody once gave you exactly the right advice and you did not use it. You understood every word. You still had to get it wrong yourself before it meant anything. That gap is the subject here, and it is the reason a machine that has read everything still cannot do what a senior practitioner does.
There is more language in the world than anyone can attend to. The obvious fix, an AI that reads it all for you and tells you what is true, does not work: the judgement that would take requires having been wrong at your own expense, and nothing a model does costs it anything. So the judging stays with people, and almost nothing we currently inspect can tell whether a person actually judged or simply agreed.
- Why Does It All Feel Like Too Much?
- Why Can't You Just Be Told?
- What Holds a Claim Up?
- How Would You Know?
- Who Pays If It's Wrong?
- What Would You Build Instead?
Six short pieces, each drawn from a working paper. Each one stands on its own; this is the second. They are ordered so that every piece ends by raising the question the next one is called. The full map is in Six Questions, In Order.
The argument
Advice you did not use
Think of something you were told early on that turned out to be completely true, and that you ignored, or nodded at, or filed away. Then think of the day it finally landed, which was almost certainly the day it cost you something.
Nothing about the sentence changed. You did. And the thing that changed you was not hearing it again.
Four different things, not one pile
There is an old and useful ladder here. Data is a reading: 38.2 degrees, 160 over 95, four millimetres of deflection. Information is that reading put in context. Knowledge is the pattern across many such readings, and what to do about it.
Then there is wisdom, and most people treat it as simply more knowledge. It is not. It is a different kind of thing, arrived at a different way.
The last step is the odd one
The first three can be handed over. I can give you my data. I can give you my information. With more effort I can teach you what I know.
I cannot give you what twenty years of being wrong did to me. Not because it is mystical, but because the mechanism that produced it ran on me, and ran over time, and it needed me to be the one carrying the cost.
This is the whole claim: wisdom cannot be handed over. It has to be travelled through.
The loop that produces it
What you pay attention to decides what you experience. What you experience, especially where it hurts, changes your judgement. And changed judgement changes what you notice next time, which is where the loop closes.
What makes a consequence real
Not every bad outcome teaches anything. Three things have to be true, and if any one of them is missing the loop does not turn.
- 01You cannot undo it. A result you can roll back and try again is a simulation of a consequence. Useful for learning a game. Not the same thing.
- 02It lands on you. Not on the next shift, not on a user you never meet, not on an abstraction called the business. On the person who made the call.
- 03It changes what you are. You are not the same practitioner afterwards. The next time a case looks textbook-clear, something in you hesitates, and you could not switch that off if you wanted to.
It met every specification
In July 1940 a suspension bridge opened across the Tacoma Narrows in Washington State. It had been designed to the standards of the day and it met them. Four months later, in a wind of around forty miles an hour, well under what it had been built to withstand, it twisted itself apart and fell into the water.
The calculations were not wrong. They were calculations of the things anybody knew to calculate. What brought the bridge down was a behaviour that was not in them, and the profession only learned to look for it by watching this happen.
That is what experience supplies and knowledge cannot: not better answers to the questions you asked, but the questions you did not know were there.
Watching, doing, imagining
There are three ways of getting at why something happens, and they are not interchangeable. Watching gives you what tends to go with what. Doing means changing something on purpose and seeing what follows. Imagining means asking what would have happened if you had not.
But the helicopter really crashed
This is the strongest objection and it deserves the strongest version. A program taught itself to play Go by playing millions of games against itself, and found moves nobody had thought of. A research helicopter taught itself aerobatics, and crashed real aircraft doing it. That is not reading about intervention. That is intervention.
Conceded. And both of them happened inside a world where every outcome could be scored.
A surgeon does not get a score. Neither does a policy, a hire, or a bridge. The result turns up later, partly, mixed in with everything else, and somebody has to carry it anyway.
The failure is always the same shape
Legal briefs citing cases that do not exist. A load calculation produced just as fluently whether or not the relevant figure was ever supplied. These are not lies, and they are not random errors. They are all the same thing: no sense of where its own knowledge stops.
Wisdom-shaped, without the history
A model trained on the writing of people who did traverse the loop will produce output that looks like their judgement. Measured hedge for measured hedge, it may be indistinguishable.
What is absent is not the words. It is everything that made those words trustworthy in the person who first wrote them: the cases that went badly, the hesitation installed by the hundredth presentation, the knowledge of which way they personally tend to be wrong.
What it costs, measured
Two groups, one tool
Researchers gave students an AI assistant to research a scientific question, and compared the reasoning they produced against what they produced without it. Both groups were given the same task. Only one group had been trained in the subject.
Same task, same tool, opposite direction. The only thing that differed was what each group already knew.
Stadler, Bannert and Sailer, 2024, with a 2026 preprint adding domain expertise as the moderating variable. The follow-up is a preprint and has not yet been peer reviewed. The study reports which way each group moved rather than the size of the effect, so neither does this.
If you have ever caught an AI being confidently wrong about something you happen to know well, you have already run this experiment on yourself. The reason you caught it is the whole finding.
What made the difference
The earlier study in the same line of work found something more uncomfortable than a drop in quality. The students using the assistant reported a lower mental workload while producing worse reasoning. It felt easier at exactly the moment it was going wrong, which is why nobody catches it from the inside.
The students who already had judgement could interrogate what came back. They knew which claims to be suspicious of, which numbers smelled wrong, what the output had quietly left out. The others could only accept or reject it whole.
The tool does not supply judgement. It multiplies whatever judgement you walked in with, including none.
Where that judgement came from
Nobody is born able to interrogate an output. It comes from a specific and rather unglamorous apprenticeship: the debugging at eleven at night, the hundredth presentation of the same condition, the case that collapsed on one witness you had not prepared.
Why nobody notices in time
Cutting the junior work looks like a clean win for years. Output goes up. Costs go down. Nothing visibly breaks, because the senior people who are carrying the judgement were formed under the old arrangement and are still there.
The bill arrives when they retire, and by then the decision that caused it is ten or fifteen years old and nobody can point at it.
Limits, and what to do with it
What this does not claim
Short versions tend to overstate. Here is what this one is not saying.
- ✕AI is not useful. It made the medical students measurably better. The argument is about what it cannot supply, not about whether it helps.
- ✕People are reliably wise. Obviously not. The claim is narrower: human judgement can be held to account, can be improved by being wrong, and is carried by somebody who pays.
- ✕This is permanent. The paper concedes that systems which act in the world move the gap rather than close it. And if somebody has a better account of what wisdom is, the author would rather hear it than defend this one.
- ✕Here is the experiment that proves it. There isn't one. Wisdom does not show up cleanly in a controlled trial, and the paper says so rather than dressing an argument as a finding.
Three things that follow
- Only use it where you could catch it being wrong.If you cannot interrogate the answer, you are not using a tool, you are taking dictation. The medical students could push back. That is the whole difference.
- Keep the crucible in the work.If every piece of slow, close-up, get-it-wrong-yourself work is removed from a junior role, that role stops producing seniors. That is a design decision, whether or not anybody makes it deliberately.
- Protect the pipeline.The cost of cutting it is real, delayed by a decade, and paid by people who were not in the room. It is the cheapest thing to cut and the slowest to rebuild.
Where this comes from
The argument here is the one made in a working paper called The Wisdom Gap. What the paper adds is the evidence, the sources and the objections, including the ones that have not been answered.
It is part of a series on what judgement is, why it has to be carried by people who bear the consequences of being wrong, and what that means for building with machines that bear none.
The idea behind this paper started in an exchange with Louise Vigeant, who writes Think Therefore AI. The argument took its shape there before it had a name.