I Built the Frame. I’m Still Inside It.
I can recite every way LLMs fool people. Here is why my expertise makes it worse, not better, and why I am the most exposed person in this essay.
The Inoculation Feeling
I finished last week’s piece feeling better than I should have.
I had named the asymmetry. I had laid out how fluency waves itself through in the direction you already lean, how the alarm fails silently, how you cannot catch it from the inside. I had even confessed my own accent, gone first, and implicated myself before turning it on anyone else. And when I hit publish, I felt a quiet click I want to describe precisely, because it is the whole subject of this piece. It was the feeling of having handled the thing. Of having looked the bias in the face, named it out loud, and therefore, somehow, settled my account with it.
That click is the failure. Not a risk of the failure. The failure itself, arriving on schedule, in the exact moment I felt safest from it.
Because nothing in the act of naming a bias removes it. I wrote a whole piece about a mechanism that switches off your skepticism in the direction you lean, and the writing left my own skepticism exactly where it was, plus one new and dangerous asset: the sense that I had dealt with it. I had not dealt with it. I had described it. Those are different things, and the gap between them is where I now live, more exposed than the reader who never thought about any of this at all.
That is the uncomfortable thought I left at the door last week, and this is me walking through it. The people best equipped to explain how the counterfeit works are not the safest from it. They may be the easiest marks in the room. And I have to start, again, by counting myself first, because I have less standing than anyone to write this sentence and more reason than anyone to need it.
The Finding With My Name On It
There is a study I have to tell you about, and I have to tell you about it in the worst possible way, which is by pointing it at myself.
A team at Aalto University, led by Robin Welsch, sat people down to reason through problems with ChatGPT and then asked them how well they thought they had done. The ordinary expectation is the Dunning-Kruger pattern: the least able overestimate themselves the most, and competence comes with a more honest read on your own limits. With AI in the room, that pattern broke. Everyone overrated themselves. But the part that put my name on it is this: the people who scored highest on AI literacy, the ones who understood the systems best, were not better calibrated. They were worse. The more someone knew about how these tools work, the more confidently wrong they were about their own performance.
Sit with the shape of that, because it is not a small finding dressed up. It is the precise inversion of the thing we tell ourselves. We assume understanding the tool buys us a clearer view of our own footing with it. The study says the opposite. Understanding bought a worse footing, or at least a worse sense of where the footing was.
And I am the case study. I write about this. I can name the failure modes, recite the limitations, and explain to a passenger in the back of my car why the fluent answer on the screen is not the same as a true one. By the only measure the study cared about, I am deep in the population that calibrated worst. Not despite the knowledge. With it. The expertise is not my exemption from the finding. It is my qualification for it.
I want to be careful not to oversell what they measured. They tested how well people judged their own reasoning while using the tool, not how well they graded the tool’s output, and the wider literature is honest that the miscalibration does not always run toward overconfidence. And this is a correlation, not a causal mechanism. The literate were more confident and less accurate about themselves; the study does not tell me which one drove the other, only that they traveled together. But the association held through a randomized replication, and it ran the wrong way, the opposite of the bet I would have placed. That is enough to unsettle the thing I assumed, which is all I am asking it to do. The heavier claim comes next, and it comes from a study built to carry it.
Why Knowing Makes It Worse
There are two kinds of knowing in this story, and almost everything depends on not confusing them. Call them knowledge of the subject and knowledge about the tool. One is what the doctor has about medicine. The other is what I have about how LLMs fool people. The whole argument lives in the distance between them, so I will keep the two apart from here on.
Louise Vigeant walked me to the distinction through a pair of studies I have not stopped thinking about since. The first, by Stadler and colleagues, is one of the lynchpins behind the now-familiar worry that AI makes thinking easier and worse at the same time. Students were assigned at random to research a question with ChatGPT or with Google. The ChatGPT group reported a lighter cognitive load and produced weaker reasoning. Easier in the hand, worse on the page.
Then the same researchers added one variable, and it changed the shape of everything. The follow-up is still a preprint, and worth holding as such, but it is built as an experiment rather than a snapshot: assign the tool, vary the expertise, watch what moves. Medical students and social science students both researched the safety of a medical question, with the tool handed out at random. The load dropped for everyone, expert and non-expert alike. But the quality split. The social science students, out of their depth on a medical question, reasoned worse with the AI, exactly as the first study predicted. The medical students, on home ground, reasoned better with it than without. Because the tool was assigned rather than chosen, the study earns a verb the last one could not: here, the expertise did the work, and you can watch it do it.
Stop on the part that should unsettle you, because it is the accent piece made literal. The drop in load was identical for both groups. The tool felt just as helpful in the hand it harmed as in the hand it helped. The feeling of being assisted carried no information about whether you were being assisted. Smooth is smooth whether or not it was earned, and the person inside the smoothness cannot tell from the smoothness. The fluency is not the thing hurting them. It is the residue of a loop that used to back it and now does not, the tell that the loop has been severed. Blaming the smoothness is like blaming the dial for the reading. What changed is behind the dial.
So knowledge of the subject genuinely protects you. The medical student caught what the social science student could not, because she had paid for that knowledge in the only currency that counts, the long accrual of being wrong about medicine in ways that cost her until the right shape of it stuck. That is domain expertise, and it is real armor.
Now set the other study beside it. Welsch’s people were not measured on their command of any subject. They were measured on their knowledge about the tool and on how well they understood the LLMs themselves. And that knowing did not protect them. The more fluent they were in how the machine works, the worse their read of their own performance got. I have to keep the studies honest about their own weight. Stadler assigned the tool and can say expertise did the protecting. Welsch can only say the literacy and the miscalibration rode together. One is a mechanism you can watch. The other is a pairing you cannot yet explain. But the pairing runs the wrong way, and that is enough.
There it is, in the gap between the two. Knowledge of the subject is armor. Knowledge about the tool is not. And they feel exactly the same from the inside, which is why they are so easy to swap without noticing. Both wear the clothing of competence. Both produce the calm of a person who knows what they are doing. But only one of them was paid for in the domain, and only the paid-for one catches the error in front of you. The other just persuades you that you would have.
I have to say which one I carry. My expertise is knowledge about the tool. I can name how the counterfeit works, recite the limitations, and explain the asymmetry to a passenger at a red light. The moment I point this tool at any question outside the few things I have actually paid to know, I am the social science student, holding a lighter load and mistaking it for a steadier hand. And on the specific subject of how AI fools people, the one place I am supposed to be the expert, my fluency is not the cure. It is the accelerant.
The Thing Knowledge Isn’t
I want to name the distinction plainly now, because it has a name, and because the name turns on me last.
What the medical student has isn’t more knowledge than the social scientist. It is a different kind. She has been wrong about medicine, repeatedly, in settings where being wrong had a cost, and the cost did the teaching. What she carries into the exchange with the machine is not a stock of facts. It is calibration, the residue of consequence, the thing that lets her feel the wrongness of a plausible answer before she can fully say why. That is not knowledge. That is closer to judgment, and judgment is built one way only, by repeated contact with consequence until the contact changes how you see.
I should be precise about that contact, because it is not always your own bill that gets paid. The surgeon is shaped by the patient she lost and by the death she watched a mentor preside over. The pilot is calibrated in the simulator by crashing a hundred times, with the only casualty being the clock. Consequences can be carried vicariously, watched, drilled, and inherited from the people who paid in full before you. What it cannot be is skipped. You can borrow the toll from someone who paid it, but the toll has to have been paid by someone, in contact with a real cost, or the calibration never forms. The downloadable version, the fact without the contact, is the one that does not take.
Which is why knowing the failure mode is knowledge, and being protected from it would take the other thing. The other thing cannot be read, summarized, or absorbed from a study. It has to be paid for, by you or by someone else whose paying you genuinely sat with. This is the architecture of the Wisdom Gap, the argument I have been building for a year: that the distance between what a system knows and what it can wisely do is exactly the distance of consequence carried forward, and that no amount of the first ever closes into the second on its own. Knowledge is downloadable. Wisdom is not, because wisdom is just knowledge that has been through the loop and paid the toll.
And here is where the floor gives way under me.
I have the frame. I built the frame. I can explain the Wisdom Gap to you with the unhurried fluency of a man who has spent a year inside it, and that fluency is, by its own definition, knowledge about a failure mode. It is the kind Welsch’s people had, the kind that did not protect. My framework for why knowledge is not wisdom is itself only knowledge. Holding it has taught me to recite the gap with great precision. It has not carried me through the loop that would close it. If anything, it has handed me the most disarming exemption story I own, because a man who has theorized consequence-bearing for a year feels entitled to assume he has banked some, and theorizing the toll is not the same as paying it.
So the frame does not stand outside me, marking the distance for everyone else to measure. I am inside it, at the widest part of the gap, holding the most elaborate possible account of a protection I have not actually earned.
The Part You Can Use
This is the part that is supposed to be useful, so here is the use, though it is harder to hold than last week’s. Last week’s tool was built for the layman in all of us: notice where the checking feels unnecessary, and check there. This week’s is for the expert in all of us, and the expert has a subtler failure. He does not skip the check. He performs it beautifully and clears himself.
So the tell has moved. Last time it was reluctance, the hour you cannot be bothered to spend. This time, it is fluency itself, the quality of your own explanation for why you are fine. But I have to be careful here, because “your good explanation is the symptom” is the kind of claim that swallows everything and proves nothing. If every account of why you are right counts as evidence that you are wrong, the test is rigged and useless, and I have built a machine that can never be switched off.
So here is the line that keeps it honest, the one thing that separates a real check from a recited exemption. A real check goes and touches something outside your own head. It fetches a fact, a source, a number that could have come back the other way and sunk you. A recited exemption only touches your own framework. It explains, fluently, using the theory you already hold, why you need not look. The test is not whether you have a story. Experts have stories, and some of them are true. The test is whether the story sent you out to the world or only deeper into yourself. One can be wrong and find out. The other can only ever be confirmed.
That is the question I now make myself sit with. Not “what would I need to believe for this to be wrong,” but: did I go and check, or did I recite why I did not have to? The first move can come back and hurt me. The second never can, which is exactly what should worry me about it. When I catch myself answering a doubt with my own theory of doubt, naming the bias instead of testing the claim, that is the framework purring, the loop running open, and calling itself rigor.
But notice the flaw in leaving it there, as a question I sit with. A question I raise only when I feel the need is already lost, because the feeling is exactly what goes quiet on the things I believe. Hand the trigger to the mood, and the mood will protect them. So the check cannot wait to be summoned. It has to run cold, on a schedule I do not set in the moment: every clean, agreeable landing, not just the ones that snag. And it has to be mechanical enough to survive my own fluency. Three dull moves, run whether or not I think I need them. Can I verify this against something outside my own head? What would have to be true for it to be wrong? Can I say it back plainly with the smoothness stripped off? The dullness is the whole virtue. A check I can talk my way out of was never a check. It was a mood in the costume of one.
I will be honest about the ceiling, because pretending there isn’t one would be the exact performance I am warning against. The instrument I use to catch myself reciting is more articulacy, the same compromised tool, one level up. There is no clean floor here, no final move that lands me outside the gap looking in. The most I can do is treat my own articulacy as a suspect rather than a witness, and let the smoothness of my self-defense be the thing that triggers the search rather than ends it. And I should not mistake the discomfort that produces for the destination. Sitting in doubt is not the same as being right; a man can be anxiously, elaborately wrong. The doubt is not the wisdom. It is only the friction that slows the nod long enough to go and look.
Where the Frame Could Break
There is a fair objection to everything I have just built, and I would rather raise it than simply not mention it. A theory that treats every confident claim of wisdom as proof of its absence has armored itself against all possible evidence. If “I have grown wiser” only ever counts as a symptom, then I have not made an argument. I have built a trap with no door, and a trap with no door catches nothing, because there was never a way to be wrong inside it.
So let me say what would change my mind, because a frame that cannot name its own failure condition is just the counterfeit wearing my favorite clothes. The Wisdom Gap predicts one thing testably: that knowledge about a failure mode, held without ever having paid for the failure, does not protect you in the moment the way consequence-borne calibration does. If you sat people down, taught them a failure mode cold, no contact and no cost, and they then caught that failure live, in real time, as reliably as the people who had been burned by it, the frame would be wrong. Knowing would be enough, and the consequence would be decoration. Welsch is one run at something close to that test, and it came back the way the frame predicts. But it is one run, correlational, and a cleaner study could run it back the other way and take the spine with it. I am naming the result that would unconvince me, which is the one thing the trap-with-no-door can never do.
And I owe you the same honesty about this essay, because writing it looks like a counterexample. I felt the false security, caught it, and turned it into a piece. Is that not the calibration I claim I lack? No, and the reason matters. I caught it slowly, days later, in writing, with the cost already absent, performing the cheap retrospective version of a thing that only counts when it fires live and in time to matter. Catching it on the page is the downloadable move. It is not the armor. The armor would have stopped the click before I published last week, and it did not.
The Easiest Mark
I notice the pull, even now, to land this somewhere clean. To gather the frame and the studies and the two kinds of knowing into a last turn that leaves you nodding and leaves me sounding like a man who walked all the way out to the far side of the thing. I felt that same pull last week, and last week I called it handled and hit publish, and the click I mistook for safety was the failure starting. So I have learned at least this much: the clean landing is the tell, and I am the last person who should trust the version of it I can write most convincingly.
What I am left holding is smaller and more honest. Knowing how the trap works is not standing outside it. A year of frame and a head full of failure modes does not lift me out of the gap. On the worst days, it digs me in because it hands me a more sophisticated reason to feel safe than you have, and the sophistication is the danger, not the safety. And what is true of one man at a red light is true of every room where these tools get adopted because they feel helpful rather than because anyone measured whether they were, which is most of them, and none of that is my subject today, only my reason for not trusting the feeling. The expert is not the one the counterfeit struggles to reach. He is the one it reaches most quietly, through the very fluency he mistook for armor. That includes me. Especially me.
But here is the part I can actually stand on, the only part that is not just more articulacy. The proof of any of this was never going to be in the essay. It is in the next time. The next fluent voice I agree with, the next answer that arrives smooth and complete on a subject I have not paid to know, the next click of having handled it. That is where the calibration either fires or it does not, live, in time to matter, with the cost present instead of safely absent. This page is the downloadable version. The armor, if I ever build any, gets built out there, one caught click at a time, and most of them I will miss.
So I am not going to tell you I have closed the gap. I am going to go driving, and a passenger is going to say something fluent that I want to believe, and we will find out. That is the only test that counts, and it is not one I get to pass in writing. It is just the next, most comfortable place for the bias to hide, and the work is going out to look for it there.
Originally published on Substack.
