What Holds a Claim Up?

Two people, one report, two mistakes

One reads it and decides nothing is really true, that it all depends on who is talking. The other reads the same pages, does the arithmetic carefully, and announces an answer.

The trouble with the second one is that nothing in the report could have settled that question, and no amount of careful arithmetic adds what was never there.

These look like opposite mistakes. They are the same one. Both ended up holding a conclusion with nothing underneath it.

This is not a new problem

Eighteen hundred years ago a Roman emperor wrote himself an instruction about exactly this.

"Do not say more to yourself than the first impressions report."

Marcus Aurelius, Meditations, Book 8

Something arrives. We add to it: a cause, a motive, a story. Then we treat the addition as though it had arrived with the observation. That is the whole failure, named before anybody had a printing press, let alone a feed.

Everything rests on something

A bridge holds because of what it stands on. Not always directly underneath: on a suspension bridge the load runs up the cables, over the towers and down into the anchorages at each end. But wherever the path goes, it ends in the ground, and an engineer can walk it.

A claim works the same way, and you can walk that path too. Almost nobody does.

This is not about lying. Most claims that fail were made by careful people who believed them. What failed was underneath.

The ground: what happened, and why

Two different things sit at the bottom. What happened is the cup fell, the plane came down at 9:42, the price was 4.7 times income. You can often see these.

Why it happened is different. Nobody has ever seen a why. It has to be worked out, and working it out is the hardest thing in this whole explanation.

How you get to a "why"

Watching a hundred things fall off a hundred tables will never give you gravity. Watching only tells you what happened. To get to why, you have to do two more things.

You were not there

Almost nothing reaches you directly. It travelled: somebody saw it, somebody wrote it down, somebody decided it mattered enough to pass on, somebody shortened it. What arrives is not the thing. It is a picture of the thing, and every step it took moved it further from the ground.

Travelling does not improve it. A summary of a summary is not a better account of what happened. It is one more step away from it, and every step is a place where something could have been dropped, sharpened or misread.

And one new thing, which is not a longer chain

A newspaper report, however poor, is a chain that starts at something that happened. Somebody was there, or spoke to somebody who was. You can walk back down it: ring the reporter, read the document they cited, find the person they quoted.

An AI answer is a chain that starts somewhere else. What went into the model was text, in enormous quantity, most of it pictures that other people made. What comes out is text shaped by patterns in that text. The training reads the text. It does not go to the places the text describes, and it does not test whether the reasoning inside the text was any good.

This does not mean an AI answer cannot be checked. It means you cannot check it by reading it. The only route is to go behind it, to whatever it was built from, and check that instead. Almost nobody does, and often the trail back is missing.

Four questions worth asking

These are the questions people ask when being wrong is expensive: an accident investigator, an engineer signing off a structure, a doctor choosing a treatment. They are questions about what is holding the claim up, not about how confident the person sounds.

1. Is the cause real?Not "does this story hang together" but "has anybody changed something on purpose and watched what happened".
2. Would people who had every reason to disagree agree anyway? Three newspapers running the same wire story are not three witnesses. Agreement only counts when disagreement was genuinely possible.
3. Has anyone seriously tried to break it?Being open to criticism produces no criticism. Only criticism does. Has somebody with the skill and a reason to win actually attacked it?
4. Has it lasted?Does anyone keep a record of what was claimed before, and does anyone go back and read it?

Passing is not the same as being true

Clear all four and you have something worth taking seriously. You do not yet have something to act on. The questions bring a claim closer to the ground. They never land it there.

Somebody has to decide, and no procedure does it

This is the whole point, so it is worth saying slowly. After the questions have been asked and answered well, a person still has to look at the claim, look at the decision it is about to be used for, and say: that is strong enough for this.

Nothing performs that step. Not a checklist, not a model, not a committee process. It takes someone who has done the work long enough to know where their own judgement is good and where it is not, and who is willing to say so out loud when saying nothing would be easier.

The Federal Reserve, 2006

America's central bank: the one body with the independence, the data and the standing to say out loud that something was building up in the housing market. Here is a case where every support was in place and the thing fell down anyway. It is the most useful case there is, because nothing was broken.

Those two facts were public. What made them dangerous was how they were joined to each other.

The Fed had everything you would want

The Federal Reserve had independence written into law, with terms longer than any president's. It had better access to the data than anybody. Its disagreements were recorded. Its meeting transcripts were published. Its decisions were argued over for decades afterwards.

Every support described in Part One was present and in working order. The people in the room were not fools and were not lying.

The four questions, as they actually went

The sentence that shows how it happened

From the Federal Open Market Committee's own transcripts, on whether house prices could be unsustainable:

"there are stories one can tell about fundamentals that would explain or rationalize housing prices"

A member of the Federal Open Market Committee, the Federal Reserve's rate-setting body, from the published transcripts

Every part of that is defensible. What it is not is a test. A story that hangs together was allowed to do the work that evidence about what was actually happening should have done.

Nobody made the call

Not because the Fed lacked the data. Not because the system was broken. Not because anyone was stupid or corrupt. The last step belongs to a person, and in that room, at that time, no one took it.

What this does not claim

Short versions tend to overstate. Here is what this one is not saying.

• ✕The Federal Reserve caused the financial crisis. It did not. The case here is narrower: a question was in front of an institution built to answer it, and the answer was never given.
• ✕Good structure produces good judgement. The opposite, and that is the entire point. The Fed in 2006 had the structure and did not make the judgement. If structure were enough, none of this would be worth writing down.
• ✕AI output cannot be trusted or checked. It can be checked, but not by reading it. You have to go behind it to the sources, which is expensive and often impossible. That is a specific limitation, not a verdict.

Three things to ask about anything you are handed

1. How far has this travelled from what actually happened? Count the hands it passed through. If you cannot count them, that is the answer.
2. Who tried to break it, and what happened when they did? Not whether criticism was allowed. Whether it occurred, and whether anything was retested afterwards.
3. Who is standing behind it, and what does it cost them if it is wrong?A claim nobody pays for when it fails is a different kind of object from one somebody does.

Where this comes from

The argument here is the one made in a working paper called The Architecture of Truth. What the paper adds, across about fifty pages, is the evidence, the sources and the objections, including the ones that have not been answered.

It is part of a series on what judgement is, why it has to be carried by people who bear the consequences of being wrong, and what that means for building with machines that bear none.

The slides for this piece: PDF · PPTX