Architecture & Attention
Plain-language edition

How Would You Know?

Two people hand in the same document. One of them thought about it. The other agreed with it. Every record of those two pieces of work is identical, and no inspection anybody currently runs can tell them apart. That is the subject here, and it is why most AI rules are pointed at the wrong thing.

19 sheets · the argument of the working paper AI Governance and the Architecture of Practice · James Maconochie

Where this sits

There is more language in the world than anyone can attend to. The obvious fix, an AI that reads it all for you and tells you what is true, does not work: the judgement that would take requires having been wrong at your own expense, and nothing a model does costs it anything. So the judging stays with people, and almost nothing we currently inspect can tell whether a person actually judged or simply agreed.

Six short pieces, each drawn from a working paper. Each one stands on its own; this is the fourth. They are ordered so that every piece ends by raising the question the next one is called. The full map is in Six Questions, In Order.

Part one

The problem

Two people, one document

Two loan officers get the same mortgage application. The system returns the same recommendation for both: risk tier B plus, approve at a quarter point over prime.

The first officer reads it, notices a gap in the applicant's employment history that the model has weighted lightly, and telephones the applicant to ask about it. Satisfied, she approves. The second reads the recommendation and approves.

Fig. 1. Same file, same recommendation, same approval, same day. The phone call leaves no field behind.

What the audit can reach

Now imagine inspecting those two decisions, with every power a regulator currently has. You can read the file. You can test the model for bias. You can check that the applicant was given reasons. You can review the approval.

All of it is identical. The thing that differed happened in somebody's head and on a telephone, and it is not in the record. Not because anybody hid it. Because there is no field for it.

The question is not whether, it is how

Most rules ask whether AI was used, and require it to be disclosed. That is the wrong question, because both officers used it.

As a scaffoldThe output is where thinking starts. You read it looking for what it missed, what it weighted oddly, what you would have asked that it did not.

As an amplifierThe output is where thinking stops. It is fluent, it is plausible, it agrees with itself, and agreeing with it is quicker than not.

Same tool, same document, opposite use. That is the variable that matters, and it is invisible in everything we currently inspect.

1993, and a factor of ten

A final-year engineering project. The analysis package reported that the structure would stand, with the confident internal consistency such packages have.

10×How far out the stress figures were, because loads had been entered in kilograms where the program expected newtons.

One checkA back-of-envelope calculation, done by hand, found it.

The software was not broken. It answered exactly the question it was given, and it had no way to know the question was wrong. Everything downstream of that answer was fluent, consistent and false.

The only thing standing between that output and a bad structure was somebody who still knew roughly what the answer ought to look like.

What you did not check

There is a name for the gap between what you signed your name to and what you would have signed had you done the work yourself. Call it the debt.

A companion piece, Why Can't You Just Be Told?, argues that judgement is built by being wrong at your own expense. What follows here is the same erosion seen from the other end: what happens to a profession when the occasions for being wrong quietly stop arriving.

01It is invisible when you take it on. Nothing happens. The document goes out. It is probably fine. Most of the time it is fine.
02It compounds. Each time you accept rather than interrogate, the next acceptance is slightly easier and your own standard moves slightly.
03It only appears at scale. Never in one decision. In a cohort, over years, as a capacity that used to be there and is not.

Fig. 2. Schematic, not measured. Every step is the same size, which is the point: no single one is worth noticing.

Twelve a year, a hundred and twenty a decade

Two researchers are asked to review the same paper. Both start from an AI-drafted review. The first adds a question about a competing method the draft ignored, and deletes a question that misread the notation. The second improves the tone and submits.

One review is nothing. A reviewer does perhaps twelve a year, which is a hundred and twenty in a decade. Somewhere in that hundred and twenty, the second reviewer stopped being able to do what the first one did, and no particular review was the one where it happened.

Why good prose gets through

None of this is a failure of character. It is a property of fluent writing.

Text that is well organised, confident and internally consistent reads as competent, and reading as competent is enough to stop most people looking harder. The 1993 structural analysis had exactly that quality. So does almost everything a model produces now, and it produces it in quantity, at speed, on demand.

Checking is slow and disagreeable. Agreeing is fast and feels like work. The incentive runs the wrong way every single time.

Editing the tone is not reviewing

And from the outside, the two look the same. The second reviewer did change the document. There is a tracked edit. There is evidence of engagement.

Any requirement to "review AI output before use" is satisfied completely by changing three adjectives. This is the trap waiting for every framework that asks for engagement without asking what the engagement produced.

Part two

What the rules currently do

What we inspect

Take lending, which is among the most heavily regulated decisions anybody makes with a model. There are rules requiring the applicant be told why. Rules against discriminatory outcomes. Rules requiring that models be validated, documented and monitored. Rights of appeal.

These are real protections and they work on what they are aimed at. Every one of them is aimed at the artifact: the model, the decision, the paperwork, the outcome.

Risk below the line anybody looks at

There is a structural precedent and it needs stating carefully, because the substance is not the same.

In the years before 2008, a great deal of risk accumulated in a place the inspection regime of the day did not reach. The substance there was different: corrupted incentives, loans written with no documentation, ratings that did not describe what they rated. None of that is what is being alleged here.

What is shared is only the shape: something real building up underneath the layer everybody is watching, invisible in every report, until it is large enough to show up as an outcome nobody can trace to a decision.

The alert nobody reads

Hospitals have had decision support for decades, and with it a long record of what happens to a safeguard people stop attending to.

over 90%The rate at which clinicians override decision support alerts, reported repeatedly across the literature on these systems. Reported across the clinical decision support literature rather than from a single study

A warning that is overridden nine times in ten has stopped being a safeguard and become a formality. Nothing in any record marks the day it changed, and the compliance report still says the alert is in place.

Part three

What it would have to do instead

Four things, and not four more audits

If the variable is whether the person thought, then rules have to act on the conditions for thinking rather than on the paperwork that comes out.

Fig. 3. Drawn as a list, these are four compliance items. Drawn as a system, they are the conditions under which somebody can still think.

What that looks like on a Tuesday

None of this is abstract, and it is worth seeing in three different trades.

In a hospitalRecord what the physician would have ordered before the recommendation is shown. Put a floor under the time allowed per decision. Watch how often a clinician's own reasoning and the system's agree, and from which direction.

In lendingRequire the officer to name one factor the model did not weight. Cap how many files one person can decide in a day. Track how the divergence between officer and model is distributed across a team.

In peer reviewKeep junior reviewers working on unassisted manuscripts alongside a mentor. Require one issue the draft missed. Cap reviews per year, with a floor on time per review.

Notice what these have in common. None of them inspects the output. All of them change what finishing the work requires.

It bolts onto law that already exists

This is not a proposal for a new regulator. Each of the four has somewhere to attach in doctrine that is already there.

Fig. 4. Four attachment points in doctrine that already exists. The paper is explicit that the first of these is forward-looking: the standard of care for working alongside a model has not been settled yet.

How it would be gamed

Any requirement becomes a target, and the paper names the three failures in advance rather than waiting to be told about them.

01Documenting instead of thinking. Producing a record that proves you engaged, which takes the time you would have spent engaging.
02Playing the measure. If divergence from the model is being counted, diverge decoratively. The measure improves and nothing else does.
03Ceremony. The form is completed, the box is ticked, everybody knows it means nothing, and the arrangement survives because taking it away would look like lowering standards.

Gaming any of these still leaves the debt exactly where it was. That is the one consolation, and it is a thin one.

The problem with no answer

Everything in this part rests on somebody senior doing the cultivating: setting the standard, showing a junior what interrogation looks like, naming the number that smells wrong.

The people best placed to do that are the people who have worked alongside these tools the longest, which means they are carrying more of the debt than anyone they would be teaching.

The paper concedes this and does not solve it. It is the hardest problem in the framework and it is the author's own: he has been working this way for years and has no way to audit what it has cost him that does not rely on the very judgement in question.

Part four

Limits, and what to do with it

What this does not claim

Short versions tend to overstate. Here is what this one is not saying.

✕Here is the legislation. There isn't any. The paper proposes no law, no regulator and no institutional structure. It argues that the current target is wrong and says what a better one would have to reach.
✕Don't use AI for serious decisions. Not the argument. The argument is about the difference between two ways of using it, one of which is fine.
✕Do these four and you are safe. Necessary, not sufficient. None of them can supply professional culture or institutional willingness, and the paper says so.
✕Surface tells you who is at fault. It flags, it does not convict. It cannot tell hollowing-out from somebody who is genuinely excellent with the tool, and treating a flag as a verdict would be its own failure.

Try it on the rule you already have

Most organisations have now issued something close to this: disclose when you have used AI, and review the output before you rely on it. It sounds reasonable. Hold it against the three questions.

What does it inspect?Whether you declared it, and whether you looked. Both are properties of the document. Neither is a property of your thinking.
Would both officers pass it?Yes. Both used the model. Both would disclose it. Both reviewed the recommendation before approving. The rule cannot separate them, and separating them was the only thing worth doing.
What does complying cost?A tick in a box and a few minutes reading. Nothing that requires you to have thought, which means nothing that stops you not having.

That is not a bad rule. It is a rule aimed at the artifact, in a problem that lives somewhere else.

Where this comes from

The argument here is the one made in a working paper called AI Governance and the Architecture of Practice. What the paper adds is the cases, the sources and the objections, including the ones that have not been answered.

It is part of a series on what judgement is, why it has to be carried by people who bear the consequences of being wrong, and what that means for building with machines that bear none.

The slides for this piece: PDF · PPTX