Architecture & Attention
Plain-language edition

Who Pays If It's Wrong?

Hundreds of billions of dollars are being spent on a particular bet about how artificial intelligence gets better. The bet may well come off. This is about what happens to everybody else if it does not, and about why that is a question almost nobody is being asked.

17 sheets · the argument of the working paper When the Music Stops · James Maconochie

Where this sits

There is more language in the world than anyone can attend to. The obvious fix, an AI that reads it all for you and tells you what is true, does not work: the judgement that would take requires having been wrong at your own expense, and nothing a model does costs it anything. So the judging stays with people, and almost nothing we currently inspect can tell whether a person actually judged or simply agreed.

Six short pieces, each drawn from a working paper. Each one stands on its own; this is the fifth. They are ordered so that every piece ends by raising the question the next one is called. The full map is in Six Questions, In Order.

Part one

The bet

You are probably in it already

If you have a pension, a workplace retirement plan, or money in an index fund, you own a piece of this. Not because you decided to back artificial intelligence. Because the funds that most people's savings sit in hold the whole market, and a handful of AI-exposed companies have become a very large part of what the whole market is.

Nobody asked you. There was no form. The exposure arrived through the ordinary machinery of saving sensibly.

That is the thing this piece is about. Not whether the bet is a good one; that is genuinely arguable, and the paper behind this says so repeatedly. The question is who is holding it.

What the bet actually says

Underneath the spending is an argument with four steps. It is rarely written down in one place, because each step is usually used to justify the next one rather than stated on its own.

Fig. 1. The rule underneath is the diagram. The boxes are only its evidence. This is a reasonable bet, and the paper says so in the same breath as calling it a bet.

You have seen this shape before

A companion piece in this series, What Holds a Claim Up?, pulls apart the housing market of 2006 and finds four claims stacked the same way, each one standing only while the one beneath it does.

Fig. 2. The paper is careful here and so is this sheet. 2006 was corrupted incentives, loans written with no documentation, and ratings that did not describe what they rated. None of that is being alleged here. What is shared is the structure, and only the structure.

That caveat is the first thing that would fall off a short version, and it is also the thing that keeps the comparison worth making.

Four people who think the first step is wrong

The whole chain rests on step one. So it matters that four well-credentialled researchers, from four different directions, think step one does not hold.

Yann LeCunA Turing Award winner and for over a decade Meta's chief AI scientist. His line is that large language models are useful but are a turning off the road to human-level intelligence, not a road to it.

Gary MarcusThat the reasoning gap is not closing at the rate the scaling story requires, and that what looks like reasoning is often retrieval wearing its clothes.

Rodney BrooksThat when people watch a system do one thing well they immediately assume it can do the neighbouring things, and are usually far too optimistic about that.

François CholletThat memorisation is useful and is not intelligence. On his own benchmark, built to test exactly that, one leading model scored about 5 per cent where people score 84.

And the honest part

These four are not the consensus. Plenty of serious researchers still expect scaling plus refinement to keep delivering, and the paper says so in its own limitations section: picking four critics and finding they agree with each other risks making the profession sound more settled against scaling than it is.

Useful and overhyped are not mutually exclusive

There is a strong pull, in any argument about this, to pick a side: either the technology is real and the money is fine, or the money is mad and the technology is a trick. Both of those are more comfortable than the position the paper actually takes.

Useful and overhyped are not mutually exclusive.
The sentence the paper flags as the most important one in its section on historical parallels

A technology can be genuinely important and the financial structure built around it can still have run ahead of what it has been shown to return. Those are two claims about two different things, and holding both is not fence-sitting.

Part two

Where it is thin

Three cracks

None of these says the bet fails. Each one is a place where the structure is thinner than the spending assumes.

01The adoption gap. The distance between organisations trying AI and organisations running something important on it. Pilots are plentiful. Production is not.
02Who is actually making money. Most of the revenue so far belongs to the companies selling the equipment rather than the companies using it. One leading AI firm is reported to have lost around five billion dollars in 2024.
03The gap between a demonstration and a deployment. A system that is ninety per cent right is a remarkable demonstration. For work where being wrong carries a cost, the remaining ten per cent is the entire problem.

How much of the market this now is

The reason a wrong bet would not stay with the people who made it is arithmetic rather than conspiracy.

about 30%

of the total value of the S&P 500 sat in seven large technology companies, as of late 2025.

S&P Global, 2025. A point-in-time figure that moves with the market; it is quoted here for order of magnitude, not as today's number.

roughly three quarters

of the index's returns since ChatGPT came out are attributable to a small set of AI-exposed companies.

S&P Global, 2025. The paper is explicit that this is a directional estimate rather than a measurement: published figures range from about 60 per cent to over 80, depending on the window, the index method, and what counts as AI-related.

about 60%

of American workers are in an employer-sponsored retirement plan, most of them holding broad market funds.

Investment Company Institute, 2024.

Put those together and a decline concentrated in those companies does not stay in those companies. It arrives in accounts belonging to people who were never asked about any of this.

Part three

Who pays

Four ways the cost travels

The paper's argument is not that a correction would be painful. Corrections are painful. It is that the pain lands somewhere specific, and that the somewhere is not where the decisions were made.

Fig. 3. Four routes out of one decision, and one place they all arrive. The asymmetry is the argument; the four channels are only how it travels.

The one in your account

A target-date fund is the most sensible, least adventurous thing a person can do with retirement money. It is also, right now, a concentrated position in a handful of companies, arrived at by following advice that was good advice.

The people who made this bet deliberately have options an ordinary saver does not: they can spread the risk, they can take out insurance against it, and they can get out at a time of their choosing.

So the group that made the decision and the group carrying the consequence are largely different groups of people. That is the paper's central distributional claim, and it does not require anybody to have behaved badly.

The towns that built for it

Data centres land in particular places: northern Virginia, central Oregon, west Texas, parts of the Midwest. The construction work is real and it is temporary. The permanent staffing of a finished facility is small relative to how much ground it covers.

Meanwhile local governments issue bonds and plan budgets against the tax revenue those facilities are expected to produce.

There is a precedent and it is specific. Around 2000, the same thing happened with fibre-optic cable: towns built for traffic that did not come, and were left with unused capacity in the ground and budgets planned around growth that never arrived.

The current build is larger, and it is more concentrated in fewer places.

Nobody announces this one

The way work actually disappears is not a press release about artificial intelligence. It is a role that does not get refilled when somebody leaves, a team restructured into a smaller one, a set of tasks that quietly moves.

What the evidence does and does not say

Whether AI ends up destroying more work than it creates is genuinely unsettled among economists, and the paper does not claim to settle it. Serious work points both ways and the data is too young. What is much better established, and is a separate question, is that the transition costs something, and that it costs it to particular people, in a particular decade of their lives.

The author of the paper writes himself into this section rather than observing it from outside. After twenty-five years in technology consulting he was made redundant in September 2023, and spent roughly a year looking for the same kind of job before deciding to do something else. He notes that he had savings, a family, and a background that made the pivot possible, and that most people in that position have none of those. That cost does not appear in any aggregate statistic.

The slowest channel

The fourth route is the one with no obvious victim, which is why it tends to go unmentioned.

ComplexityWhen the only people who understand a system work for the companies building it, the people meant to govern it end up structurally dependent on them. That is not corruption. It is worse, because nobody has to do anything wrong.

SpeedCapability moves in months. Legislation moves in years. The result is not deliberate deregulation; it is deregulation by default.

Which level decidesWhere national rules override state and local ones, the places carrying the local consequences are the ones with the least say over them.

AttentionThe public has a limited amount of it and this subject is tiring. That is a problem with how the information reaches people, not a problem with the people.

Each of these makes the next one harder to fix, and between them they leave a gap that gets filled by the organisations that most needed governing.

Part four

Limits, and what to do with it

If not this, then what

The paper does not stop at the diagnosis, but it is modest about the alternative, and so is this sheet.

The proposal is to shift the aim from building something that replaces the person to building something that makes a person's judgement go further. That is also a cheaper thing to build, because it does not require solving general intelligence first.

That argument has its own piece

A companion piece, What Would You Build Instead?, sets out what such a system looks like and what it would commit to. It is not repeated here. The paper itself is careful to call this a direction rather than a specification, and says plainly that its author would need collaborators to turn it into one.

What this does not claim

This is the sheet a short version most needs, because compression is exactly where a careful argument turns into a loud one.

✕A crash is coming. Not the claim, and not a prediction. The paper's stated purpose is to identify where the structure is fragile and who would be hurt if it gave, which is a different exercise from forecasting that it will.
✕AI is a fraud, or a bubble, or useless. The technology is treated throughout as real and important. Fragility is a claim about the financial structure built on top of it, not about whether the thing works.
✕The experts agree scaling has stalled. They do not. Four critics are surveyed and the paper concedes in its own limitations that four critics are not a consensus, and that many serious researchers expect scaling to keep delivering.
✕AI is destroying jobs. The economics is unsettled and the paper leaves it unsettled. What it argues is narrower: that the transition has costs, that those costs fall on identifiable people, and that aggregate figures hide them.
✕The author has no stake in this. He states the opposite. Having been displaced from the industry himself, he flags that this may incline him to read the signals as more threatening than they are and to favour answers that keep people in the picture.

Where these numbers come from

No other piece in this series has this sheet, and this one needs it. The figures here are not all the same quality, and the paper says which are which in an appendix rather than letting the strongest-sounding ones carry the argument.

AdoptionOfficial statistics put business use of AI at under about 10 per cent on a narrow definition. Private surveys using broader definitions reach upward of 50. The direction agrees across all of them; the size does not. The paper uses the low figure deliberately.
Market concentrationThe three-quarters-of-returns figure is directional. Published estimates run from about 60 per cent to over 80 depending on method.
Company lossesRevenue and loss figures for private AI companies come from business journalism, not audited accounts. Informed estimates, not verified facts.
Bond issuanceThe projection of AI-related debt reaching 1.5 trillion dollars by 2030 is a forecast, not an observation. It indicates expected scale and nothing more.
JobsWhether AI displaces or complements labour on balance is genuinely open in the academic literature. The paper cites both sides and resolves neither.
EnergyThe figure for growth in large technology companies' energy use comes from investment-bank equity research rather than from company disclosures or government data.

Three questions worth asking out loud

Who decided, and who is holding it?If those are the same people, this is just investing. If they are different people, it is something else, and the difference is the whole subject.
What would have to be true for this to work?Ask it of the four steps. If step one fails, how much of the rest survives? A chain where everything rests on one claim is not four risks. It is one risk, four times.
Where is the number from?Survey or census. Audited or reported. Measured or projected. These are not pedantry; they are the difference between a figure that can bear weight and one that cannot.

Where this comes from

The argument here is the one made in a working paper called When the Music Stops. What the paper adds is the sources, the limitations and the open questions, including the ones the author cannot answer.

It is part of a series on what judgement is, why it has to be carried by people who bear the consequences of being wrong, and what that means for building with machines that bear none.

If we do not choose the architecture, the architecture will choose us.
The closing line of the paper

The slides for this piece: PDF · PPTX