Architecture and AttentionArchitecture & Attention
Plain-language edition

What Would You Build Instead?

Almost all the argument about AI is about whether the current approach will get there. Very little of it is about whether there is the right thing to be aiming at. This is the second question, and it has a better answer.

15 sheets · the argument of the working paper AHI: The Case for Augmented Human Intelligence · James Maconochie

Where this sits

There is more language in the world than anyone can attend to. The obvious fix, an AI that reads it all for you and tells you what is true, does not work: the judgement that would take requires having been wrong at your own expense, and nothing a model does costs it anything. So the judging stays with people, and almost nothing we currently inspect can tell whether a person actually judged or simply agreed.

  1. Why Does It All Feel Like Too Much?
  2. Why Can't You Just Be Told?
  3. What Holds a Claim Up?
  4. How Would You Know?
  5. Who Pays If It's Wrong?
  6. What Would You Build Instead?

Six short pieces, each drawn from a working paper. Each one stands on its own; this is the sixth. They are ordered so that every piece ends by raising the question the next one is called. The full map is in Six Questions, In Order.

Part one

The goal is the problem

01

The question nobody is asking

Will machines reach human-level general intelligence, and when? That question has hundreds of billions of dollars behind it and most of the argument.

Here is the other one. Suppose it works. Is that the thing we were trying to build?

This piece assumes the diagnosis

A companion piece, Why Can't You Just Be Told?, sets out why a system trained on text cannot acquire the judgement a practitioner acquires by being wrong. That argument is not repeated here. This one takes it as given and asks what follows for anybody actually building something.

02

Three things wrong with the goal

The benchmark problemWe measure progress by tests, and the tests keep being passed by systems nobody would trust with the underlying job. Passing is not the thing the test was standing in for.
The returns problemEach increment costs more and buys less. That is an observation about the curve, not a prediction about the ceiling.
The displacement problemThe goal has quietly moved. We started out asking how to help people think better, and ended up asking how to build something that does not need them.

The first two are arguments about the method. The third is an argument about the destination, and it is the one that matters.

03

How the goal moved

The field did not start here. In 1960 Licklider wrote about man-computer symbiosis, a partnership in which each side did what it was good at. In 1962 Engelbart proposed augmenting human intellect: not replacing the person, but raising what a person could do.

Somewhere between then and now the aim became autonomy. Not a better partner, but a thing that does not need a partner.

Nobody voted on that. It happened because autonomy is easier to measure, easier to sell, and easier to describe in a funding round than a partnership is.

04

Replace, or amplify

This is the whole constructive argument in one distinction, and once you see it you cannot unsee it.

A tool that replaces a function has to reproduce that function. All of it. Every part, to a standard at least as good, or the replacement is worse than what it replaced.

A tool that amplifies does not. It has to find the one place the person is actually stuck, and widen it. Everything else can be left alone, because the person is still there doing it.

TO REPLACE THE JOB TO AMPLIFY THE PERSON gather what is knownspot what is missing weigh it against the casedecide answer for the decision BUILD ITBUILD IT BUILD ITBUILD IT BUILD IT spot what is missingweigh it against the case decideanswer for the decision gather what is known THE PERSONTHE PERSON THE PERSONTHE PERSON five problems, all unsolved one problem, the one that was slow
Fig. 1. The left column is why the hard version is hard. The right column is why the useful version is tractable this year.
05

Glasses do not see for you

Every tool that has genuinely extended what people can do works this way, and none of them works the other way.

GlassesDo not see on your behalf. They fix one specific failure in one specific part of seeing, and leave the rest of your eye alone.
A calculatorDoes not do your mathematics. It removes the arithmetic so you can spend the attention on which sum to do and whether the answer is sensible.
A wheelDoes not travel for you. It changes the cost of a step so that a journey that was impossible becomes ordinary.

Not one of these had to reproduce the thing it helped. That is not a coincidence, it is the reason they worked.

06

So what is the bottleneck?

If the design question is which single thing to widen, then it matters a great deal what is actually narrow. And the thing in short supply is not intelligence.

It is attention. There is more worth reading than anybody can read, more worth checking than anybody can check, and no amount of raw capability on the machine's side adds a single hour to the person's day.

A system designed to produce more output is widening something that was never the constraint. A system designed to make a person's attention go further is widening the thing that was.

Part two

What you would build

07

Five things such a system would do

08

Four commitments, drawn as one thing

Those five are a stance. These four are what you would actually have to put in the architecture, and they are not four features bolted on. They are one loop.

ATTENTION-AWARE THROUGHOUT spends the person's attention where it changes the answer, and nowhere else A QUESTIONWORK, IN STEPS AN OUTPUTTHE PERSON DECIDES with a stake on it the person is inside it with its gaps marked and is answerable for it corrections go back in the four commitments: checkpoints inside the work · uncertainty on the output correction returning · attention spent deliberately
Fig. 2. The companion piece argues a model cannot have a loop of its own, because nothing costs it anything. This is a loop the system does not have to have, because the person in it does.
09

What honest uncertainty looks like

Of the four, this is the one you could build on Monday and the one almost nothing currently does. The same underlying state of the evidence can be presented two ways, and only one of them lets a reader do anything.

ONE STATE OF THE EVIDENCE, TWO PRESENTATIONS SEAMLESS nothing here tells you where to push HONEST THE SOURCES DISAGREE HERE NOTHING FOUND ON THIS AT ALL now you know which two to check same five lines, same model, different contract with the reader
Fig. 3. The seams are not a defect to be smoothed away. They are where a reader with judgement can get a grip.
10

Three tools that would follow

A research toolThat leads with where the sources disagree rather than resolving the disagreement quietly on your behalf.
A writing toolThat asks a clarifying question before producing eight hundred words built on a guess about what you meant.
A decision toolThat states the assumptions its recommendation rests on, so you can disagree with an assumption rather than with a conclusion.

None of these is technically hard. All of them are commercially inconvenient, because each one makes the product look less capable than its competitor in a demonstration.

11

One of them exists

These commitments are not a wish list written by somebody who has never built anything. The author has a working prototype, a multi-agent research system, built to the commitments above: several agents with different jobs, a person inside the loop at the checkpoints rather than reviewing at the end, and disagreement between agents surfaced rather than averaged away.

It is a prototype and it is offered as one. What it demonstrates is not that the problem is solved, but that these commitments are things you can actually build with, rather than principles that evaporate on contact with an implementation.

Part three

Limits, and what to do with it

12

What this does not claim

Short versions tend to overstate. Here is what this one is not saying.

13

Who decides which is which

The honest hole in all of this. If the distinction is between augmenting a person and replacing them, somebody has to decide which is which, and that is not a technical judgement.

The same system is augmentation to the person using it and replacement to the person it was bought instead of. There is no view from nowhere that settles it, and the people best placed to decide are rarely the people who carry the consequence.

This paper does not solve that. It says the decision is a public one and should be made in public, which is a weaker answer than anybody wants and a more honest one than pretending the technology decides.

14

Three questions for any tool you are offered

  1. What does it assume about the person using it?That they have judgement worth preserving, or that they are an obstacle between the output and the outcome?
  2. Does it ever say it does not know?If it has never once told you that the sources conflict or that it found nothing, it is not that it has always known.
  3. After a year of using it, what can you do that you could not before?If the honest answer is "produce more of what I used to produce", the tool widened throughput. If it is "notice things I used to miss", it widened you.
15

Where this comes from

The argument here is the constructive half of a working paper called AHI: The Case for Augmented Human Intelligence. What the paper adds is the sources and the objections, including the ones that have not been answered.

The diagnosis it rests on is in the companion piece, Why Can't You Just Be Told?, and in the paper behind it, The Wisdom Gap.

The system amplifies. The human decides.

The closing line of the paper

The slides for this piece: PDF · PPTX