An Answer-Shaped Object: What large language models actually give you

Every discussion about AI seems to circle the same argument.
Is it correct?
Is it hallucinating?
Does it actually understand what it’s saying?
Those are interesting questions. They’re also slightly beside the point.
Because when you interact with a large language model, what you actually receive is something simpler and stranger.
You receive an answer-shaped object.
The Shape of the Thing
A large language model does one mechanical task.
It takes a sequence of tokens — words or fragments of words — and predicts the most statistically likely next token based on patterns learned from enormous amounts of human writing.
Then it repeats that process again and again.
From the outside, this produces something that looks very familiar:
You type a question.
Something that looks like an answer appears.
But what the system actually produced is not knowledge in the human sense. It is a probabilistic artefact shaped by the patterns of human expression.
In other words, it’s an answer-shaped object.
It has the structure of an answer.
It behaves like an answer.
Sometimes it is a very good answer.
But mechanically speaking, it is something else: a statistical estimate formed from the accumulated ways humans have talked about similar things before.
The Eiffel Tower Test
Imagine asking a model:
What does it look like from the top of the Eiffel Tower?
The response will probably be vivid. It may describe the Seine winding through the city, the pale rooftops of Paris, the way Montmartre rises in the distance.
The model has never been there.
But thousands of people have. They wrote travel blogs, novels, photo captions, diary entries. The model’s training data contains those descriptions. When you ask the question, it navigates that landscape of human testimony and constructs a likely description.
What comes back is not direct experience.
It’s an average of human description.
An answer-shaped object built from the statistical distribution of how people talk about that view.
Useful Compared to What?
Most of the AI debate asks one question:
Is the answer true?
But there’s another question that matters just as much.
Is it useful?
Consider a blind person asking that Eiffel Tower question.
They are not comparing the model’s answer to the real view. They cannot see the view. The comparison is between the answer and nothing at all.
Measured against nothing, the value of that answer-shaped object changes dramatically.
It becomes a navigational estimate — a way to participate in a conversation about something that would otherwise be inaccessible.
It isn’t sight.
But it isn’t nothing.
This is a domain I know. I am blind. I have spent four decades working in access and assistive technology. When I ask a model what the view looks like from the Eiffel Tower, I have absolute domain expertise on what that answer is worth to me. I know what I’m holding. I know its limits. I know how to use it.
But What About the Hard Cases?
Now change the question.
My husband seems depressed. How can I help?
The model will produce an answer-shaped object. It will probably be structured, compassionate, and plausible. It may suggest listening without judgement, encouraging professional help, being patient. It will sound like good advice. It will have the shape of good advice.
But the success function here is vastly more complex than the Eiffel Tower.
With the view from the tower, I needed a mental picture and I got one. I could evaluate it against my own experience of how descriptions work, against what I know about Paris, against decades of navigating the gap between sight and language. I had the domain knowledge to judge the estimate.
With the depressed husband, I might not. I’m not a therapist. I’m not a psychiatrist. I’m a self-styled philosopher of access and assistive technology — just some old widow with cats. The answer-shaped object might be genuinely helpful. It might give me language for a conversation I don’t know how to start. Or it might be subtly wrong in ways I cannot detect, precisely because it sounds so plausible.
This is where the shape becomes dangerous. Not because the model is malicious, but because the convincingness of the shape scales independently of the accuracy of the content. The answer sounds most authoritative exactly where you are least equipped to judge it.
Dead Reckoning
In navigation, there is a method called dead reckoning.
If you don’t have GPS or a fixed reference point, you estimate your position using your previous position, your direction, your speed, and elapsed time.
The estimate drifts over time.
It isn’t ground truth.
But it is still incredibly useful because the alternative is having no idea where you are.
Large language models work in a similar way. They provide dead-reckoning knowledge — estimates derived from the accumulated patterns of human expression.
But dead reckoning has a crucial property: its usefulness depends on the waters you’re in.
In open ocean with nothing else to steer by, a rough position estimate is invaluable. In a narrow harbour with rocks, the same margin of error kills you.
The question isn’t just is this better than nothing? It’s does the person receiving this know what kind of water they’re in?
The Human Layer
The answer-shaped object is not the end of the process. It is the beginning.
A human being — with experience, context, and judgement — decides whether that object is useful.
The model produces the estimate.
Your wetware evaluates it.
The model does not know.
The model does not understand.
The model generates answer-shaped objects.
The human decides what they are worth.
And that evaluation is not evenly distributed. The people for whom the answer-shaped object fills the biggest gap — those without expertise, without access, without certain sensory channels — may also be the people least equipped to judge when the estimate is drifting. The human layer is not optional. It is load-bearing. And we should be honest about the fact that it bears more weight for some people than others.
Dismissal-Shaped Objects
The loudest voices in the AI debate often belong to people who have reduced their entire position to a single gesture:
It’s all slop. Confabulation. AI bollocks.
They think they’ve said something profound. They haven’t. They’ve refused to think about baselines.
When someone dismisses all model output as worthless, they are comparing it to an idealised standard — expert knowledge, direct experience, verified truth — that many people asking the question never had access to in the first place.
They are also, without apparent irony, doing exactly what they accuse the model of doing. They are producing dismissal-shaped objects — responses that have the form of a considered position without the substance of one. Low-effort criticism shaped like insight. The human equivalent of slop.
If you want to argue that large language models are dangerous, argue it properly. Show me where the estimate drifts. Show me who gets hurt when the shape deceives. Show me the rocks in the harbour.
But don’t wave your hand and say “it’s all confabulation” as though that’s the end of the conversation.
It’s the beginning.
What We Actually Get
Large language models do not give us truth. They give us answer-shaped objects — navigational estimates drawn from the vast archive of human expression.
Sometimes those estimates are exactly what you need.
Sometimes they are dangerous.
The difference depends on context, on stakes, on what you know and what you don’t.
What we do with them is still, unmistakably, a human job.
And that job starts with being honest about what we’re holding: not knowledge, not slop, but something in between.
An answer-shaped object.
The interesting question was never whether it’s real.
It was always whether you know what to do with it.