Buried in the tail: why generative models don't imagine

Generative models (like LLMs) are already trained on a large share of what humans have written: books, papers, forum threads, patents, manuals. Each new model reads more. Follow that to the end, a model that has read every sentence ever written and knows everything that was ever thought, and one question is left: would it ever think something new?

The flying car test

Ask a model today about flying cars and it answers fluently: lift, noise, certification, battery density, air traffic control. It could do this long before anyone put a working prototype on stage, because people have written about flying cars for a century. Fiction, magazine covers, failed startups, forum arguments about why they will never work. All of it is in the corpus.

Now run the thought experiment properly. Take a world where nobody has ever written about a flying car. Not the phrase, not the idea, not the sketch. Train the same model on everything that world has written. Does it ever say "what if the car flew?"

The honest answer is: almost never, and it would have no way of knowing that it should.

Not zero, but buried

It's tempting to say the probability of an unseen idea is zero by definition. That's not quite right, and the real reason is more interesting.

A language model assigns some probability to every possible sequence of words. "What if the car flew" is a valid sequence, built from words it has seen. So the probability isn't zero. The problem is everything around it. For every novel sentence that turns out to be a good idea, there are billions of novel sentences that are nonsense, and from inside the model they look about the same: rare, unlikely, off the beaten path. Training pushes probability toward what people actually wrote. Sampling, especially the careful kind used in products, pulls further toward the typical.

So the new idea isn't missing. It's buried, at the same depth as every bad idea, with nothing to mark it. Imagination was never about producing an unusual sentence. Random noise does that. It's about knowing which unusual sentence is worth keeping, and that judgment doesn't come from the corpus. By construction, the corpus has nothing to say about it.

Titan

Ask a model about a colony on Titan, Saturn's largest moon, and it will give a good answer. Surface temperature around -180°C. A nitrogen atmosphere about one and a half times as thick as Earth's, thick enough that you wouldn't need a pressure suit, only a very warm one. Gravity about a seventh of Earth's. Lakes of liquid methane and ethane. It might even mention that the thick air and low gravity together would let a person fly with strapped-on wings.

Every one of those facts is in the corpus. Planetary scientists wrote them down, and science writers repeated them. The answer is assembly, and very good assembly. What it isn't is someone standing on Titan in their head, walking out of the airlock and asking what breaks first. That process, simulating a world and deriving consequences nobody wrote down, is how the idea of a Titan colony got onto paper in the first place. The model got the output of that process. It didn't get the process.

Newer reasoning models narrow this gap. They can chain steps, check their own arithmetic, work through a derivation. But each step is a move of a kind seen before, and the chain still has no outside world to bump into. It can be coherent and wrong at the same time, with nothing to tell it so.

The librarian

None of this is new about machines. It's an old fact about people.

If intelligence were the amount of text you had taken in, the only exam we'd need would be a log of hours spent in the library. Nobody believes that. A great librarian may know every book on the shelves, where each one sits and what it says, and still not be the person who builds the rocket to Mars. The rocket builder may have read far less. What they have is a habit of asking what isn't on the shelves, and the stubbornness to go and find out.

David Deutsch, building on Karl Popper, puts this well in The Beginning of Infinity: knowledge grows by conjecture and criticism, not by accumulating observations. You guess something that doesn't follow from what you've seen, and then you let reality try to kill the guess. Reading is how you learn the previous guesses. It isn't how you make the next one.

A generative model is the most complete librarian ever built. That is a real achievement. It's just a different job from the rocket.

Where it shines: carrying ideas between rooms

The perfect librarian does have one advantage no human has: it has read every room of the library at once.

A surprising amount of progress is transplant, not invention. Toyota took the idea of restocking shelves only when they emptied from American supermarkets and turned it into just-in-time manufacturing. Hospitals took checklists from aviation cockpits and cut surgical deaths. The problem in the second industry had already been solved in the first. What was missing was a person who had stood in both rooms.

That is exactly the gap a generative model fills. A logistics manager who asks how a bottleneck in their warehouse resembles a bottleneck in a chip fab, or a CPU scheduler, or an airport gate, gets an answer drawn from all three, instantly. The patterns and the fixes already exist. The model's reach across domains is wider than any single career. For most businesses that is enough, because most problems are not new. They are old problems wearing new clothes.

The counterexamples, and what they share

There are machine results that look like genuine novelty. AlphaGo's move 37 against Lee Sedol in 2016 was a move professionals said no human would play, and it won. DeepMind's FunSearch found new constructions in an open maths problem. AlphaEvolve found a faster way to multiply 4×4 matrices than the one that had stood since 1969.

Look at what each of them had that a chatbot doesn't. AlphaGo played millions of games against itself, and the rules of Go said who won. FunSearch and AlphaEvolve paired a language model with a program that ran every proposal and scored it. In each case the generator only proposed. Something outside it, a game, a compiler, a test, decided what survived.

That's Deutsch's loop again, conjecture and criticism, with a machine doing both halves. And it points to where the limitation actually sits. It isn't a limit on machines. It's a limit on a generator trained on the past with no contact with the world. The corpus can only contain the record of previous experiments. New knowledge comes from running a new one.

So what are these things?

A generative model is a mirror of everything humans have written, and an extraordinarily good one. It will answer any question that someone, somewhere, has already half-answered, and it will connect answers that were never in the same room. That covers a large share of useful work.

What it won't do on its own is look at a world where nobody has imagined a flying car and imagine one. Not because the words are missing, but because nothing in a record of the past can tell it which new idea is worth having. That part still takes someone who will guess, and then go and check.

Not just words

Everything above was told with language models, because text is where it's easiest to see. But nothing in the argument depends on words. An image model trained on every photo and painting, a music model trained on every recording, a protein model trained on every known sequence: each follows the same recipe. Learn what's likely from a record of the past, then sample from it. Each gives some probability to every possible output, and each buries the genuinely new one at the same depth as the noise. An image model can draw a flying car if someone has drawn one before. In a world with no such drawing, it is no better placed to put wings on a sedan than a language model is to write "what if the car flew?" The limit isn't in language. It's in learning only from what has already been made.