Brexit

Back in 2019, OpenAI released GPT-2, and for a few weeks it felt like the most exciting and faintly alarming thing on the internet. The largest model, 1.5 billion parameters, was held back at first — OpenAI staged the release across the year, citing worries about mass-produced misinformation, and only published the full weights on 5 November 2019. So naturally, the moment I could run it, I gave it the most misinformation-friendly prompt I could think of: a couple of sentences about Brexit, and a request to keep going.

What follows below is the unedited output. I have left it exactly as the model produced it, because the point of this post is not the content — the content is nonsense — but why it is such confident, fluent, citation-laden nonsense, and what that taught me about every language model I have run since. If you want the modern equivalent of this experiment, I wrote up a proper local-model setup in running LLMs on your own metal with Ollama; the contrast between 2019 and now is the real story.

Note: the block-quoted text below is machine-generated by GPT-2 and is deliberately preserved as a historical artefact. It is incorrect, internally contradictory, and should not be read as fact about Brexit or anything else.

The raw output

If the EU’s withdrawal from the EU means that all citizens of the UK will be excluded from the Single Market, then all people from EU states will be required to leave the EU. This means that at a minimum, all citizens of the EU state will be able to enter the UK without the need for a visa. This could even happen when the UK does not yet have access to the Single Market: in the early stages of the negotiations UK citizens may be excluded by some EU institutions.

EU citizens can also be subject to these restrictions at the time of departure, which is a good thing in some cases: for example EU citizens in the UK of legal permanent residence could enter only if they are living in the EU state of residence for one year and for four years each. The European Court of Justice in Brussels has ruled that this is not appropriate because of the “right of Member State to access the EU market and/or the single market for citizens of the Member State concerned” (CFRJ 2002/12/9, p. 11).

A Downing Street source said: “The Government of the day is prepared to see the UK as an independent independent state, which should not have a single decision on which of three different EU regions it wants to maintain.” Theresa May has been in Brussels before the referendum and while a majority of Scots support Brexit, some are unhappy that the Government of the day has been unable to agree on one of the key demands of the new British government.

Why it reads so well

Notice what the model gets right. The register is perfect: it sounds like a wire-service article that has been lightly edited by someone who reads a lot of them. The sentence rhythm is journalistic. It invents a “Downing Street source” and a quote, because that is what these articles do. It even fabricates a legal citation — “CFRJ 2002/12/9, p. 11” — complete with a page number, because real reporting cites things and the model has learned the shape of a citation without any concept of whether the document exists.

That last point is the whole game. GPT-2, like every language model after it, is a next-token predictor. It was trained on roughly 40 GB of text scraped from outbound Reddit links (the WebText dataset), and all it ever learned to do is answer one question, over and over: given everything so far, what word is most likely to come next? It has no model of the European Court of Justice, no register of EU treaties, no idea that Theresa May had already resigned by the time this was generated. It has a statistical sense of what sentences about Brexit tend to contain, and it produces more of them.

This is why the output contradicts itself within two sentences — “all citizens of the UK will be excluded from the Single Market” followed immediately by citizens being “able to enter the UK without the need for a visa.” Local coherence is cheap for a next-token predictor; global coherence is expensive, and 2019-era models simply did not have it. The text is grammatical all the way down and meaningful almost nowhere.

It is worth remembering the panic that surrounded this. When OpenAI announced GPT-2 in February 2019, they declined to release the full 1.5-billion-parameter model precisely because of output like the block above — text that looked exactly like a credible news article and could, in theory, be mass-produced to flood the internet with plausible fabrications. Critics called the staged release a publicity stunt; defenders called it responsible disclosure. By the time the full model shipped that November, OpenAI reported they had “seen no strong evidence of misuse,” and released the weights. With seven years of hindsight, both sides were a little right: the technology was not yet good enough to be the disinformation engine people feared, but the trajectory was exactly as worrying as advertised. The Brexit text is the embryo of a problem we are still arguing about.

The lesson that aged extremely well

I keep coming back to this experiment because the failure mode never went away — it just got harder to spot. Modern models are vastly better at global coherence, so the obvious tells (contradicting themselves mid-paragraph) are mostly gone. What remains, and what GPT-2 demonstrated so nakedly, is confident fabrication of specifics. The fake legal citation with a page number is the ancestor of every hallucinated API method, invented court case, and plausible-but-wrong configuration flag that LLMs still produce today.

The practical takeaway I took from a toy model in 2019 is the same one I apply to production models now:

1. Fluency is not knowledge. The smoother the prose, the more carefully you should check the facts, because fluency is exactly the thing these models optimise for and accuracy is not.
2. Specifics are where they lie. Vague claims are usually defensible. The dangerous output is the precise one — the version number, the citation, the named source — because that is the model imitating the form of authority without the substance.
3. A citation is not a source. “CFRJ 2002/12/9, p. 11” looks authoritative and refers to nothing. Treat every model-supplied reference as a claim to be verified, not a verification.

If you want the modern, defensible way to make a model answer from real documents instead of inventing them, the technique is retrieval-augmented generation — I explained the mechanics and the limits in RAG explained: how AI stops making things up. RAG does not make models honest; it gives them something true to copy from, which is a different and more achievable goal.

Trying it yourself (then and now)

In 2019 the experiment looked roughly like this — clone the repo, download a model size, and sample:

1
2
3
4
5
6

git clone https://github.com/openai/gpt-2.git
cd gpt-2
python3 -m pip install -r requirements.txt
python3 download_model.py 1558M          # the full model, ~6 GB
python3 src/interactive_conditional_samples.py \
    --model_name 1558M --temperature 0.8 --top_k 40 --nucleus 0.9

The two knobs that mattered then still matter now. temperature controls how adventurous the sampling is: low values make the model repeat itself in dull loops, high values make it wander off into word salad, and somewhere around 0.7–0.9 you get the readable-but-untrustworthy output above. top_k and nucleus (top_p) sampling restrict the model to only the most probable next tokens, which is what stops it veering into gibberish — at the cost of making it blander.

Today I would not bother with the GPT-2 codebase at all. The same experiment in 2026 is a one-liner against a local model, and the tokens-and-sampling concepts transfer directly — I unpacked them in what is a token: how LLMs read, reason, and bill you. The vocabulary GPT-2 sampled from was about 50,000 byte-pair-encoded tokens, and every model since has worked the same way: it never sees words, only token IDs, and “generating text” is really just repeatedly choosing the next ID from a probability distribution over that vocabulary. Once that clicks, the fake citation stops being mysterious — “CFRJ 2002/12/9, p. 11” is just a high-probability sequence of tokens given the surrounding journalistic context, with no anchor to anything real on the other side.

Troubleshooting the experiment

A few things reliably went wrong when people tried to reproduce this, and the modern equivalents fail the same way:

• The model just repeats one phrase forever. Your temperature is too low, or top_k is set to 1. Raise the temperature toward 0.8 and let nucleus sampling open up the candidate pool. Pure greedy decoding loves to get stuck.
• Output is total gibberish. Temperature is too high, or you have disabled top_k/top_p entirely. Rein it back in. Unrestricted sampling from the full vocabulary is almost always garbage.
• It runs out of memory. The 1.5B model wanted real RAM/VRAM in 2019 and many people quietly fell back to the 124M or 355M variant, which produces noticeably worse coherence. If your local model is suddenly much dumber than expected, check you actually loaded the size you think you did.
• The “facts” are wrong. They are always wrong, or at best accidentally right. This is not a bug to fix; it is the nature of the tool. See the entire post above.

Is it worth it — who is this for?

Running GPT-2 in 2026 is purely historical curiosity, and that is genuinely the value of it: there is no better way to feel what a language model actually does than to run an old, small, honest one that fails visibly. The modern models hide their failure modes behind better coherence and a confident tone, which makes them more useful and, in exactly the same measure, more dangerous to trust blindly.

So who is this for? Anyone who is about to wire a language model into something that matters and wants a gut-level understanding of where it will let them down. Spend an afternoon watching a small, honest model fabricate citations with total confidence, contradict itself across a paragraph, and quote sources that have never existed, and you will never again read fluent model output as if fluency implied truth. That single lesson — bought cheaply from a 2019 toy — has saved me from shipping more bad output than any amount of theory. The Brexit text above is wrong in every particular, and that is precisely why it was worth keeping.