Ibex: anatomy of a GPT-2 hallucination

What a 2019-era language model got wrong about a wild goat, and why it matters

16-03-2022

Contents

A few years ago I fed the word “ibex” to a GPT-2 model and let it run. What came back was a few hundred words of confident, grammatically immaculate prose claiming, among other things, that the ibex is “the most common wild goat in North America”, that it was “primarily introduced by the Romans during the fourth century”, and that it can weigh “roughly 400 pounds”. Every one of those is false. The animal it described does not exist. And yet it reads like an encyclopaedia entry, which is precisely the problem worth dwelling on — because the failure mode that produced it is the same one sitting underneath today’s far larger models. I kept the output. This post is the autopsy: what the real ibex is, where the machine went wrong, and what a 2019-vintage text generator getting a goat this badly wrong teaches you about trusting any of them. The original front matter on this post carried a disclaimer — that the content was machine-generated and “should not be taken as an expression of any truth whatsoever” — which was honest, and also exactly the wrong way round from how most generated text reaches readers today, with no warning label at all. <h2 id="the-real-animal-briefly">The real animal, briefly</h2><div class="ad-unit ad-in-article" aria-label="Advertisement"> Advertisement <ins class="adsbygoogle" style="display:block;text-align:center" data-ad-client="ca-pub-3726833845844946" data-ad-slot="3291553914" data-ad-format="auto" data-full-width-responsive="true"></ins> <script>(adsbygoogle = window.adsbygoogle || []).push({});</script> </div> The Alpine ibex (Capra ibex) is a wild goat of the European Alps, not North America. By the early 19th century, after centuries of hunting accelerated by firearms, the entire species had collapsed to roughly 100 individuals, clinging on around the Gran Paradiso massif on the Italian-French border. In 1821 the Government of Piedmont banned hunting it, and in 1854 King Victor Emmanuel II declared Gran Paradiso a royal hunting reserve — a piece of self-interested aristocratic conservation that happens to be the only reason the species survives. Every Alpine ibex alive today descends from that remnant Gran Paradiso stock. In 1920 the king’s grandson, Victor Emmanuel III, donated the land to the Italian state and it became Gran Paradiso National Park. The recovery since has been remarkable: a few thousand by the mid-20th century, and a stable population of well over 30,000 across the Alps today. Males carry the enormous backward-sweeping horns the animal is known for; a big one weighs perhaps 100 kg, not the 250 kg the model invented. There’s a genuine genetic footnote here that the model could never have produced, because it requires actual knowledge rather than pattern-matching: because every living Alpine ibex traces back to that tiny Gran Paradiso founder population, the species carries unusually low genetic diversity — a real, studied consequence of squeezing an entire animal through a bottleneck of a hundred individuals. That is the kind of specific, non-obvious, true fact that distinguishes knowing a subject from imitating the sound of someone who does. The model produced the second thing with total confidence and none of the first. That’s the documented record — names, dates, places, numbers you can check. Now hold the machine’s version against it. <h2 id="line-by-line-where-it-went-wrong">Line by line, where it went wrong</h2> The output didn’t fail by going silent or producing gibberish. It failed in the most dangerous possible way: by producing fluent, structured, plausible text that is wrong. <ul> <li>“the most common wild goat in North America” — wrong continent, wrong premise. The ibex is Eurasian; there are no wild ibex native to North America. The model has confidently relocated an entire species.</li> <li>“borrowed from Latin ibexis (‘bear’)” — invented etymology. “Ibex” is indeed Latin, but the fake gloss “(‘bear’)” is fabricated, and the sentence then free-associates into German words that don’t mean what it claims.</li> <li>“primarily introduced by the Romans during the fourth century” — a fabricated history with a specific-sounding date, which is the tell. Made-up facts often come more precisely dressed than real ones, because precision is part of the encyclopaedic style the model is imitating.</li> <li>“in 1885 there were 1,075,000 people” — a number with no referent, dropped in to sound authoritative. There was never a census of ibex at a million-plus.</li> </ul> Notice the pattern. The model has learned the register of a reference article — the cadence, the “X is a Y found in Z”, the confident dates and counts — without any of the underlying facts being anchored to anything. It is generating the shape of knowledge. The text also degrades in a characteristic GPT-2 way as it runs on. The opening sentences are coherent; by a few hundred tokens in, it has wandered into self-contradiction, repeated phrases (“the killing of the animals they kill”), and place-name salad — “Istanbul Şırnak, Istanbul Şırnak, Ankara Şiyagır” — that is the model losing the thread and falling back on locally probable tokens. This is the signature of a smaller model with a short effective attention span: it can hold a plausible sentence together but not a plausible document. Larger successors hold coherence far longer, which paradoxically makes their fabrications harder to spot, because the giveaway of “it stopped making sense” arrives much later, if at all. <h2 id="why-this-happens">Why this happens</h2><div class="ad-unit ad-in-article" aria-label="Advertisement"> Advertisement <ins class="adsbygoogle" style="display:block;text-align:center" data-ad-client="ca-pub-3726833845844946" data-ad-slot="3291553914" data-ad-format="auto" data-full-width-responsive="true"></ins> <script>(adsbygoogle = window.adsbygoogle || []).push({});</script> </div> GPT-2 (and every model since) is, mechanically, a next-token predictor. You can see the whole loop in a few lines using the Hugging Face <code>transformers</code> library, which is roughly how I produced the original text: <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code>1 2 3 4 5 6 </code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-python" data-lang="python">from transformers import pipeline gen = pipeline("text-generation", model="gpt2-large") out = gen("The ibex is", max_length=200, do_sample=True, top_p=0.9, temperature=0.9) print(out[0]["generated_text"]) </code></pre></td></tr></table> </div> </div>That’s it. Given the tokens so far, the model emits a probability distribution over the next token, one is sampled, it’s appended, and the loop runs again. Nowhere in that loop is there a fact, a database, or a check against reality. The model learned, from a large pile of web text, that after “The ibex is” the likely continuations are noun phrases describing a wild animal — so it produces one. Whether that animal lives in the Alps or Alberta is not something the loop is built to get right; it’s a roll weighted by whatever co-occurred in the training data, nudged by the <code>temperature</code> and <code>top_p</code> you set. The two sampling knobs above are worth understanding, because they’re where “confidently wrong” partly comes from. A high <code>temperature</code> flattens the distribution and lets the model take more creative leaps — more interesting, more wrong. Crank it down toward zero and you get safer, blander, more repetitive text. Neither setting makes the output true; they only trade fluency against caution. There is no value of <code>temperature</code> at which the model starts checking facts, because there is nothing in it that could. It’s tempting to think a bigger training corpus fixes this, and partly it does — a model that has seen the Gran Paradiso story a thousand times is more likely to reproduce it correctly than one that saw it twice. But “more likely” is the operative phrase. Scale shifts the probabilities; it does not install a fact-checker. The model that gets the ibex right most of the time will still, occasionally and with the same serene confidence, relocate it to Alberta or invent a census — and you have no built-in signal telling you which run you’re looking at. The ibex article is what happens when the dice land badly on a small model; the identical dice are still being rolled, just with better odds, on every model since. If you want the mechanism in more depth — what a token actually is and how the prediction is billed and bounded — I went through it in <a href="/story/what-is-a-token-how-llms-read-reason-and-bill-you/">what is a token, and how LLMs read, reason and bill you</a>. The ibex article is just that machinery running without a safety net. <h2 id="the-fix-the-industry-eventually-reached-for">The fix the industry eventually reached for</h2> The interesting epilogue is that the whole industry hit this wall and responded the same way: stop asking the model to know things, and start feeding it the things it needs to know at query time. That’s retrieval-augmented generation — pull relevant documents from a real source, put them in the prompt, and constrain the model to answer from them. It doesn’t make the model truthful; it makes the truth available in the context window so the model has less room to improvise. I unpacked how and why that works in <a href="/story/rag-explained-how-ai-stops-making-things-up/">RAG explained: how AI stops making things up</a>. Had a GPT-2 pipeline been handed the actual Gran Paradiso history before writing, the ibex article would have been dramatically better — not because the model got smarter, but because the facts were sitting in front of it. It’s also worth noting this wasn’t a one-off. I ran the same kind of experiment again later, with similarly entertaining results, in <a href="/story/playing-around-with-gpt-2-sequel/">playing around with GPT-2, the sequel</a>. The failure is reproducible because it’s structural, not a bug. <h2 id="what-goes-wrong-and-how-youd-catch-it">What goes wrong, and how you’d catch it</h2> If you’re running a text generator yourself and want to avoid shipping an ibex-grade fabrication, the practical checks are mundane: <ul> <li>Never trust a generated specific. Dates, place names, population figures, etymologies — the more precise and authoritative they sound, the more they need an independent source. Precision is a style the model imitates, not evidence.</li> <li>Ground it. Put the source material in the prompt (retrieval) rather than relying on what the weights “remember”. The original ibex text had nothing to anchor to, which is exactly why it drifted.</li> <li>Lower the temperature for factual tasks. It won’t buy truth, but it reduces the wilder confabulations.</li> <li>Keep a human reading the output for anything that will be published. The whole reason the ibex article is funny rather than dangerous is that I read it and didn’t put it in front of anyone as fact.</li> <li>Assume the failures scale with your audience. One wrong sentence read by you is a curiosity; the same sentence served to thousands of users as an answer is a liability. The review effort should rise with how many people will read the output unchallenged.</li> </ul> <h2 id="is-it-worth-running-these-things-at-all">Is it worth running these things at all?</h2> Yes — with your eyes open. A GPT-2-era model is a genuinely useful tool for drafting, brainstorming and pastiche, and the modern descendants are far more capable. But the ibex article is a permanent, slightly comic illustration of what these systems actually do: they generate fluent text shaped like knowledge, and the resemblance to knowledge is the whole point and the whole danger at once. Who is this for? Anyone about to wire a language model into something that produces output other people will read as fact. Run the experiment yourself first — pick a topic you know cold, let the model write about it, and count the errors. That exercise teaches more about the real limits of these tools than any amount of vendor marketing or breathless headline ever will, and it costs you nothing but a single wild goat’s dignity.

Written by Smarc

Founder and editor of vo.rs. A lifelong tinkerer who self-hosts far more than is sensible, hardens Linux boxes for fun, and prods the latest AI tools to see what they can really do. The how-to guides here are the notes Smarc wishes had existed the first time round.

Tagged#ai #gpt-2 #nlp #llm

Ibex: anatomy of a GPT-2 hallucination

What a 2019-era language model got wrong about a wild goat, and why it matters

Related Content

Fine-Tuning vs Prompting vs RAG: Picking the Right Tool Without Wasting GPU Hours

Talking to Your Documents: A Practical RAG Pipeline with Open-Source Tools

Prompt Injection: The SQL Injection of the AI Era

What Is Agentic AI, and Why Is Everyone Suddenly Talking About It?