What "AI" Actually Is (and What the Sales Deck Left Out)

I sat through a vendor pitch last month where the phrase “the AI understands your business” appeared on four separate slides. Nobody in the room asked what that sentence actually meant, so I did, and the answer boiled down to “it’s a language model with our documentation stuffed into its prompt.” That’s a perfectly reasonable product. It is not understanding. The gap between what’s on the slide and what’s running on the server is where most AI disappointment lives, and it’s worth closing that gap yourself before someone else’s marketing closes it for you.

The Word Doing Too Much Work

“AI” used to mean a research field: search algorithms, expert systems, planning, robotics, the whole sprawling attempt to make machines behave usefully in the world. Somewhere in the last decade the word narrowed, in everyday use, to mean one specific family of technique: large statistical models trained on enormous amounts of text or images, tuned to predict the next likely token or pixel. When someone today says a product “has AI,” they almost always mean it calls a large language model, sometimes with some retrieval or tool-use bolted on. Everything else — the classic planning algorithms, the constraint solvers, the actual robotics stack — has quietly stopped being marketed as AI at all, because the LLM boom ate the word.

That narrowing matters because it hides how much of “AI” is really “statistics on a very large scale.” A language model doesn’t consult a model of the world and reason about your business. It has learned, from an enormous corpus, which sequences of words tend to follow other sequences of words, and it generates plausible continuations one token at a time. That process produces genuinely useful output an astonishing amount of the time, because human language encodes a huge amount of structure and knowledge that a big enough statistical model can capture. But “captures the statistical structure of language” and “understands your business” are different claims, and only one of them is true.

A Short History of the Same Old Bait

This isn’t the first time the word has been stretched to cover more than the mechanism justified. In the 1960s, “AI” meant programs that manipulated symbols according to logical rules — expert systems that encoded a domain specialist’s knowledge as a tree of if-then rules. They were genuinely useful for narrow, well-defined tasks like diagnosing certain equipment faults, and genuinely oversold as a path to general machine reasoning. That wave collapsed into what historians call an “AI winter” once the gap between the pitch and the delivered capability became too obvious to ignore.

Chess engines went through their own version of this. Deep Blue beating Garry Kasparov in 1997 was reported, at the time, as a machine “thinking” several moves ahead. It was doing brute-force search through a game tree with hand-tuned evaluation functions, aided by custom silicon — extraordinary engineering, and not remotely the same cognitive act a grandmaster performs. Each generation of the technology gets a fresh round of “the machine finally understands,” and each time the underlying reality is a specific, well-defined computational technique dressed in the vocabulary of a mind. Large language models are the current wave of that pattern. The technique is different and, in fairness, considerably more general-purpose than expert systems or chess search ever were. The marketing instinct to describe it as understanding rather than computing is exactly the same instinct as before.

Three Things Living Under One Umbrella Term

When a product claims to run on AI, it’s almost always one of three distinct things, and distinguishing them tells you what to actually expect.

The first is a base or instruction-tuned language model — the raw predictive engine, trained to continue text plausibly and then fine-tuned to follow instructions and answer helpfully rather than just autocomplete. This is the thing people mean when they say “ChatGPT” or “Claude” generically. It has no persistent memory of you between separate conversations unless a product wraps one around it, no access to live data unless something feeds it in, and no ability to check its own answers against reality unless a separate system does that checking.

The second is retrieval-augmented generation, where the product searches your documents or database for relevant passages and pastes them into the model’s prompt before asking it to answer. This is what most “chat with your data” tools actually do, and it’s a genuinely clever pattern for grounding answers in real content instead of relying purely on what the model memorised during training. It’s also why the same model can seem dramatically smarter or dumber depending purely on how good the retrieval step is — a model fed the wrong three paragraphs will confidently answer from the wrong three paragraphs.

The third is an agent: a language model wrapped in a loop that lets it call tools, read results, and decide what to do next, repeating until it judges the task done. This is the pattern behind coding assistants that can run commands and browsers that can click through a website. It’s the most capable-looking of the three because it can take multi-step action, and also the least predictable, because every step compounds the model’s tendency to occasionally invent a plausible-sounding but wrong answer.

Almost every product pitching “our AI” is doing one of these three things, dressed in language that makes it sound more autonomous and more comprehending than the mechanism underneath actually is.

Where the Slide Gets Loose With the Truth

The sales deck rarely lies outright. It just leans hard on ambiguous verbs. “Understands,” “knows,” “reasons about,” and “learns from your feedback in real time” are the four phrases I’d flag hardest, because each one papers over a specific limitation.

“Understands” almost always means “was given relevant text in its prompt,” not “built an internal model of your domain.” “Knows” usually means “was trained on data that happened to include this,” which is a coin flip depending on how recent or niche the information is. “Reasons about” typically means “produces text that reads like a chain of reasoning,” which correlates with correctness a good deal of the time but is not the same guarantee a human expert’s reasoning gives you. And “learns from your feedback in real time” is the boldest one — most deployed products don’t retrain on your corrections at all; your feedback might get logged for someone to review later, or fed back into the same prompt for the rest of that one conversation, and then it’s gone.

None of this makes the underlying technology a scam. It makes the technology a tool with a specific, learnable shape, and the vendor’s job is to sell you the shape as bigger than it is. Your job, evaluating the pitch, is to ask the boring follow-up question: “when you say it understands X, do you mean it was trained on X, or that X gets included in the prompt every time?” The answer changes what the tool can and can’t do for you, and most salespeople have genuinely never been asked that plainly.

Watching the Machinery Directly

The fastest way to build calibrated intuition is to run a small model yourself and watch what happens when you push on its edges. A local model on ordinary hardware behaves identically, mechanically, to anything running behind a polished product UI — it’s just missing the retrieval and tool layers that make the finished product look smarter.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

$ ollama run llama3.1
>>> What's the capital of the fictional country Wakanda?
Wakanda is a fictional country from the Marvel universe, and its
capital is depicted as Birnin Zana in the comics and films.

>>> What's the capital of Freedonia, from the 1933 Marx Brothers film?
Freedonia's capital, as portrayed in "Duck Soup," isn't given an
explicit name in the film itself — it's simply referred to as
Freedonia throughout.

>>> Now invent a plausible-sounding capital for Freedonia anyway.
A fitting invented capital could be "Marxburg," playing on the
Marx Brothers' name while sounding like a genuine European-style
city name.

Nothing here is understanding in any deep sense. The model is doing exactly what it was trained to do: continue text plausibly given what came before, drawing on patterns from an enormous training corpus and, when pressed, generating a fluent-sounding invention rather than admitting a blank. Ask the same model to summarise a long document you’ve fed it and you can watch the same instinct surface in a subtler, more dangerous form: pushed for a length or level of detail the source material doesn’t support, it will pad with plausible-sounding generalities rather than say the source was thin. That same mechanism, wrapped in retrieval over your company wiki and a nicer chat window, is what gets sold to you as “AI that understands your documentation.” The wrapping is real engineering work and can be genuinely valuable. The word “understands” is doing more marketing than description.

Troubleshooting: Reading Past the Pitch

A few habits catch most inflated claims before you’ve signed anything.

Ask what happens when the model doesn’t know something. A well-built product will say “I don’t have information on that” or cite a source it couldn’t find. A poorly built one will confidently invent an answer, because nothing in the pipeline checks its output against anything. Ask for this specific failure mode in the demo, live — vendors who’ve engineered around it will show you happily; vendors who haven’t will change the subject.

Ask whether the model’s knowledge is static or live. If it’s a base model with no retrieval, its factual knowledge is frozen at whatever point its training data ended, and it will not know about anything that happened after — including changes to your own product, if nobody’s fed those changes back in. If it’s retrieval-augmented, ask how often the underlying index refreshes, because a stale index gives you confidently wrong current answers dressed up as fresh ones.

Ask what data actually leaves your network, and where. “AI” is often used to launder the fact that your documents are being sent to a third-party API for every single query. That may be entirely acceptable for your use case, but it should be a decision you make deliberately rather than one buried under the word “AI” on slide six.

Ask what happens on a bad day — specifically, what the product does when the underlying model’s API is slow or unavailable. A surprising number of “AI-powered” products have no fallback path at all, which tells you the AI component isn’t a feature layered onto a working product; it is the product, with all the fragility that implies.

Finally, run the same tricky question through the product twice in a row. Language models are probabilistic; the same prompt can yield subtly different answers on different runs. A vendor demo is, by definition, the one run that went well. Ask to run it again on something you know the ground truth for, and watch what changes.

Is It Worth Your Attention?

Genuinely, yes — the underlying technology is real, useful, and improving quickly, and dismissing it as hype is as much a category error as swallowing the marketing whole. The move that actually pays off is learning the shape of the mechanism: a predictive text engine, sometimes handed relevant documents, sometimes given tools to act with, always capable of a fluent, confident, wrong answer because nothing in its architecture distinguishes “confident” from “correct.” Once you can see that shape, every pitch deck becomes a lot easier to read, because you’re evaluating an engineering decision instead of a promise.

The practical upshot for anyone actually deploying this stuff: budget time for evaluation against your own real questions, not the vendor’s rehearsed ones, and treat any claim using the words “understands” or “knows” as shorthand for a specific, checkable mechanism rather than a magic property. If you want the mechanics of the piece that decides how much of this costs you and how much context the model can actually hold onto in a single conversation, that’s a related read worth your time: what an LLM context window actually costs you. And if the statistical framing above interests you more broadly, most machine learning is a spreadsheet you won’t admit to makes the same point from the training side rather than the inference side.