What an LLM Context Window Actually Costs You
The invisible metre running under every long AI conversation
Contents
A colleague pasted an entire project’s documentation into a chat assistant, asked one question, and was baffled when the bill for that single exchange came back higher than a week of normal use. Nothing went wrong. That’s just what a context window costs when you fill it, and almost nobody explains the mechanism before you’re the one holding the invoice. Once you understand what’s actually being paid for, both the cost and the model’s occasional forgetfulness stop feeling mysterious.
What the Window Actually Is
A context window is the maximum amount of text — measured in tokens, not words or characters — that a language model can hold in view at once during a single request. Every message you’ve sent, every reply the model has given, any document you’ve pasted in, and the model’s own new response all have to fit inside this one shared budget. There is no separate long-term memory sitting behind it; the “conversation” you experience is an illusion maintained by resending the entire transcript, or as much of it as fits, with every single new message.
This is the detail that trips people up. When you send message five in a long chat, the system isn’t quietly recalling messages one through four from some persistent store. It is very likely re-sending the full text of messages one through four, plus your new message, as one long prompt, every single time. The model has no idea a “conversation” is happening at all in the way a human would mean it; it only ever sees one long block of text and predicts what comes next in that block. Continuity is a product decision built on top of a stateless engine, not a native feature of the engine itself.
Why This Is Also a Billing Mechanism
Because the entire transcript gets resent each turn, and because most providers charge per token processed, the cost of a long conversation grows with the square of its length in the worst case: turn ten pays to reprocess turns one through nine as well as generating a new reply, and turn twenty pays to reprocess nineteen prior turns. A short back-and-forth costs very little. A long working session, or a single message with a large document pasted in, can cost far more than intuition suggests, because you’re not paying for your one question — you’re paying to have the entire history read again, in full, by a very large model.
| |
Providers do apply optimisations — caching of repeated prefixes, for instance, can make resending an unchanged system prompt or document cheaper on subsequent turns than the naive maths above suggests. But the underlying shape holds: length compounds, and a single enormous paste at the start of a conversation gets re-billed, in full or in part, on every turn after it, for as long as it stays in the transcript being sent.
Input and Output Are Priced Differently, on Purpose
Nearly every provider charges a different rate for input tokens (everything sent to the model) than for output tokens (everything the model generates), and output is almost always the pricier of the two. The reason is mechanical rather than arbitrary. Processing a long input can be done substantially in parallel across the tokens, because the model already has the whole input in hand and can compute over it as a block. Generating output cannot be parallelised in the same way, because each new token depends on every token generated before it — the model must complete one forward pass per output token, in strict sequence, before it can start the next. Long inputs are cheap to churn through; long outputs are an inherently sequential, and therefore slower and costlier, process.
This asymmetry has a practical consequence worth internalising: asking a model to “be thorough” and write you a very long answer is often a bigger cost driver than pasting in a moderately large document and asking a short, pointed question about it. If you’re trying to control spend, trimming your expected output length usually buys more than trimming your input.
Where the Window’s Limit Actually Bites
The context window’s size is a hard ceiling, not a soft suggestion. Once a conversation’s total token count — everything sent plus everything already generated — exceeds the model’s window, something has to give. Different products handle this differently: some simply refuse the request and tell you to shorten it; many silently truncate, dropping the oldest messages from what actually gets sent to the model while still displaying your full scrollback in the chat window. The second behaviour is the one that causes genuine confusion, because from the user’s side nothing looks wrong — the messages are still there on screen — while the model has, in fact, already lost access to them and is quietly answering as if they never happened.
This is also why a model can appear to “forget” something you told it earlier in a long session even though you never opened a new chat. It didn’t forget in any cognitive sense; the message carrying that information fell outside the window that got sent on this particular turn. Long working sessions with a lot of pasted code or documentation are the most common place this bites, because a few large pastes can consume most of the available window on their own, leaving little room for the actual conversation before older turns start dropping off the end.
The Middle of a Long Prompt Gets Skimmed
Hitting the hard ceiling isn’t the only way a large context window disappoints you. Even well inside the limit, models tend to attend to information at the very start and the very end of a prompt more reliably than information buried in the middle — a pattern researchers call “lost in the middle.” Paste in a fifty-page document and ask a question whose answer sits on page twenty-five, and you’ll get worse results, on average, than the same question asked about a shorter document where that fact sits near the top or bottom. The window technically holds all fifty pages. The model’s effective attention to the least prominent parts of that window is measurably weaker than its attention to the edges.
The practical fix is to structure what you send rather than trust the window to treat every token equally. Put the single most important instruction or fact either right at the start or right at the very end of the prompt, closest to where the model is about to start generating. If you’re pasting a long document and asking about one specific section, consider extracting and repeating that section near your actual question rather than relying on the model to find it unaided somewhere in the middle of a much longer paste.
The Headline Number Versus the Usable Number
Providers increasingly advertise eye-catching maximum window sizes — several hundred thousand tokens, sometimes over a million — and it’s worth treating that headline figure the same way you’d treat any other spec on a sales sheet: as the theoretical ceiling, not the number that describes your actual experience. A model can technically accept a million tokens and still perform noticeably worse on a needle-in-a-haystack retrieval task at nine hundred thousand tokens than it does at ninety thousand, for exactly the lost-in-the-middle reasons above. The maximum window tells you what the model won’t reject outright. It tells you nothing about how reliably it will use every part of what you send.
This gap between the advertised ceiling and the practically useful range is also where cost surprises compound with quality problems. Filling most of a huge window because the provider says you can costs a large amount in reprocessed tokens on every subsequent turn, and there’s a reasonable chance the model is skimming a meaningful fraction of what you paid to send. The economical and the accurate strategy point the same direction here: send what’s relevant, not what fits.
Sizing a Local Model’s Window Yourself
If you run models locally, the context window becomes a parameter you set yourself, bounded by your hardware rather than by a vendor’s product decision, and it’s worth understanding the trade-off explicitly rather than just maxing it out.
| |
A larger window means more memory reserved before you’ve even sent a message, because the model allocates a key-value cache proportional to the maximum context length it might need to handle, not the length of what you’ve actually sent so far. That cache holds the intermediate representation of every token in the current context so the model doesn’t have to recompute it from scratch on each new token — useful for speed, expensive for memory. Doubling the configured window roughly doubles that reserved overhead on many implementations, which is a real and immediate cost even before a single token of conversation happens. On a GPU with limited VRAM, setting the window far larger than any single conversation will realistically need is a common and avoidable way to run out of memory, and it buys you nothing if the model attends poorly to the middle of that window anyway.
Troubleshooting Long Conversations
If a long-running chat starts giving answers that ignore something you established early on, the window has very likely already dropped that message from what gets sent, or it’s still technically included but buried in a part of the prompt the model is skimming. The fix isn’t to repeat yourself and hope; it’s to restate the relevant fact closer to your current message, or start a fresh conversation with a short, deliberately written summary of the state you need it to have, rather than the entire meandering history.
If costs on a hosted API are climbing faster than the number of questions you’re asking would suggest, check what’s actually being resent each turn. A chat product that keeps the full transcript in every request, including a large document pasted at the start, will keep re-billing that document on every single subsequent message for as long as it stays in the window. Summarising the document once, replacing the original paste with that summary, and continuing from there can cut the ongoing cost dramatically without losing the substance of what it said.
If a local model becomes visibly slower as a conversation lengthens, that’s the sequential nature of generation compounding with a growing input: more tokens to process before the first new token can even be produced. Trimming old turns you no longer need, rather than letting the transcript grow indefinitely, keeps response times sane.
If you’re building anything on top of an API rather than just chatting through a UI, log the actual token counts the provider reports for each call during development. Estimating token counts from word counts is only ever a rough guide — different tokenisers split text differently, and code, unusual formatting, and non-English text can all inflate the real count well past what a word count would suggest.
Is It Worth Understanding This Detail?
Yes, and not just for the accounting. The context window explains, mechanically, some of the strangest-seeming behaviour people report from AI products: apparent memory loss, costs that don’t track with how much you feel like you asked, and long sessions that gradually get both slower and less accurate. None of that is a mysterious flaw in the model’s intelligence. It’s the direct, predictable consequence of a stateless engine being made to look conversational by resending an ever-growing transcript, priced per token, against a hard ceiling, with weaker attention to whatever ends up buried in the middle. Once you can see the metre running, you stop being surprised by the bill, and you start making deliberate choices — trimming, summarising, restating, placing the important fact at the edge rather than the middle — instead of accidental ones. For the wider picture of what’s actually happening inside the box doing this processing, what “AI” actually is (and what the sales deck left out) covers the mechanism this piece assumes. And if you’re weighing whether to run models like this yourself rather than through a hosted API, local AI on your own metal running LLMs with Ollama is the practical starting point.
Related Content
