I Don't Care About Your 2-Million-Token Context Window

A context window is a capacity number, not a capability number. Most models break 30 to 40 percent below their advertised limit, and they break in cliffs rather than slopes. You are being sold a warehouse and charged rent on the whole thing while you use one shelf.

Every launch cycle, the number gets bigger. 128K became 1 million. 1 million became 2 million. And every launch cycle, I watch clients read that number and conclude something that is not true: that the constraint they have been living with just got lifted.

It did not. I want to explain why, because this is the most expensive misunderstanding in enterprise AI right now, and unlike most of what I write about, it costs you money on two axes simultaneously — you pay more per call and you get worse output for it.

Here is my position, stated plainly so you can disagree with it properly: the size of your prompt is a measurement of how poorly you understand your own process. And a bigger context window does not fix that. It funds it.

The Number on the Box Is Not the Number You Get

Start with the mechanical fact, because the argument rests on it and it is well-documented rather than a matter of opinion.

Effective context length — the length at which a model actually outperforms a cheap baseline rather than merely accepting the input — is systematically shorter than advertised context length. Research surveying the field found that most models overclaim: GPT-4 advertised 128K with an effective context around 64K, and ChatGLM advertised 128K with an effective context around 4K. Not a rounding error. A factor of thirty-two.

The pattern held as windows grew. By early 2026, the working rule was that models break 30 to 40 percent before their claimed limit — a 200K model becoming unreliable around 130K — and that the degradation is sudden rather than gradual. Sharp drops, not smooth decline.

That word — sudden — is the part that should worry you more than the shortfall itself. A slope you can budget around. A cliff you fall off without noticing, because the model does not tell you it has stopped reading properly. It just keeps answering.

A model that has lost the middle of your prompt does not throw an error. It writes you a confident paragraph. That is the whole problem in one sentence.

What the context rot research actually found

The Chroma study is the one I send to clients, because it demolishes the intuition rather than just qualifying it. Running extended needle-in-a-haystack benchmarks across GPT-4.1, the Claude 4 family, Gemini 2.5 and Qwen3, the team found three things that should change how you architect.

• Degradation is non-uniform. Some models hold at 32K and collapse at 64K. Others hold together until they abruptly do not. The documented window size correlates only weakly with how well the model actually uses that window.
• Semantic similarity drives decay more than length does. When the needle is distinct from the haystack, models find it fine. When distractors resemble the answer, accuracy drops sharply — and the drop gets worse as length grows.
• Models performed better on shuffled haystacks than on logically coherent documents, across 18 models. Document flow itself appears to interfere with attention.

Read the second and third findings together, then think about what you actually put in your prompts.

Your knowledge base is not a haystack of random text with one distinct needle in it. It is forty documents about the same subject, written in the same voice, using the same vocabulary, half of them superseded versions of the other half. Every single one of them is a semantically similar distractor to every other one. You have constructed, by accident, the exact worst case the research describes — and then you have made it worse by keeping it in coherent document order.

The distractor effect compounds. A single distractor measurably reduced baseline performance; four compounded it further. And accuracy falls by over 30 percent when the key information sits in the middle of the context rather than at either end — the lost-in-the-middle effect that has been in the literature since 2023 and is still being designed around rather than designed out.

Why Your Benchmark Is Lying To You Specifically

Here is where I get less popular. The benchmarks that reassure you about long context are measuring a task you do not have.

The original needle-in-a-haystack test asks a model to find one planted sentence in a long document. That is single-needle retrieval. Your production workflow is almost never single-needle. It is: read this contract, this policy, this email thread and this spreadsheet, and integrate four facts that live in four places into one decision.

That is multi-needle, and the gap is not small. Analysis across the 2026 frontier field found that single-needle scores overstate production capability by 15 to 40 points. RULER-style reasoning-over-context tests — which are closer to what you actually do — typically run 10 to 25 points below single-needle NIAH for the same model.

So the reassuring number is inflated twice: once because effective context is below advertised context, and again because the benchmark measures finding rather than reasoning.

The uncomfortable conclusion about retrieval

The same analysis reached a conclusion the long-context marketing does not want you to reach: for multi-hop reasoning over long context — legal analysis, research synthesis, technical documentation — retrieval with explicit chunk selection beat naive long-context for nearly every model tested, and even the strongest long-context performer benefited from retrieval above 400K.

Let me translate that out of benchmark language. The thing you were told the 2-million-token window would let you stop doing — the retrieval work, the chunking, the deciding what matters — is the thing that still beats the 2-million-token window at your actual job.

You were sold the big context window as permission to skip the work. The research says the work is what wins. The window just lets you skip it more expensively.

And the academic work keeps sharpening the distinction. Long-context evaluation is now being split explicitly into retrieval tasks versus holistic-understanding tasks, precisely because collapsing them into one score hides where models fail. Unified evaluations comparing RAG against long-context directly exist for the same reason. Nobody serious treats one number as one answer anymore.

The Real Reason This Bothers Me

None of the above is why I am writing this. The mechanics are just evidence. The reason I am writing this is what I watch happen inside businesses when a bigger window ships.

At Hexona we run automation across six continents. The single most common architecture I inherit from a client's previous vendor is this: one enormous prompt, containing everything anyone thought might be relevant, sent to the most expensive model available, on every single call.

When I ask why a particular block of context is in there, the answer is almost never a reason. It is a history. Somebody added it eight months ago because a specific output was wrong once. Nobody removed it, because nobody could prove it was safe to remove. So the prompt grew. It only ever grows.

That is not an AI architecture. That is sediment.

And it is the same failure I described in You Don't Have an AI Problem, You Have a Systems Problem, wearing a different costume. The prompt is enormous because nobody owns the process. Nobody can say which inputs the decision actually requires, so everything goes in, and the model is asked to do the analysis that a human should have done before the model was ever involved.

A 2-million-token context window is the technical permission slip for never deciding what matters. That is exactly why it sells so well.

Big context is the new 'just buy the tool'

I have made the argument that process ownership beats tool delivery, and this is the purest example of it I have found. Dumping the SharePoint into the prompt feels like progress. It is fast, it is demoable, and it produces something on the first try. It is also the precise moment you decide not to understand your own process — and you will pay for that decision every day for the next two years, at a per-token rate.

Compare it to the alternative. Sitting down with the person who actually makes the decision and asking: what do you look at? In what order? What would change your answer? That conversation takes an afternoon. It produces a specification. The specification produces a prompt that is 4,000 tokens instead of 400,000, runs on a cheaper model, and ships clean more often.

Nobody wants that afternoon. Everybody wants the window.

The Context Budget

So here is the framework I now apply on every engagement, and the one I would like you to steal.

You already accept the idea of a latency budget. You would not let a page load time drift from 200 milliseconds to 40 seconds because nobody was watching. Somebody owns that number and defends it.

Do the same thing with tokens.

A Context Budget is a per-workflow ceiling on prompt size, set deliberately, defended actively, and reviewed like any other operational constraint. It has four rules.

Rule 1: Every block of context must have a stated reason

If you cannot say in one sentence why a block is in the prompt — what decision it changes, not what it might theoretically help with — it comes out. 'Just in case' is not a reason. It is an admission.

Rule 2: The budget is set by the task, not by the model

This is task-model matching applied to the input rather than the model. A classification task does not get a 200K budget because the model supports 2 million. It gets what the classification requires. The model's ceiling is not your allowance.

Rule 3: Growth requires justification, the same as spend

Prompts grow the way headcount grows: quietly, one defensible increment at a time, until the total is indefensible and nobody remembers approving it. If a prompt grew this quarter, somebody should be able to say what it bought.

Rule 4: Measure the ratio before and after every increase

This is where it connects to the Automation Ratio — the percentage of AI-assisted outputs that ship without human correction. It is the only metric I have found that predicts ROI, and it is the only honest test of a context change.

Add 100K tokens of context. Did the ratio move? If it did not, you just bought degradation and paid for the privilege. If it went down — which happens more often than anyone admits — you have empirically demonstrated the distractor effect in your own business, and you should remove the context and tell somebody.

Adding context and not measuring the ratio is not engineering. It is hope with a rate card attached.

The Objections, Answered Honestly

'But long context is getting better'

It is. I am not arguing the curve is flat. I am arguing that the advertised number has never once matched the effective number, in any generation, from any lab, and there is no reason to assume this generation is the exception. The right response to a bigger window is not to use more of it. It is to wait for someone independent to measure where the cliff moved to.

And note what is being verified and what is not. When the 2-million-token figure was circulating ahead of launch, there was no model card, no pricing page, and no public model ID backing any of it. A number from a leak is not a specification. A specification is not a benchmark. A benchmark is not your workflow. Three gaps, all of them where the money leaks out.

'But my use case really does need the whole document'

Some do. Genuinely. Full-document legal review, long-horizon code comprehension, research synthesis across a corpus — these are real long-context tasks and I am not pretending otherwise.

But be honest about the ratio of those to the rest of your stack. In my experience it is somewhere around one in twenty. The other nineteen workflows are routing, classification, extraction, drafting and summarisation — tasks with a small, knowable input set that got a giant prompt because giant was the default.

This is the same distribution I found writing about why most tasks need a rule rather than an agent. The genuinely hard cases are real, and they are rare, and the entire industry prices and architects as though they were typical.

'But tokens are cheap'

They are cheaper than they were and more expensive than they are about to be. Sonnet 5 introductory pricing expires August 31. Standard pricing and the tokeniser multiplier land September 1. The Fable 5 grace period ends September 30, with credits-only pricing from October 1.

And that is before the hardware. The memory crisis has been reshaping the cost floor under all of this for months. Anyone who built an architecture on the assumption that input tokens are free is about to discover they were financing a habit, not a capability.

Meanwhile, 30 to 46 percent of US enterprise API tokens are already routing to Chinese models on price. If your prompt is ten times larger than it needs to be, you are not choosing between vendors. You are choosing to pay ten times the bill at whichever vendor you land on.

'But trimming context might break something'

It might. That is what the Automation Ratio is for. Trim, measure, keep or revert. This is a one-afternoon experiment with a number at the end of it, and the reason it does not happen is not risk. It is that nobody owns the prompt.

What I Would Actually Do

If you run an automation stack and today's launch cycle made you wonder whether you should be doing something differently, here is the something. None of it requires a new model.

• Print your five largest prompts. Not read — print. Length is abstract on a screen and undeniable on paper.
• For each block of context in each one, write the sentence that justifies it. The blocks with no sentence are your first deletion.
• Set a Context Budget per workflow, in tokens, with a name attached to it. An unowned budget is a wish.
• Measure the Automation Ratio per workflow before you trim, so you have a baseline that means something.
• Trim to the budget, re-measure, and record the delta. Most teams find the ratio holds or improves. That result is worth more than any benchmark you will read this month.
• Where a workflow genuinely needs a corpus, build retrieval with explicit chunk selection rather than pouring the corpus into the window. The research says it wins; my invoices agree.
• Re-run the whole exercise the next time a lab doubles a number. The number will double again. Your process will not have changed.

If your team does not have a repeatable way to do this, start with the workflow discipline rather than the tooling — the no-code automation workflow guide covers the sequencing I use before any model is chosen.

The Bottom Line

I have never once seen a business fail at AI because its context window was too small. Not once, on any continent, in any industry.

I have seen dozens fail because nobody could say what their process actually needed, so they gave the model everything and hoped. The window did not cause that. But every time the number doubles, the pressure to do the hard part drops a little further, and the bill goes up a lot.

Eighty percent of executives report no measurable AI ROI. I have written at length about why, and this is one of the load-bearing reasons: they bought capacity when they needed clarity. Those are not the same purchase, and only one of them is for sale.

The endpoint of AI-first operations does not look like enormous prompts. It looks like Bending Spoons — $2.57 million of revenue per employee, 90 percent of pull requests AI-generated, and humans doing specification and judgment only. Specification. That is the whole discipline. Knowing what the task needs, precisely, before you hand it to a machine.

Boring, disciplined moves compound. A 4,000-token prompt that ships clean beats a 400,000-token prompt that ships confident. One of those has a number behind it. The other has a spec sheet.

So no, I don't care about your 2-million-token context window. I care what percentage of what you shipped last week needed a human to fix it. Tell me that number, and I will tell you whether your context window was ever the constraint.

It was not.

Frequently Asked Questions

What is the difference between advertised and effective context length?

Advertised context is the token ceiling the model will accept. Effective context is the length at which it still outperforms a cheap baseline. Survey research found most models overclaim substantially — GPT-4 advertised 128K against roughly 64K effective, ChatGLM advertised 128K against roughly 4K. The 2026 working rule is that models break 30 to 40 percent below the claimed limit, and that the failure is a cliff rather than a slope. Plan against the effective number, which only independent testing gives you.

Is RAG dead now that context windows are enormous?

No, and the evidence points the other way. Across the 2026 frontier field, retrieval with explicit chunk selection beat naive long-context for multi-hop reasoning on nearly every model tested, and even the strongest long-context performer improved with retrieval above 400K. Long context and retrieval are not rivals. Retrieval is how you control what enters the window, which is the only lever you actually hold.

Why would adding more context make my output worse?

Two documented mechanisms. First, distractor interference: the Chroma study found that semantically similar but irrelevant content actively misleads the model, and the effect worsens with length. Your business documents are semantically similar to each other by definition. Second, positional bias: accuracy falls by over 30 percent when key information sits in the middle of the window. Adding context does not just dilute — it actively competes with the signal.

How do I set a Context Budget if I have never measured this?

Baseline your Automation Ratio per workflow first — the percentage of outputs shipping without human correction. Then halve the prompt and re-measure. If the ratio holds, halve it again. You are searching for the point where the ratio actually moves, and that point is your budget. Most teams are shocked at how far down it sits. Do this per workflow, never in aggregate; aggregate hides which processes work.

Are there workloads that genuinely need a million-plus tokens?

Yes — full-document legal review, long-horizon code comprehension, and research synthesis across a corpus are real long-context tasks. In my experience they are roughly one workflow in twenty. The other nineteen are routing, classification, extraction, drafting and summarisation, which have small knowable input sets and got giant prompts because giant was the default. Architect for the nineteen and handle the one deliberately.

Doesn't a bigger window at least reduce engineering effort?

It reduces the effort you can see and increases the effort you cannot. You skip the retrieval build and inherit an unaudited prompt that grows every quarter, degrades unpredictably, and bills per token forever. That is not less work. It is deferred work at interest — the same pattern behind 74 percent of agent deployments being rolled back: complexity deployed ahead of the ability to govern it.

What is the single change with the highest return?

Print your largest prompt and write, next to each block of context, the sentence explaining what decision it changes. Blocks without a sentence get deleted. It costs an afternoon, requires no procurement, no vendor and no migration, and in every engagement I have run it has been the highest-return hour of the quarter. If that feels too simple to matter, that is the point — it is why nobody does it.

Related Reading

Continue the argument:

1. Stop Chasing the Biggest Model — task-model matching, and why the frontier is rarely the answer.

2. The Automation Ratio: The Metric That Predicts Survival — the only honest test of whether a context change helped.

3. You Don't Have an AI Problem, You Have a Systems Problem — the failure this article is a special case of.

4. You Don't Need an Agent, You Need a Rule — minimum complexity as a design principle, applied to architecture.

5. The 'AI Business' Advice Is Wrong: Tool Delivery vs Process Ownership — why dumping the knowledge base in feels like progress.

6. Satya Nadella's Learning Loop Warning — what happens when systems stop teaching you about your own process.

7. Gartner's $206 Billion AI Agent Spending Forecast for 2026 — the scale of the spend that this discipline is protecting.

About the Author

Hamza Baig is the founder of Hexona Systems, an AI automation agency operating across six continents, and the AI Automation Institute, where he has trained more than 40,000 entrepreneurs in practical AI systems design.

He has been featured in the GHL Top 50, Yahoo Finance and Brainz Magazine. His work focuses on the gap between what AI can do in a demo and what it reliably ships in production — the Automation Ratio, task-model matching, and governance-first architecture.

Read more analysis on the Hamza Automates blog, get in touch about your automation stack, or follow @hamza_automates on Instagram for daily breakdowns.

Research referenced: Chroma context rot benchmarks; Hsieh et al. effective-context analysis; Liu et al. lost-in-the-middle; Kamradt needle-in-a-haystack; U-NIAH and Dolce long-context evaluation work; 2026 frontier long-context retrieval analysis. Opinions are the author's own.

Further reading on the evaluation methodology: U-NIAH unified RAG and long-context evaluation, Dolce retrieval-versus-holistic-understanding task differentiation, and the enterprise platform context these decisions now sit inside.