If a prompt is the specification you give a model, then the context window is the size of the whiteboard you’re allowed to write that spec on. In the previous article we treated a prompt as “input text plus intent.” Now we’ll zoom out and ask a deceptively simple question:

How much text can the model actually pay attention to at once?

Modern LLMs boast wild numbers here. OpenAI’s GPT‑4.1 and 4.1 mini advertise context windows up to around 1 million tokens. Anthropic’s latest Claude Sonnet models and Google’s Gemini 1.5/2.x family similarly reach 1M–2M tokens in some configurations.

Those numbers sound like “infinite memory.” They’re not. This article is about what a context window really is, how to budget it, what breaks at the edges, and when you should stop throwing more tokens at the problem and switch to RAG or fine‑tuning instead.

The short answer

A context window is the maximum number of tokens (roughly sub‑word chunks) a model can use in a single request. That budget includes:

• Your input: instructions, system prompt, chat history, retrieved docs, tool outputs…
• The model’s output: every token it generates in that response.

If your model has a 128k token window and you send a 100k‑token prompt, you cannot ask for a 60k‑token answer. You’re already near the limit with just the input.

Crucially:

• Context is working memory, not long‑term memory. Once a token falls outside the window, it no longer influences the model’s next token.
• Larger windows are powerful but come with cost and latency penalties, and recall quality can degrade as you approach the limit.

The rest of this guide is about treating that window like a budget instead of a mystery.

What a context window is (input + output)

Let’s get precise.

Tokens, not characters

Models operate on tokens, not characters or words. In English, a rough rule of thumb is:

• 1 token ≈ 0.75 words, or
• 1 token ≈ 3–4 characters of typical English text.

So a 1,000‑token prompt is roughly a few paragraphs; 100k tokens is a decent‑sized report or a medium codebase.

Production SDKs often give you token counters or estimators; use those instead of guessing.

One window, two directions

The context window counts everything involved in one completion call:

1. Prefill phase (input side) The model reads all the prompt tokens: system messages, user messages, retrieved context, tool traces, etc. This cost scales with input length.

2. Decode phase (output side) The model generates output token by token. This cost scales with output length. For many applications, output length dominates total latency.

The sum of input tokens + output tokens must be ≤ the model’s context limit. Many vendors phrase this explicitly in their docs.

Chat history and “memory”

In chat‑style APIs, “memory” is just previous messages you resend:

System: You are a meticulous legal summarizer...
User: Summarize this contract: <20 pages of text>
Assistant: <summary>
User: Compare that contract to this one: <20 more pages of text>

Every turn, the provider’s client or your own backend is bundling up that entire exchange as a sequence of messages and re‑sending it - until something has to give.

When the total token count approaches the limit, UIs or SDKs will:

• Throw an explicit “context length exceeded” error, or
• Silently truncate older messages (dangerous if you’re not aware).

This is why context windows are not “memory size” in a human sense. They’re just the slice of the conversation that still fits on the whiteboard right now.

Model‑specific limits (why they differ)

Different model families have different context windows, and the numbers can change by:

• Model size and architecture,
• Provider,
• Deployment flavor (API vs chat UI vs on‑prem),
• Even subscription tier in some hosted products.

Some current examples:

• OpenAI GPT‑4.1 / 4.1 mini / 4.1 nano: up to about 1M tokens of context in the API (exact usable size and quotas depend on deployment).
• OpenAI o‑series: reasoning‑optimized models like o1 / o3 often ship with 128k–200k token windows and separate output caps (e.g. tens of thousands of tokens).
• Anthropic Claude: Claude 3.5/4.x Sonnet typically exposes 200k+ token windows, and newer Claude Sonnet 4 variants have been announced with up to 1M tokens for coding‑heavy workloads.
• Google Gemini 1.5+: models like Gemini 1.5 Pro and Flash support 1M–2M token contexts, with dedicated docs just on “long context.”

Why the differences?

• Architectures: vanilla Transformer attention scales poorly with sequence length; newer designs and tricks (FlashAttention, sparse patterns, context caching) make longer windows feasible, at a cost.
• Training & eval: not all models are actually good at long‑context reasoning, even if they technically accept that many tokens. Some papers and provider blogs show accuracy dropping as context grows, especially on “needle‑in‑haystack” tasks.
• Hosting constraints: a model might support 200k tokens via API, but the public chat UI caps you at ~32k and silently manages the rest.

Bottom line: don’t assume your chat interface exposes the same limits as the underlying model. Always check the model docs and your specific deployment.

Budgeting: prompt, retrieved docs, and expected output

Treat your context window like a project budget with multiple line items:

1. System / role prompt and global rules
2. Tool schemas and function definitions
3. Conversation history
4. Retrieved docs / knowledge snippets
5. Misc instrumentation (e.g. logit bias hints)
6. Expected output

Your job is to make the important stuff fit comfortably.

A simple budgeting formula

Let:

• C_max = model context limit (e.g. 128k),
• S = system + tool definitions (tokens),
• H = chat history you keep,
• D = retrieved docs + extra context,
• O_max = max output tokens you allow.

Then you must ensure:

S + H + D + O_max ≤ C_max

In practice, you also want a safety margin:

S + H + D + O_max ≤ 0.8 * C_max

This leaves headroom for tokenization quirks and things like hidden tool traces.

Rough percentage heuristics

For many interactive apps on ~128k windows, a workable starting budget is:

• 10–20% for system prompt + tools
• 20–30% for chat history
• 40–50% for retrieved docs / context
• 10–20% for the response itself

For huge 1M–2M contexts, you rarely want to fill the whole window; latency and cost will explode. Providers consistently recommend minimizing total tokens to reduce both cost and response time.

Two key production facts:

• Output dominates latency: A Databricks and OpenAI guidance both note that average latency scales roughly with the number of output tokens times a per‑token generation time.
• Prompt size affects time‑to‑first‑token: The entire prompt must be processed before the first token appears; as context grows, TTFT can become the main bottleneck.

So if your app feels slow, the cheapest levers are:

• Reduce expected output length.
• Aggressively cut or compress docs and history.
• Keep the system prompt as lean as you can.

Truncation vs. summarization vs. compression

When things no longer fit, you have three main options. Each has different failure modes.

1. Truncation: “chop it off”

This is the default behavior in many SDKs and UIs: if tokens exceed the limit, drop something:

• Oldest chat messages,
• The tail of a document,
• Occasionally entire tools or examples.

Pros:

• Simple, fast, no extra cost.
• Works okay when older content is genuinely low‑value (e.g. early small talk in a long chat).

Cons:

• You can silently drop critical context (constraints, caveats, edge cases).
• Debugging becomes painful: the model is “ignoring” something you think you sent, but it never sees it.

Use truncation when:

• You’re trimming obviously irrelevant history,
• Or you’ve explicitly designed your prompt to make older content expendable.

2. Summarization: “shrink it in human words”

Instead of just dropping text, you can ask a model (often a smaller or cheaper one) to summarize:

• Long chat history → dialogue summary,
• Long docs → topic‑wise summary,
• Logs or telemetry → aggregated stats.

This keeps the salient facts in fewer tokens, at the price of an extra step.

You’ll see this in:

• “Chat with docs” tools that maintain a rolling conversation summary rather than keeping the full history,
• Multi‑turn assistants that periodically condense “what we’ve decided so far.”

But summarization is lossy:

• Important caveats may vanish.
• Subtle instructions (“don’t mention X to the user”) can get mangled.
• Repeated summarization of summaries can distort things badly.

To mitigate that, use structured summaries:

Role: Conversation summarizer.
 
Task: Compress the chat so far into <= 500 tokens.
 
Format:
- Goals: ...
- Decisions made: ...
- Open questions: ...
- Constraints / policies: ...
- User preferences: ...
 
Preserve anything that affects future decisions, even if it seems minor.

Done well, this is usually better than blind truncation for important state.

3. Compression: “store the gist as data”

Compression is a wider toolbox that includes:

• Question‑focused summaries: summarize “for future Q&A about X”, not generically.
• Key–value memory: extract facts into tuples (entity, attribute, value, source) and store them in a database.
• Embeddings / RAG: ingest large text, then later fetch only the most relevant chunks per query.

This is where the boundary between “prompt engineering” and “system design” blurs. You aren’t just shaping the text; you’re changing how knowledge is stored and retrieved.

Compared to summarization:

• Compression can preserve more fine‑grained details and be more query‑aware.
• It usually requires more infrastructure (vector DBs, feature stores, etc.).

In practice, robust systems use a mix:

• Truncate clearly irrelevant history,
• Summarize medium‑importance state,
• Compress high‑value knowledge into structured or retrievable form.

Chunk‑and‑order strategies for long inputs

Let’s say you really do need to feed a big document, codebase, or transcript to the model. Even with a large window, the order and chunking of that content matter a lot.

Long‑context research and vendor guidelines converge on a few key patterns.

Choose sensible chunk sizes

For RAG and long‑context tasks, a common range is:

• 500–2,000 tokens per chunk for text,
• Smaller, more semantic segments for code (per file, class, or module).

Chunks that are too big are hard to retrieve accurately and waste tokens; too small and you lose coherence.

Order: question near the end, context around it

Multiple experiments (and provider docs) point to a simple rule:

Ask the question after you’ve given the context, and keep the question close to the relevant snippets.

A common layout:

System / tools
Global instructions
 
[High-level summary of the corpus]
 
[Chunk 1]
[Chunk 2]
...
[Chunk N]
 
User question:
<your task here>

Where:

• The high‑level summary helps the model maintain global structure.
• The question at the end makes it easy for the model to “attend backward” to what matters.

For multi‑doc tasks, group chunks by topic or section instead of random order, so attention patterns have something coherent to latch onto.

Map–reduce patterns

When you’re near the edge of the window, do the work in two (or more) passes:

1. Map: For each chunk, ask a small model or the same model to produce a structured mini‑summary or analysis.
2. Reduce: Feed only those summaries (and maybe a handful of original chunks) into a second pass that synthesizes the final answer.

This pattern underlies many “analyze this 400‑page PDF” tools and shows up in provider long‑context guides as a way to balance quality, cost, and latency.

The key idea: don’t shove everything into one giant prompt by default. Use chunking and ordering to give the model clearer structure.

Testing visibility (edge‑case prompts)

How do you know what the model can still “see” inside a long prompt? You test it.

Anthropic’s “needle in a haystack” experiments for Claude’s long context window are a nice inspiration: they hide a fact in a long document, then ask the model to find it. You can run similar tests for your own stack.

The secret‑token test

1. Create a weird phrase, e.g. MAGIC-UNICORN-472.
2. Paste it in a paragraph at the very beginning of a long context.
3. Fill the rest with dummy text or real docs.
4. End with:

At the very beginning of the text I gave you, there was a secret code phrase.
Reply with ONLY that exact phrase, nothing else.

5. Increase total length until the model starts failing.

Then repeat with the secret phrase in the middle and end of the prompt. You’ll often see differences in recall as you move it around.

The numbered paragraph test

Build a prompt like:

You will see numbered paragraphs 1–N.
Exactly one of them contains the phrase "BANANA-42".
Your task: reply ONLY with the number of the paragraph that contains it.
 
<paragraph 1: ...>
<paragraph 2: ...>
...
<paragraph N: ...>

You can auto‑generate this in a script and grow N until the model breaks. Log the total tokens and you have an empirical visibility curve for your specific model + deployment.

Why bother?

Because marketing claims like “1M tokens of context” don’t guarantee perfect recall across all that space. Empirical studies and provider docs repeatedly note that accuracy depends heavily on prompt structure, token position, and query framing.

Your own tests tell you where “safe operating zones” really are.

UI tips (show token counts, progress)

If you’re building tools (like PractiqAI’s prompt‑training tasks), your UI can make or break how users interact with context limits.

Show token counts (or at least a bar)

At minimum, surface:

• Approximate tokens in the current prompt,
• Percentage of the model’s window used.

For example:

“Using ~9,400 / 32,768 tokens (29% of window). Estimated response: 1,000 tokens.”

Even a simple traffic‑light bar (green < 50%, yellow 50–80%, red > 80%) helps users self‑regulate.

Visualize what’s included

When you add retrieved docs and history, show what actually made it into the prompt:

• A collapsible “Context” section listing included documents with sizes.
• Badges like “3 of 7 retrieved docs included (max token budget reached).”
• A “Show truncated content” notice if something was dropped.

That transparency prevents a lot of “why did it ignore my document?” confusion.

Separate “thinking” from “speaking”

Behind the scenes, inference has two phases: prefill (processing the prompt) and decode (generating tokens).

You can reflect that in the UI:

• Stage 1: “Analyzing input (processing 84k tokens…)”
• Stage 2: “Generating answer…”

This both educates users and makes long‑context latency feel less mysterious - especially when TTFT is large.

Let advanced users tune output length

Provide a “max answer length” slider or presets: “short / medium / long / exhaustive.” Internally, these map to different max_tokens or output budgets.

OpenAI, Claude, and Gemini docs all emphasize that shortening outputs is one of the most reliable ways to cut latency and cost. Surfacing that control in the UI makes the trade‑off explicit.

Failure modes at the limit

Things get weird before you actually hit a hard “too many tokens” error. Some common failure modes:

1. Silent truncation

SDKs or UIs may trim older messages or tail content without telling you. Suddenly:

• The model “forgets” earlier instructions,
• Or contradicts decisions made “a long time ago.”

ChatGPT‑style products sometimes operate with a much smaller effective context per chat than the underlying model’s API limit. Always assume some pruning is happening in long sessions.

2. Partial or aborted responses

At large contexts and output sizes, you might see:

• Responses stopping mid‑sentence,
• Sudden generic endings (“…and so on”),
• Tool timeouts.

Community reports for long‑context Gemini and others show responses timing out or aborting once prompts rise above ~100k tokens, long before the theoretical 2M‑token limit.

3. Instruction drift

As you stack more docs and history, global instructions (e.g. “Always answer as JSON”) can get drowned out:

• The model starts ignoring formatting constraints,
• Safety rules fall out of scope if they live only in old messages,
• Answer style changes mid‑response.

This isn’t just truncation; attention itself becomes less focused in very long contexts, and multiple studies find accuracy can degrade at the high end of the window.

Mitigation:

• Repeat critical constraints close to the question.
• Keep a short, sharp system prompt; don’t bury it under noise.

4. RAG performance plateau

Long‑context RAG studies show that simply pouring more retrieved text into the window doesn’t always improve QA performance; beyond a certain point, accuracy can flatten or even drop.

Often:

• Fewer, more relevant chunks + good prompt design beats
• A giant blob of vaguely related text.

When to prefer RAG or fine‑tuning

At some point, you have to decide: should I just buy a bigger context window, or is there a better architecture?

When long context alone is enough

Good candidates:

• Analyzing a single long artifact: a contract, a research paper bundle, a log file, a notebook.
• “One‑shot” data extraction where all relevant info is in that one bundle.
• Code reviews on a bounded codebase that comfortably fits in a 128k–200k window.

Here, RAG might be overkill; a powerful long‑context model (e.g. GPT‑4.1 or a 1M‑token Claude or Gemini) plus good chunking / ordering can be enough.

When to reach for RAG

RAG (Retrieval‑Augmented Generation) shines when:

• Your knowledge base is large and growing (docs, wiki, tickets, logs),
• You need freshness (documents updated daily),
• You want grounded answers with citations,
• Different users see different subsets of data.

Instead of jamming your entire corpus into the context window, you:

1. Store documents as embeddings + text,
2. At query time, retrieve the top‑k relevant chunks,
3. Feed only those chunks + question into the model.

Even with 1M‑token windows, RAG is usually cheaper, faster, and more controllable than “send everything.”

Where fine‑tuning fits

Fine‑tuning is rarely a substitute for RAG; it’s a complement:

Use fine‑tuning when you want to bake in:

• Style and tone (“draft in our brand voice”),
• Output formats and templates,
• Domain‑specific reasoning patterns (e.g. how your company evaluates risk).

It doesn’t magically upload your entire Confluence into the model’s weights - that’s still better handled via RAG or long context. But a fine‑tuned model + RAG often lets you:

• Use shorter prompts,
• Get more consistent structure,
• Spend fewer tokens per request.

In short:

• Use long context when the problem is “one big thing.”
• Use RAG when the problem is “many things, changing often.”
• Use fine‑tuning when the problem is “same kind of answer, over and over.”

Quick worksheet: design your token budget

Let’s turn all this theory into a small worksheet you can actually fill out for your app.

Grab a notebook (or a PractiqAI task 😉) and jot down answers.

Step 1: Pick your model and limit

• Model: ___________________
• Official context window (C_max): ______ tokens

Check your provider’s docs for the exact number and any per‑deployment quirks.

Step 2: Decide on a safety margin

• Safety margin (%): 10–30% is typical
• Usable context (C_use) = C_max × (1 − margin)

Example: If C_max = 128,000 and margin = 20%, then C_use = 102,400.

Step 3: Estimate your output

What’s the longest answer you’re willing to tolerate?

• Max answer length (words): ______
• Approx tokens (O_max) = words × 1.3 (rough heuristic)

Write down O_max = ______.

Step 4: Budget your scaffolding and history

Estimate:

• System + tools (S): ______ tokens
• Max chat history you’ll keep (H_max): ______ tokens

Tip: start small. Many apps work fine with only the last 3–6 turns plus a compact summary of earlier decisions.

Step 5: Derive your document budget

Your remaining budget for retrieved docs and extra context:

D_max = C_use − O_max − S − H_max

Write down:

• D_max = ______ tokens

If you want to include, say, up to N_docs docs of ~doc_size tokens each:

N_docs ≤ floor(D_max / doc_size)

For example, if D_max = 60,000 and your average chunk is 2,000 tokens, you can safely include about 30 chunks. In practice, you might cap at 10–15 to keep noise low.

Step 6: Define trimming rules

Finally, specify clear policies:

• When history exceeds H_max, I will:

• [ ] Truncate oldest turns

• [ ] Summarize old turns into a rolling state

• When retrieved docs exceed D_max, I will:

• [ ] Keep only the top‑k highest‑score chunks

• [ ] Merge overlapping / duplicate chunks

• Critical instructions I will repeat near the question:

Once you’ve written this down, you have a concrete context policy instead of vibes.

Turning context windows into a skill (with PractiqAI)

Context windows sound abstract until you have to debug a real failure:

• The model “forgets” a constraint,
• Ignores half a PDF,
• Or blows up latency because the prompt quietly grew 5× over a week.

The fastest way to internalize these trade‑offs is to practise on realistic tasks with objective feedback - exactly what PractiqAI is designed for.

You’re given tasks with:

• A clear goal and success criteria,
• A limited context budget (prompt + docs + answer),
• A judge model that checks whether the output actually meets the conditions,
• Optional subtasks that reward you for things like “return only the code” or “stay under X tokens.”

You get points, feedback, and - after enough practice - certificates that reflect real capability, not just theory. As you climb those courses, budgeting context stops being an annoying detail and becomes a reflex:

• You naturally keep system prompts lean,
• You chunk and order long inputs sensibly,
• You feel when to use truncation, summarization, or RAG.

If the previous article taught you how to write good prompts, this one is about learning to size them properly. Put both together, and you stop treating LLMs like magic boxes and start treating them like powerful - but finite - tools with clear limits you can design around.

Now go pick a model, run the worksheet, and maybe try a “secret‑token” test or two. Your future self, staring at a misbehaving 1M‑token prompt at 2 a.m., will be very grateful.