Context Window Calculator: KV Cache & VRAM by Context Length

Context window and KV cache reference

Native maximum context and FP16 KV cache size for one request slot at common context lengths, alongside the size of the weights at Q4_K_M. Add roughly 0.75 GB of runtime overhead for the total. Use the calculator above for other quantizations, Q8_0 KV cache, or multiple parallel slots.

See what hosted models cost per token in the model directory, or put two side by side with the model comparison tool.

Tokens, words, and pages

Context windows are measured in tokens, not words. A token is a chunk of text from the model's tokenizer — often a whole common word, sometimes a fragment. For English prose, one token averages about three quarters of a word, so a 32K window holds roughly 24,000 words (about 50 pages) and a 128K window roughly 95,000 words (about 190 pages). Source code, JSON, and non-English text tokenize less efficiently, so budget more tokens for them. Every tokenizer is different; treat these as planning figures.

How much context does your workload need?

Chat and support assistants usually sit comfortably in 8K–16K: a system prompt and a few turns of history. RAG over company documents needs room for the retrieved passages on top of the question — 16K–32K is typical, and retrieving more chunks is only useful if the window can hold them. Coding agents read files, diffs, tool output, and their own plan, so 32K is a practical floor and many tools work better at 64K or more. See running Claude Code against a local model for a real example. Long-document analysis — contracts, transcripts, codebases — is where 128K windows earn their memory cost.

Allocating more context than you use still reserves the memory. Size for the workload, then confirm the whole footprint — weights, cache, and overhead — with the VRAM calculator.

What is the KV cache?

When a transformer generates text, every token it has already seen produces a set of attention keys and values at every layer. Rather than recompute those for the whole sequence on each new token, the model caches them — this is the KV cache. It is the working memory that lets the model "remember" your prompt and its own output as it streams a response.

The weights of the model are fixed: a 7B model at Q4_K_M is roughly the same size whether your context is 2K or 200K. The KV cache is the part that scales with the conversation. That is why a model that loads fine at 4K can run out of memory the moment a coding agent asks for a 32K context.

Why does it grow linearly with context?

For standard grouped-query attention (GQA), the cache stores one key vector and one value vector per token, per layer, per KV head. Hold the architecture fixed and the only variable is the number of tokens — so memory is directly proportional to context length. The rough formula is:

Double the context, double the cache. Go from 8K to 128K and the cache is 16× bigger. This tool computes it per-model using each model's real layer count, KV-head count, and head dimension, and accounts for sliding-window layers that cap their cache at the window size.

GQA vs MLA: some models compress the cache

Not every architecture pays the full GQA price. DeepSeek-style models use multi-head latent attention (MLA), which stores a single compressed latent vector per token per layer instead of a full key/value pair for every head. The result is a dramatically smaller KV cache at the same context length — sometimes an order of magnitude less. The calculator uses each model's declared attention type, so an MLA model will show a much flatter growth curve than a comparable GQA model.

The --parallel gotcha

llama.cpp's --parallel flag splits the context window across concurrent request slots. If you set a 32K context with --parallel 4, each slot only gets 8K — but the server still allocates KV cache for all four slots at once. The memory cost multiplies by the number of slots. Set the parallel-slots field above to see this: serving four users at 8K each costs the same KV memory as one user at 32K.

How does this relate to Wide Area Intelligence?

Every node you add to Wide Area Intelligence has its own context window setting(default 4096). Coding agents and long-document workflows typically need 32K, which is exactly where the KV cache starts to matter. Use this calculator to confirm a node's GPU has the headroom before you raise the context window, then route requests to it. The gateway speaks the OpenAI API at https://wideareaai.com/api/v1, and the X-WAI-Node header pins a request to the node you sized here.

Add a node and set its context window →

Frequently asked questions

What is a context window?

The context window is the maximum number of tokens a model can consider at once — your system prompt, the conversation or documents you send, and the response it generates all count against it. Anything beyond the window is truncated or must be summarized.

How many words fit in a 128K context window?

As a rough rule for English prose, one token is about three quarters of a word, so 128K tokens is roughly 95,000 words — around 190 pages at 500 words per page. Source code, non-English text, and data such as JSON tokenize less efficiently, so fewer words fit.

Does a bigger context window need more VRAM?

Yes. The model weights stay the same size, but the KV cache grows linearly with the number of tokens in context. At long contexts the cache can rival or exceed the weights, which is why a model that loads at 4K can run out of memory at 32K or 128K.

How can I run a long context on a smaller GPU?

Quantize the KV cache to Q8_0 (roughly halves it), reduce --parallel so fewer slots each reserve their own cache, pick a model with sliding-window or MLA attention, or use a smaller weight quantization to free room for the cache.

Is the model's advertised maximum context the one I should use?

Not necessarily. Allocate the context your workload actually needs. Chat rarely needs more than 8K–16K, coding agents typically want 32K or more, and long-document work may justify 128K. Every extra token of allocated context costs memory whether you use it or not.