—
Details
How to use KV Cache Calculator
Configure decoder attention and cache precision to estimate key/value tensor memory. Calculations run locally and update as you type.
- Choose MHA, GQA or MQA and enter model architecture values.
- Set the cache precision, cached tokens and simultaneous sequences.
- Read the tensor payload, per-token memory and formula; allow separately for runtime overhead.
How it’s calculated
Bytes = 2 × layers × KV heads × head dimension × cached tokens per sequence × concurrent sequences × bits per cached element ÷ 8. The factor 2 represents keys and values with equal dimensions and precision.
All layers are assumed to use dense, full-context self-attention and the same geometry. Token count includes prompt and generation; for a preallocated static cache, enter the reserved maximum rather than the current used length.
The result excludes model weights, activations, allocator padding, page/block tables, quantization scales and zero points, temporary dequantization, replication, and runtime workspaces. Packed 4-bit is an ideal payload lower bound. Sliding windows, hybrid layers, cross-attention, prefix sharing, tensor parallelism, latent attention and unequal sequence lengths need model-specific accounting.
Sources
- Cache behavior and allocation assumptions. Hugging Face cache strategies
- KV tensor geometry. Transformers paged attention documentation
About KV Cache Calculator
During autoregressive decoding, attention reuses stored keys and values for earlier tokens. This calculator estimates their dense tensor payload across equal-length concurrent sequences. Query heads affect the result through the selected attention arrangement; GQA stores the KV heads, not an expanded copy for every query head.
MHA uses one KV head per query head. MQA uses one KV head, and GQA accepts a divisor of the query-head count. Cache precision is independent of model-weight precision. The calculator uses integer arithmetic for exact byte totals and presents both binary GiB and decimal GB.
Questions
Does changing query heads change GQA memory?
For fixed KV heads and head dimension, query heads do not change the stored dense cache size. They must be divisible by the KV-head count. MHA uses all query heads as KV heads.
Is this total GPU memory usage?
No. It is the aggregate KV tensor payload. Model weights and runtime overhead consume additional memory, and device sharding or replication changes per-device requirements.
Can I enter quantized cache precision?
Yes. Choose 8 or 4 bits. This is ideal packed data only; implementation-specific scales, metadata and temporary buffers must be added separately.