How to use it
- Enter the number of parameters in billions and pick the precision the weights are stored in.
- Copy layers, KV heads and head dimension from the model’s configuration.
- Set the context length you plan to use and how many sequences run at once.
- Pick the KV cache precision (16-bit unless your runtime quantizes the cache) and an overhead allowance.
- Enter the memory of your GPU, or the sum of several, to see whether it fits and the longest context that does.
Formula
- Weights = parameters × bytes per parameter (4, 2, 1 or 0.5 bytes).
- KV cache = 2 × layers × KV heads × head dimension × context × batch × bytes per value.
- Total = (weights + KV cache) × (1 + overhead ÷ 100).
- Longest context = (memory ÷ (1 + overhead ÷ 100) − weights) ÷ (KV cache per token × batch), rounded down.
Models with grouped-query attention have fewer KV heads than attention heads, which is why the KV heads field is separate: use the KV figure, not the total number of heads.
Worked example
A model of 7 billion parameters at 16 bits, 32 layers, 8 KV heads of dimension 128, context 8192, one sequence, 10 % allowance:
- weights 7 × 10⁹ × 2 = 14 GB;
- KV cache per token 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB, for 8192 tokens 1,073,741,824 bytes = 1.074 GB;
- allowance 10 % of 15.074 GB = 1.51 GB;
- total 16.58 GB, which is 15.44 GiB.
With 24 GiB available it fits, and the longest context is (25,769,803,776 ÷ 1.1 − 14 × 10⁹) ÷ 131,072 = 71,923 tokens.
Tips
- Precision matters most. Going from 16 to 4 bits cuts the weights to a quarter; the KV cache only shrinks if the runtime stores it at lower precision too.
- Long contexts and batches grow linearly. Doubling the context or the batch doubles the KV cache, so a model that fits at 8k tokens may not at 128k.
- Splitting across GPUs adds memory, but each card also needs its own buffers: keep the allowance generous.
- This is an estimate from the formulas above, not a measurement; real use depends on the runtime and its settings.