How to use it
- Copy the layer count, hidden size, attention heads, KV heads, feed-forward size and vocabulary size from the model configuration.
- Choose gated or plain feed-forward, RMSNorm or LayerNorm, and rotary or learned positions.
- Switch on “tied” when the output layer reuses the embedding matrix, and “biases” when linear layers have them.
- Read the total and the breakdown.
Formula
With head dimension h = d_model ÷ heads and KV width k = KV heads × h:
- attention per layer = 2·d_model² + 2·d_model·k (Q and O, then K and V), plus 2·d_model + 2·k with biases;
- feed-forward per layer = 3·d_model·d_ff (gated) or 2·d_model·d_ff (plain), plus biases if any;
- norms per layer = 2 × d_model (RMSNorm) or 2 × 2·d_model (LayerNorm has a bias too);
- embeddings = vocabulary × d_model, twice when the output head is not tied, plus positions × d_model for a learned table;
- total = layers × per layer + embeddings + one final norm.
Worked example
The default shape: 32 layers, d_model 4096, 32 heads (head dimension 128), 8 KV heads (KV width 1024), gated feed-forward of 14,336, RMSNorm, vocabulary 32,000, untied, rotary positions.
- attention 2 × 4096² + 2 × 4096 × 1024 = 41,943,040;
- feed-forward 3 × 4096 × 14,336 = 176,160,768;
- per layer with two norms of 4096: 218,112,000, × 32 = 6,979,584,000;
- embedding and output head 2 × 32,000 × 4096 = 262,144,000; final norm 4096.
Total 7,241,732,096 parameters, 7.242 billion, which is 14.48 GB at 16 bits.
Tips
- Embeddings weigh more in small models. With a large vocabulary they can be a big share of the total; the “without embeddings” row shows the rest.
- Memory follows from the count. Multiply by 2 bytes for 16-bit weights or 0.5 for 4-bit; the LLM VRAM calculator adds the KV cache.
- Mixture-of-experts models have several feed-forward blocks per layer; this calculator counts one, so multiply the feed-forward figure by the number of experts yourself.