How to use it
- Enter the parameters and training tokens, both in billions (2 trillion tokens = 2000).
- Enter the number of GPUs, their peak TFLOP/s and the expected MFU.
- Read the total compute, the time and the GPU-hours; compare the tokens per parameter with the Chinchilla value.
Formulas
- Training compute C = 6 × N × D FLOPs
- Effective speed per GPU = peak × MFU
- GPU-hours = C ÷ (peak × MFU) ÷ 3600; wall-clock time = GPU-hours ÷ GPUs
- Petaflop/s-days = C ÷ (10¹⁵ × 86,400)
- Chinchilla-optimal tokens ≈ 20 × N; inference ≈ 2N FLOPs per generated token
Worked example
A 7-billion-parameter model trained on 2 trillion tokens with 512 GPUs at 312 TFLOP/s and 40 % MFU:
- C = 6 × 7 × 10⁹ × 2 × 10¹² = 8.4E22 FLOPs (972.2 petaflop/s-days);
- each GPU delivers 312 × 0.4 = 124.8 TFLOP/s, so the run needs 8.4 × 10²² ÷ 1.248 × 10¹⁴ ÷ 3600 = 186,966 GPU-hours;
- on 512 GPUs that is 365.2 hours, about 15.22 days;
- 2000 ÷ 7 = 285.71 tokens per parameter — 14 times the Chinchilla-optimal 140 billion tokens.
Tips
- Add a margin. Restarts after failures, evaluation and checkpointing typically add tens of percent to the ideal time.
- Check memory separately. Compute says how long; whether the model fits needs parameters, gradients and optimizer states — see the model parameter calculator for N.
- Multiply GPU-hours by your hourly price per GPU for a cost estimate.