Cross-entropy and log loss calculator
Paste a true distribution P and a predicted distribution Q to get the cross-entropy, the entropy of P, the Kullback–Leibler divergence and the perplexity. Or switch to binary labels to get the log loss of a classifier's predicted probabilities, in nats, bits or base 10.
Runs in your browser: nothing you enter is sent anywhere.
How to use it
- Choose the mode. Two distributions: P is the true distribution over classes, Q the predicted one. Binary labels: P holds 0/1 labels, Q the predicted probabilities of class 1.
- Paste the numbers in the same order, separated by spaces or commas.
- Choose the logarithm: natural (nats, as in most ML libraries), base 2 (bits) or base 10.
- Entropy H(P) = −Σ pᵢ log pᵢ (terms with pᵢ = 0 add nothing).
- Cross-entropy H(P, Q) = −Σ pᵢ log qᵢ.
- KL(P‖Q) = H(P, Q) − H(P) = Σ pᵢ log(pᵢ ÷ qᵢ) ≥ 0.
- Perplexity = base^H(P, Q).
- Log loss = −(1/N) Σ [yᵢ log qᵢ + (1 − yᵢ) log(1 − qᵢ)].
Worked example
The true class is the first of three, P = 1, 0, 0, and the model predicts Q = 0.7, 0.2, 0.1, natural log:
- H(P, Q) = −ln 0.7 = 0.356675.
- H(P) = 0, so KL(P‖Q) = 0.356675.
- Perplexity = e^0.356675 = 1 ÷ 0.7 = 1.4286.
Log loss for labels 1, 0, 1, 1 and predictions 0.9, 0.1, 0.8, 0.4: (0.105361 + 0.105361 + 0.223144 + 0.916291) ÷ 4 = 0.337539, accuracy 75 %.
Tips
- Match the base to your framework. PyTorch and TensorFlow report losses in nats; information theory books use bits. 1 nat = 1.4427 bits.
- One confident mistake dominates. A single prediction of 0.01 for a true label adds 4.6 nats to the sum.
- Distributions must add up to 1. Softmax outputs do; raw scores (logits) do not and must be normalised first.
Frequently asked questions
What is cross-entropy?
H(P, Q) = −Σ p·log q measures how many nats (or bits) you need on average to encode outcomes from P using a code built for Q. It is smallest, and equal to the entropy H(P), when Q = P. In machine learning it is the standard loss for classification.
How is it related to KL divergence?
KL(P‖Q) = H(P, Q) − H(P). The entropy of the true labels does not depend on the model, so minimising cross-entropy is the same as minimising KL divergence. With one-hot labels H(P) = 0 and the two are equal.
What is log loss?
The mean binary cross-entropy over a dataset: −(1/N) Σ [y·ln q + (1 − y)·ln(1 − q)], where y is the 0/1 label and q the predicted probability of class 1. Lower is better; always predicting 0.5 gives ln 2 = 0.693.
What does perplexity mean?
Perplexity = base^cross-entropy, the effective number of equally likely choices the model is torn between. A language model with perplexity 20 is, on average, as unsure as picking among 20 words.
Why is the loss infinite when q is 0?
Because log 0 is minus infinity: the model was certain an event could not happen, and it happened. Libraries avoid this by clipping probabilities to a tiny range such as 10⁻¹⁵ … 1 − 10⁻¹⁵; this calculator reports it instead.