TF-IDF calculator
Paste a few documents, one per line or separated by blank lines, and see which words matter most in each one. TF-IDF gives a high score to words that are frequent in a document but rare in the others, and close to zero to words that appear everywhere.
Runs in your browser: nothing you enter is sent anywhere.
How to use it
- Paste the documents: one per line, or switch to paragraphs if your documents span several lines.
- Choose the TF and IDF variants. The defaults (count ÷ words, ln(N ÷ df)) are the textbook ones.
- Read the table of each document: words sorted by TF-IDF, with count, TF and IDF next to them.
How it works
For each word w in document d:
- TF = count of w in d (or a variant of it);
- df = number of documents that contain w; N = number of documents;
- IDF = ln(N ÷ df);
- TF-IDF = TF × IDF.
Words with the same score keep the order in which they first appear in the document.
Worked example
Three documents: “the cat sat”, “the dog sat”, “the cat ran”. N = 3, every document has 3 words, so each word that appears once has TF = 1 ÷ 3 = 0.3333.
For document 2:
- dog appears in 1 document: IDF = ln(3 ÷ 1) = 1.0986, TF-IDF = 0.3333 × 1.0986 = 0.3662;
- sat appears in 2 documents: IDF = ln(3 ÷ 2) = 0.4055, TF-IDF = 0.1352;
- the appears in all 3: IDF = ln 1 = 0, TF-IDF = 0.
So dog is the word that best describes document 2.
Tips
- Use more documents for meaningful IDF: with two documents every word gets either ln 2 or 0.
- Remove boilerplate (signatures, headers) first, or it will dominate the scores.
- To count words in a single text, use the word frequency counter.
Frequently asked questions
What is TF-IDF?
A weight for a word in a document: term frequency (how often the word appears in that document) times inverse document frequency (how rare it is across all documents). Words that are common in one document and rare elsewhere score highest.
Which formulas are used?
TF is the count, the count divided by the number of words in the document (the default), or 1 + ln(count). IDF is ln(N ÷ df), where N is the number of documents and df the number of documents containing the word, or the smooth form ln((1 + N) ÷ (1 + df)) + 1 that scikit-learn uses by default. The logarithm is natural.
Why do words like the get zero?
With IDF = ln(N ÷ df), a word that appears in every document has df = N and ln 1 = 0, so its TF-IDF is 0 whatever its count. With the smooth IDF it gets a small positive weight instead.
How are words split?
A word is a run of letters and digits; an apostrophe inside a word is kept (don't), other punctuation separates words, and everything is lower-cased. There is no stemming and no stop-word list, so cat and cats are different words.
Why are my numbers different from scikit-learn?
TfidfVectorizer uses raw counts with the smooth IDF and then scales each document's vector to length 1 (L2 normalization). Choose Count and the smooth IDF to get its numbers before normalization.