How to use it
- Arrange the data with one row per subject and one column per rater. Rows are separated by semicolons:
9 2 5 8; 6 1 3 2; …. - Choose the ICC form that matches your design.
- Read the ICC, its reliability band and the F test; all six forms are listed for comparison.
Formulas
A two-way ANOVA of the n × k table gives the mean squares between subjects MSR, between raters MSC, the residual MSE, and the within-subject MSW = (SS_raters + SS_error)/(n(k − 1)). Then (Shrout and Fleiss, 1979):
- ICC(1,1) = (MSR − MSW) / (MSR + (k − 1)MSW)
- ICC(2,1) = (MSR − MSE) / (MSR + (k − 1)MSE + k(MSC − MSE)/n)
- ICC(3,1) = (MSR − MSE) / (MSR + (k − 1)MSE)
- ICC(1,k) = (MSR − MSW)/MSR, ICC(2,k) = (MSR − MSE)/(MSR + (MSC − MSE)/n), ICC(3,k) = (MSR − MSE)/MSR
The F test uses MSR/MSW for the one-way forms and MSR/MSE for the two-way forms.
Worked example
The classic example from Shrout and Fleiss has 6 subjects rated by 4 judges: 9 2 5 8; 6 1 3 2; 8 4 6 8; 7 1 2 6; 10 5 6 9; 6 2 4 7. The mean squares are MSR = 11.2417, MSC = 32.4861, MSE = 1.01944 and MSW = 6.26389, which give
- ICC(1,1) = 0.165742, ICC(2,1) = 0.289764, ICC(3,1) = 0.714841
- ICC(1,4) = 0.442797, ICC(2,4) = 0.620051, ICC(3,4) = 0.909316
The judges rank the subjects consistently (ICC(3,1) = 0.71) but disagree a lot on the level of the scores, which is why the absolute-agreement ICC(2,1) is much lower. F(5, 15) = 11.0272, p = 0.0001346.
Tips and common mistakes
- The rows must be subjects and the columns raters, not the other way round.
- Always state the model, the type (agreement or consistency) and single or average measures; the six values can differ a lot, as the example shows.
- Missing ratings are not supported: every subject needs a rating from every rater.