Fleiss's kappa
Fleiss's kappa is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items or classifying items. This contrasts with other kappas such as Cohen's kappa, which only work when assessing the agreement between two raters or the intra-rater reliability. The measure calculates the degree of agreement in classification beyond what would be expected by chance. It is named after American biostatistician Joseph L. Fleiss, who introduced it in a 1971 article.
Fleiss's kappa can be used with binary or nominal-scale. It can also be applied to ordinal data : the MiniTab online documentation gives an example. However, this document notes: "When you have ordinal ratings, such as defect severity ratings on a scale of 1–5, Kendall's coefficients, which account for ordering, are usually more appropriate statistics to determine association than kappa alone." Keep in mind that Kendall's rank coefficients are only appropriate for rank data.
Introduction
Fleiss's kappa is a generalisation of Scott's pi statistic, a statistical measure of inter-rater reliability. It is also related to Cohen's kappa statistic and Youden's J statistic, which may be more appropriate in certain instances. Whereas Scott's pi and Cohen's kappa work for only two raters, Fleiss's kappa works for any number of raters, giving categorical ratings, to a fixed number of items, under the condition that for each item, raters are randomly sampled. It can be interpreted as the extent to which the observed agreement among raters exceeds what would be expected if all raters made their ratings completely at random. It is important to note that whereas Cohen's kappa assumes the same two raters have rated a set of items, Fleiss's kappa specifically allows that although there are a fixed number of raters, different items may be rated by different individuals. That is, Item 1 is rated by Raters A, B, and C; but Item 2 could be rated by Raters D, E, and F. The condition of random sampling among raters makes Fleiss's kappa not suited for cases where all raters rate all patients.Agreement can be thought of as follows: If a fixed number of people assign numerical ratings to a number of items, then the kappa will give a measure of how consistent the ratings are. The kappa can be defined as,
The factor gives the degree of agreement that is attainable above chance, and gives the degree of agreement achieved above chance. If the raters are in complete agreement, then. If there is no agreement among the raters, then.
Definition
Let be the total number of elements, let be the number of ratings per element, and let be the number of categories into which assignments are made. The elements are indexed by and the categories are indexed by. Let represent the number of raters who assigned the -th element to the -th category.First calculate, the proportion of all assignments which were to the -th category:
Now calculate, the extent to which raters agree for the -th element :
Note that is bound between, when ratings are assigned equally over all categories, and, when all ratings are assigned to a single category.
Now compute, the mean of the 's, and, which go into the formula for :
Worked example
| 1 | 2 | 3 | 4 | 5 | ||
| 1 | 0 | 0 | 0 | 0 | 14 | 1.000 |
| 2 | 0 | 2 | 6 | 4 | 2 | 0.253 |
| 3 | 0 | 0 | 3 | 5 | 6 | 0.308 |
| 4 | 0 | 3 | 9 | 2 | 0 | 0.440 |
| 5 | 2 | 2 | 8 | 1 | 1 | 0.330 |
| 6 | 7 | 7 | 0 | 0 | 0 | 0.462 |
| 7 | 3 | 2 | 6 | 3 | 0 | 0.242 |
| 8 | 2 | 5 | 3 | 2 | 2 | 0.176 |
| 9 | 6 | 5 | 2 | 1 | 0 | 0.286 |
| 10 | 0 | 2 | 2 | 3 | 7 | 0.286 |
| Total | 20 | 28 | 39 | 21 | 32 | - |
| 0.143 | 0.200 | 0.279 | 0.150 | 0.229 | - |
An example of using Fleiss's kappa may be the following: consider several psychiatrists who are asked to look at ten patients. For each patient, fourteen psychiatrists give one of five possible diagnoses. These are compiled into a matrix, and Fleiss's kappa can be computed from the following table to show the degree of agreement between the psychiatrists above the level of agreement expected by chance. The subject and the raters sampled from a larger group are assigned a total of five categories. The categories are presented in the columns, while the subjects are presented in the rows. Each cell lists the number of raters who assigned the indicated subject to the indicated category.
In the following table, given that,, and. The value is the proportion of all assignments that were made to the th category. For example, taking the first column and the second row, respectively:
In order to calculate, the sum of is,
Over the whole sheet,
Interpretation
gave the following table for interpreting values for a 2-annotator 2-class example. However, this table is by no means universally accepted. They supplied no evidence to support it, basing it instead on personal opinion. It has been noted that these guidelines may be more harmful than helpful, as the number of categories and subjects will affect the magnitude of the value. For example, the kappa is higher when there are fewer categories.| Condition | Interpretation | |
| Subjective example: only for two annotators, on two classes. | < 0 | Poor agreement |
| Subjective example: only for two annotators, on two classes. | 0.01 - 0.20 | Slight agreement |
| Subjective example: only for two annotators, on two classes. | 0.21 - 0.40 | Fair agreement |
| Subjective example: only for two annotators, on two classes. | 0.41 - 0.60 | Moderate agreement |
| Subjective example: only for two annotators, on two classes. | 0.61 - 0.80 | Substantial agreement |
| Subjective example: only for two annotators, on two classes. | 0.81 - 1.00 | Almost perfect agreement |