The Kappa value of agreement of nurses and doctors with the AEP criteria was calculated using Cohen's Kappa.
Also Kappa-crosstabs plot individuals' rank from one round against their rank in the next round, therefore K-values measure the value of agreement for individuals between two rounds, not agreement between different participants within a round.
Nevertheless, this apparent virtue can paradoxically be altered by a skewed prevalence and a systematic one-sided variation between the ratings, which can convert a high value of agreement into a low value of kappa [ 38, 39].
Regarding the values of agreement, these were calculated by determining the frequency of predictions in the ensemble which were correct (i.e., matching the observed class).
Low kappa values of agreement between the human raters can be attributed to different strategies of distinguishing between system-generated questions and human-generated questions.
In addition, this study analyzed the sensitivity, specificity, and positive and negative predictive values of agreement between retrospectively-reported headaches experienced at least every other day, frequent medication, and prospective information recorded in headache diaries.
It was possible to compare the Norwegian study's 'sensitivity values' to our values of agreement according to the prescription database, and to compare the 'PPV values' to our values of agreement according to the CA registry.
The values of agreement between outputs from each monitor generation when assessing mcpm is shown in Table 2.
We rated the results for one performance category to be consistent if at least 75% of the extracted values of agreement coincided; otherwise, inconsistency was stated.
As presented by Byrt and colleagues [ 43], this is a manifestation of two paradoxes where, for identical values of agreement, the kappa coefficient can be unduly reduced or increased.
Kappa, alpha and delta have higher values of agreement for asymmetrical imbalance (scenarios 5 and 7) than for symmetrically imbalanced marginal totals (scenarios 4 and 6), contrary to what is desired.
