The modified Angoff method had an inter-rater reliability of 0.81 – 0.82 and a test-retest reliability of 0.59 0.74.
The ICC is exactly identical to the weighted kappa with quadratic weights [6] and is an appropriate method to determine an inter-rater agreement between multiple raters of multiple questions with a large rating scale as is the case in the current study.
Looking strictly at the variability on an intra-rater basis was the focus of our determination of appropriate methods to be evaluated on an inter-rater basis.
A prospective cohort study including an inter-rater reliability study.
Overall, an inter-rater agreement of 81% was reached.
All the tests were administered by raters with a masters degree in clinical psychology; an inter-rater reliability was established.
An inter-rater and intra-rater reliability (IRR) system was developed using a modified version of the EDC.
We were able to apply a uniform approach to the detection of possible suicide related events across hundreds of studies and used a validated method with proved inter-rater reliability for classification of events.
Rater agreement is important in clinical research, and Cohen's Kappa is a widely used method for assessing inter-rater reliability; however, there are well documented statistical problems associated with the measure.
The method to establish inter-rater reliability with these two protocols is described elsewhere (Lund et al. 2015).
