This compares to a macro-average positive judgement of 0.62 for all three annotators.
The overall judgement (average global rating) was 7.6 (SD = 0.69) on the first and 7.8 (SD = 0.60) on the last assessment, indicating a significant trend towards improvement (T = −2.1, df = 103, p < 0.05).
Each pair of doctors was judged by 48 assessors, and therefore the binomial chance expectation is that on average the correct judgement for a particular pair of doctors will be made by 24 out of the 48 assessors.
The average of individual judgements were compared with the team consensus judgements in a paired design.
At which point the rule of the average, which requires all judgements to be equally right, starts to shake, and we, the independent readers, can breathe again.
The pairs of application forms were judged by 96 assessors, 48 of whom looked at Book 1 and 48 looked at Book 2. A chance null hypothesis predicts that random guessing would result in an assessor making an average of 10 correct judgements out of 20 (50%).
In our experience, if we compared the average likelihoods directly, the separation judgement was not stable.
Boycott, who scored 1,082 runs in 34 one-day innings for England, at an average of 36.06, was passing judgement on a poor sequence of form that has seen Morgan dismissed for less than 20 in nine of his last 11 one-day international innings.
For each examiner an overall Angoff estimate was calculated as the average of all of their judgements for that exam.
Average scores, composed of two raters' judgements for each condition and patient, were computed separately for articulatory quality and pitch accuracy.
