Generally, inter-rater reliability was slight to substantial, with ICC values ranking between 0.19 and 0.86 (Table 3).
Reliability was slight (i.e., K of 0.00 to 0.15) for the majority of items (representativeness of the exposed cohort, selection of the non-exposed cohort, demonstration that outcome of interest was not present at the start of study, comparability of cohorts, and adequacy of follow-up of cohorts).
Overall, intra-rater reliability ranged from slight to almost perfect with ICC values between 0.14 and 0.82.
Regarding reliability, only a slight decrease in reliability was found by one research group, which they attributed to the 'novelty' of the test format [ 4].
Evaluation of a number of quantitative indices of item and test quality revealed certain threats to validity such as weak discriminating items, moderate reliability and a slight detectable departure from scale unidimensionality.
According to the guidelines for interpretation of Kappa by Landis and Koch [87], these mean reliabilities indicated a slight IRR.
By and large, ICC inter-rater reliability values showed only slight to fair reliability for the NET (Table 3).
We interpreted the intermediate values according to Landis and Koch (1977): values of less than 0.01 indicate poor reliability; 0.01 to 0.20, slight reliability; 0.21 to 0.40, fair reliability; 0.41 to 0.60, moderate reliability; 0.61 to 0.80, substantial reliability; and more than 0.80, excellent reliability.
Controlling for age effects did not have a noticeable effect on the reliability coefficients, with only slight decreases in the size of the correlation for some variables.
Using the qualitative interpretation scale of ICC's as proposed by Landis and Koch [ 23], these reliability scores range from slight (ICC between 0 .20) to barely substantial (ICC between 0.61 and 0.80).
