For comparisons of rating the methodological quality we calculated chance-corrected agreement (using kappa) and chance-independent agreement (using Φ) [52], [55], [56].
Currently, there are no clear guidelines available for rating the methodological quality of the performed IRT analyses.
The COSMIN checklist provides a framework of criteria for rating the methodological quality of research investigating the reliability, validity, responsiveness and interpretability of health measurement instruments.
Rating the methodological quality of eligible studies was performed independently by two review authors (CEC and AJQP) using the Newcastle Ottawa scale [ 17], which involved assessing the methods of participant selection, comparability and outcomes among eligible studies.
A review of papers meeting the eligibility criteria will be conducted by the same reviewers who judged eligibility, but this time rating the methodological quality of the primary research.
Last, the distinction between rating the methodological quality of the study and rating the quality of the instrument that is evaluated in the study may be difficult, especially for content validity.
