Two rounds of pilot tests for a total of 1000 abstracts were screened by all reviewers to calibrate screening accuracy between reviewers.
To ensure accuracy, two reviewers (CS or MJE, and AAV) will independently abstract all data; discrepancies will be resolved by discussion.
The Super Quiz Resource Guide was formed mostly from articles from peer reviewed journals, but also includes non-peer reviewed articles, which are looked over by a panel of five reviewers and then checked for accuracy by another reviewer.
The predictive accuracies the reviewers mention (from Figure 7) are only moderately lower than that obtained for typical midget cells (e.g. Figure 3C); that difference may be due to the relatively stronger increment-decrement used for evaluating the model, compared to the white noise used to train it, as is now suggested in the text.
One of us (MGW) applied the coding framework to the full-text of all included articles, which was then independently checked for consistency and accuracy by another reviewer (AG), with any changes discussed by both reviewers and a final decision arrived at through consensus.
Finally, diagnostic accuracy of junior reviewers (68%%) without any experience in US was lower than senior reviewers (89 %), although senior reviewers did not have previous experience in evaluating US signs of pneumoperitoneum.
In addition, SWoRD includes algorithms that compute individual reviewer's review accuracy, which is in turn used to support the various drawbacks of reciprocal peer reviews (e.g., variation in motivation or ability of reviewers).
Study data were extracted by two reviewers and checked for accuracy by a third reviewer including the intervention description, inclusion/exclusion criteria, baseline, demographics, and values for all outcomes at baseline and after treatment.
To assess English-speaking reviewers' accuracy in determining the eligibility of foreign-language articles for a systematic review.
Using multiple reviewers increased the accuracy of chart review, they said.
Assuming a hypothetical situation that corresponds roughly to current norms of grant reviews, we estimate a reviewer's accuracy based on the costs and number of reviewers.
