For ease of comparison, we conducted the item-type analyses on the proportion of items correct.
We evaluated tumor characteristics, the diagnosis field including terminology and immunohistochemistry (IHC) workup, and report completeness (the proportion of items recorded).
Similarly, power rates are computed as the proportion of items that are simulated as DIF items that are also flagged as having DIF.
To compare pre- and posttest scores visually, we calculated the proportion of items about key concepts and misconceptions that students answered correctly in each class.
Type I error rates are reported as the proportion of items within a replication that were generated as non-DIF items but which are flagged as DIF in the analysis.
To detect low effort in low-stakes testing, Wise and Kong (2005) developed a continuous measure of test taking effort called response time effort (RTE) as the proportion of items completed with solution behavior.
As shown in Figure 2, the following 6 of 14 items were observed to have high risk of bias, where the proportion of items categorized as 'No' was high (more than 50%): adequate spectrum (item 1), adequate reference standard (item 3), partial verification (item 5), differential verification (item 6), incorporation bias (item 7), and description of reference test execution (item 9).
Low risk of bias, where the proportion of items categorized as 'Yes' was high (more than 50%), was observed in the following five items: selection criteria (item 2), description of the index test (item 8), index test review (item 10), uninterpretable results (item 13), and withdrawals (item 14).
RIF is ultimately measured as the difference between the proportion of items recalled between these two types.
The proportion of item-level missing data was low (≤ 4%).
The proportion of items in the probe, irrelevant and target categories was manipulated across experiments.
