The authors created a 35-item rubric with ≥0.90 interrater reliability, based on three raters who met to clarify the rubric and gain consensus after beginning to grade and then grading independently.
The ACORNS is designed with a sixteen item scoring rubric that standardizes student responses that differ across contextual variables for evolution (e.g., gain vs. loss of traits, plants vs. animals, within vs. between species differences).
This resulted in a 5-item scoring rubric, which we complemented with an item about searching questions to represent the construct of reflection fully [ 10, 41].
A sample rubric item related to prediction ability is shown in Table 1.
For each assessment item, a conceptual rubric was developed to categorize and score the presence or absence of both hypothesized and emergent themes in the data.
The student must discuss two aspects of engineering practice that favour re-designing the dam, even if this will incur greater cost (for the full item and scoring rubrics, See Tremblay et al. 2012, 261 263).
Reflection scores were calculated as the sum of the scores on the 6 items in the rubric.
The full 72-item inventory with scoring rubric is given in the Supplemental Material, but we provide a few items here as typical examples.
For scoring these open-ended items, we developed a rubric that was approved by the two course instructors, D.Z. and S.G., who are biology professors, and two researchers who hold PhDs in biology education.
For many of the items on the rubrics, there were key terms that raters were instructed to look for, for example, "conservation of matter," "cellular respiration," "photosynthesis," "stomata," and "fitness".
Afterwards item descriptions displayed in the rubric were discussed by the assessors and, when experienced as unclear, revised accordingly.
