The first instrument was a truncated version of the Open Response Instrument (ORI), which was developed using questions from Bishop and Anderson (1990) and Nehm and Reilly (2007).
These differences could not be accounted for by one component of the response instrument, for example, ESR or swollen-joint count, and are likely due more to differences in precision between the two instruments.
For example, Nehm and Schonfeld's (2008) study of students' understanding of natural selection found that an open-ended response instrument revealed that students who could identify correct responses in a multiple-choice format also held alternative conceptions.
The questions we used (Table 2) were variations of questions developed by Bishop and Anderson (1990) for their Open Response Instrument and are similar in form to questions recently described by Nehm et al.
Measuring knowledge and comprehension of concepts with selected-response instruments, such as concept inventories, tells us whether learners can recognize a correct statement among a set of possible answers.
In future experiments, we intend confirming whether students apply the principles of mutation, natural selection, and speciation to explain how real organisms' populations change over time using a variety of assessment methods such as open-response instruments.
Nehm and colleagues' important work on measuring knowledge of natural selection (Nehm and Reilly 2007; Nehm and Schonfeld 2008) very thoroughly addresses the use of multiple assessments, including the CINS and open-response instruments, to measure diversity and frequency of key concepts in student answers about evolution.
We used three different assessment tools (Fig. 1b), each designed to detect different facets of understanding of the five key principles above: the Conceptual Inventory of Natural Selection (CINS; Anderson et al. 2002) and two open-response instruments, which we refer to as the "Dino Problem" and the "Concept Frame".
As with the pretest, the mean post-test score was higher for the selected-response instrument (CINS, 68%) than for the constructed-response instrument (Dino Problem, 41%).
