The 6-week interview schedule was modified slightly for the 6-month and 12-month interviews (e.g., deleting questions regarding length of stay, changing the stem of questions to reflect the appropriate data collection point).
While the stems of questions can be fairly cleanly categorized in this way, the distracters often encompass inappropriate mixed practices, such as matter energy conversions or scale mistakes.
Grubisic (2012); Grubisic et al. (2013) followed a similar approach to Cubric and Tosic by defining a set of question stem templates which assess students' cognition at the knowledge, comprehension, application and analysis levels.
The stem of the questions read: "To what extent do you need further knowledge and skills concerning the following?
All of the ambiguities were related to the stem of the questions, and the predicted answers were clear for the participants.
The same items were also completed for advice on weight loss, but in this case the stem of the questions specified 'cancer patients who are overweight'.
Except for two items, all other questions that respondents had problems with in the focus group (20 of 297), had unclear translations; the ambiguity was related to the stem of the questions and the predicted answers were clear for the participants.
The interview process with students was vital to the rewording of question stems, and several questions and their answers were improved through direct student feedback.
A corollary to these questions is whether in testing problem solving, it is the construction of question stems that is important, as opposed to the number of alternatives or examination format.
Thus, it is unlikely that the change of mixing method affected the results of the stem in question.
