At this stage, it is important to determine whether trade-offs between different criteria are acceptable, so that good performance on one criterion can in principle compensate for weaker performance on another.
Engineers describe this situation as one where trade-offs have to be made: in judging the merit of one option relative to other options, a relative bad performance on one criterion can be balanced by a relatively good performance on another criterion.
Table 2 shows the mean rubric scores of students' post-test performance on criteria related to micro-macro thinking skill.
He further stated that the relationship between criterion-referenced evidence with future performance on a criterion is not exact and precise because the criterion should be validated like the test itself (Messick 1988, p. 9).
This finding is, however, in line with the findings of Dunning et al. (2004), who also found that for novice students knowing the assessment criteria does not necessarily imply the ability to assess their own performance on those criteria.
The content of a CQ-index questionnaire typically consists of questions regarding the frequency with which quality criteria were met (never, sometimes, usually, always) and the extent to which performance on quality criteria has raised problems (big problem, small problem, no problem).
By drawing upon representative data from Australia, we illustrate the various methods and evaluate their likely performance on these criteria.
Given the input, the MCDA allowed ranking different scenarios relying on their performance on 13 criteria covering economic, job market, quality of life of local populations, technical and environmental issues.
The matrix also quantifies the detailed performance on all criteria as well as trade-offs.
A core component of any MCDA is the performance matrix which scores all programme options in terms of their performance on relevant criteria.
