This highlights the different information each provides, SDs give an indication of the aggregate judgement where as ranges summarise the outliers views.
The amount of convergence and therefore the strength of agreement is indicated by a comparison of SD (strength of aggregate judgement) and range (larger ranges being indicative of outliers views).
Expert Choice™ software translated the comparison judgements according to the 9-point scale of Saaty [41], immediately provided the Consistence Ratio (CR) of each matrix, and calculated normalised performance, aggregating judgements by geometric mean.
Although aggregating judgements will reduce the strength of the outliers views in quantitative data, with subjective views outliers can raise important issues not yet considered by others, and therefore new themes should not be ignored and rounds continued until views are exhausted.
Because quantitative analysis aggregates participants' judgements, outlying individual judgements are not represented.
The Delphi method is an iterative, usually multi-staged, method of aggregating the judgements of a number of individuals, using a series of written questionnaires, in order to arrive at a consensus.
They first refined the statistical definition of what it means for a crowd to be wise – when, exactly, some aggregate of crowd judgements can be considered better than those of selected individuals.
This combines available scientific evidence with the collective judgement of experts by deriving a consensus opinion from a group, with individual opinions aggregated into refined aggregated opinion [ 5, 21, 22].
The new methodology takes account of both quantitative and qualitative data, first aggregating the DMs' individual judgements on the performance of alternatives according to each criterion, and then integrating the criteria in order to determine the relative preference of any two alternatives.
The individual judgements were aggregated statistically to derive the group judgements.
