Proposed in a seminal paper (Zemla et al., 2001) as a benchmarking criterion at early stages of Comparative Assessment of Structure Prediction (CASP), this measure has been successfully used by many independent assessor groups (Kinch et al., 2003; Kopp et al., 2007; Wang et al., 2005; Zemla et al., 2001).
In this work, a benchmarking criteria for a worst process stream is established through risk assessment.
Three performance curves and benchmarking criteria derived thereof were used: ROC curves [35], cumulative accuracy [9, 23], and predictiveness curves [36 39].
The issues include dealing with test set bias caused by, for example, inappropriate normalization procedure, devising appropriate benchmarking criteria and formulating statistically robust feature-selection techniques.
13321_2017_230_MOESM1_ESM.docx Additional file 1. Detailed information about the classification methods, model validation, benchmarking criteria, the comparison between ROC and CA curves and the influence of RUS CV on ROC curves of novelty measures.
In addition, this work has identified challenges in the domain, including terminology diversity and inconsistency, limited documentation, sparse comparison and benchmarking criteria, occasional immaturity of solutions and lack of support, and non-existence of a standard query language.
Consequently, this study has identified challenges in the field, including the immense diversity and inconsistency of terminologies, limited documentation, sparse comparison and benchmarking criteria, and nonexistence of standardized query languages.
Finally, nowadays, benchmarking criteria may not be based of self-reported outcomes but on outcome data gathered by 'coders'coders
2. The methodical part mixes basal points (e.g. how works PLS) with more peripheral ones (e.g. different benchmarking criteria such as stability plots etc).
The area under the receiver operating characteristic curve (AUC ROC) was employed as main benchmark criterion.
The primary benchmark criterion for the success of the AD measure is the area under the ROC curve (AUC ROC).
