At very low cutoff thresholds, the ROCE metric decreases again.
The cutoff thresholds as well as the values for WhatIf and MolProbity were taken from Forrest [6].
Using a grade of cutoff thresholds, an ROC plot is created, by plotting false negatives against false positives.
Thus, the sum of the positive TPs, above cutoff thresholds set at ≥moderate, was defined as the TPC in this study.
The CCR metric is finding its highest values at larger cutoff thresholds in combination with high model qualities, making it less suitable for early-recognition problems.
We explored unbalanced species prevalence by investigating the effects of down-sampling the more prevalent response (presence or absence), and by optimizing the cutoff thresholds for declaring a species present.
The REF metric might also be an option to some extend but some cautions are warranted when used in combination with cutoff thresholds χ that are equal or larger than the fraction of true actives in the entire dataset.
Focusing on the ROCE metric, maximum values are calculated when models of high qualities are combined with cutoff thresholds χ that are smaller than 2.5%, in casu the fraction of true actives within the entire dataset of compounds.
The filter-dependent cutoff thresholds are estimated on the training set, by computing the upper bound of values, under the constraint, for all, where is the subset of the training corpus emitted by state.
A main disadvantage of the ROCE metric is the lack of a well-defined upper boundary, hence making it difficult to compare the quality of underlying models and applied cutoff thresholds.
Comparing the results of the PM and CCR metrics, both types of metric values increase with increasing model quality, but the PM metric seems to be less dependent on the applied cutoff threshold as compared to the CCR metric (in fact, the CCR metric value is increasing with increasing cutoff thresholds, while the opposite behavior is observed in the case of the PM metric).
