In addition, the crowdsourcing approach requires the organization of an active working group, and may not be suitable for independent groups planning to perform extensive benchmarking studies.
One of the first projects to employ the Enron corpus was a self-described "extensive benchmark study of e-mail foldering".
Finally, in order to investigate the scalability of the method, an extensive benchmark study is conducted, where a commercial algebraic multigrid solver is used as reference.
We present extensive benchmark studies conducted with this package on UMBC's Bluegrit and Bluewave clusters, and NASA-GFSC's Discover cluster systems.
Researchers can scale up a benchmarking study using a competition-based benchmarking model.
"We did extensive benchmarking, and our prices are in line with what is happening elsewhere," the regulatory official said.
On the basis of extensive benchmarking, we chose a custom assembly protocol using the IDBA-UD38 algorithm (Methods).
Our review of publications identifies seven principles to guide researchers in designing a benchmarking study that increases reusability, transparency, and reproducibility of benchmarking studies.
Our review of current benchmarking practices reflects the most common approach to benchmarking study coordination: the independent model, where single research groups conduct individual benchmarking studies of relevant computational problems.
