Given that we were not concerned with a particular museum exhibit and instead were simply examining what visitors to the MPM know, we elected to use a simple term identification task as a measurement of recognition and knowledge.
Recently, we have introduced SpamHunting, a successful spam filter able to address the concept drift problem by combining a relevant term identification technique with an evolving sliding window strategy.
Also, text categorization applications can be much more effective if a good relevant term identification is available.
Records without abstracts were excluded from the study as an abstract was deemed necessary for subsequent term identification.
However, it remained to be investigated which impact an extensive manual curation of a multi-source chemical dictionary would have on chemical term identification in text.
The freely available Research Randomizer program [ 41] was used to randomly assign 88 references (10%) from the gold standard to the Term Identification Set, which was established as a new EndNote Library.
The titles and abstracts of all records in the EndNote Term Identification Set were exported and saved as a text file.
In order to identify the most frequently occurring thesaurus terms describing heart failure, MeSH terms belonging to the 88 records in the Term Identification Set were exported from EndNote and saved as a list in text file format (total non-unique n = 1845).
The study design had six phases: construction of the gold standard set; term identification; filter development; filter testing; external validation in an alternative gold standard set; and an estimate of precision which we call 'the post-hoc precision estimate'.
