The internal evaluation set was created by pooling the official training and development data, then randomly splitting this pool into 6000 abstracts for training (internal training set) and 1000 abstracts for evaluation during development (internal evaluation set).
A total of 29 papers were excluded because they did not specifically concern K-111, or had no direct relevance to the metabolic syndrome, or were general review papers, leaving nine papers and two abstracts for evaluation (Table 1).
While DNorm consists of separate steps for mention and concept finding, this article aggregates the mention-level results into the abstract level for evaluation.
As presented above, the developers of MutationFinder made available a data set (http://mutationfinder.sourceforge.net) of 305 abstracts annotated with point mutations that was used for system development and 508 abstracts available for evaluation.
Medical records (n=895) were abstracted for osteoporosis evaluation [dual-energy X-ray absorptiometry (DXA), vitamin D level, serum calcium), treatment (calcium, vitamin D, osteoporosis medication, hip protectors), clinical, and systems covariates.
We initially identified 1442 articles, of which 121 were selected, based on the title and abstract, for full-text evaluation (figure 1).
For many abstracts the conference abstract evaluation is limited due to the limited information provided in the abstract.
The titles of the articles were scanned to identify pertinent studies for abstract evaluation based on the presence of an educational intervention in the title.
Studies of preparations containing B serrata in combination with other ingredients, 2 3 non-randomised trials, 4 5 and abstracts reporting incomplete data for evaluation were excluded.
For relevant abstracts, full publications were retrieved for evaluation on the basis of criteria that were established a priori.
Using the process described in Figure 1, we selected approximately 10% (n = 1311) of the original search total (n = 13,240) for evaluation of abstracts; titles excluded at this stage were clearly not of relevance e.g. relating to non-healthcare safety topics.
