In addition, based on the manual annotation of sentences carried out for building the two sentence-level test sets, articles corresponding to a sentence annotated as positive for data deposition were also considered as positive in the article-level gold standard.
The machine-assisted annotations consist of sentence segmentation, tokenization and POS tagging processed using both MedPost and Stanford tools, lemmatization using BioLemmatizer, parsing using both C&C and Stanford parsers, as well as abbreviation definition detection using Ab3P, Schwartz & Hearst and NatLAb algorithms (31).
Although there is an imbalance in the corpus that may favor the classes high/yes, we believe the behavior of the classifiers can be explained by the lower level of ambiguity in the annotation of high/yes sentences in comparison with low/no sentences.
The annotation of such sentences is required throughout the paper, including the abstract, and any supporting summary paragraphs such as 'Author summary' or 'Conclusions'.
For example, the annotation of the sentence 'a c.1864C > A transversion in exon 12 of hMSH2 gene at the heterozygous state … leading to a proline 622 to threonine (p.Pro622Thr) amino acid substitution', with two mutation annotations indicated with underlining, would correspond to values in database fields for the gene (hMSH2), exon (12), variant (c.1864C > A) and protein change (p.Pro622Thr).
To evaluate the performance of Textpresso for CCC at dictyBase and TAIR, we compared a gold-standard set of sentences and annotations with those derived from Textpresso searches.
The corpus contains three BioC files: (i) Medline abstracts of raw text, (ii) sentences that are split using the OpenNLP sentence detector and (iii) annotations of simplification constructs at the sentence level.
That is, medical experts usually use specific words or styles of sentences when they write annotations or comments.
The multiplicity of annotations per sentence serves to ensure the quality and the consistency of the annotation.
Textpresso results were evaluated at the level of sentences as well as GO annotations that could be made from those sentences.
