105 tokens were annotated by a second annotator.
For the case of increase(s), we used the data that we had labelled in the first part of this study the tag set that we used there included N oun), so we were able to retrieve hundreds of samples of both the singular and the plural form easily. 15% of the data (114 nominalization tokens) were annotated by a second annotator.
The test set was chosen instead of the training and the development dataset because the annotations were corroborated by a second annotator.
A second annotator then annotated a stratified sample of 114 (15%) of these.
However, to ensure annotation quality, for each document, a second annotator carried out validation after each round of annotation, and if in doubt, the two annotators were asked to agree on a gold standard through reconciliation.
Each person independently annotated ENVO terms in the text, with no knowledge of which pages had been assigned to a second annotator.
Specifically, many tokens that we tagged as PT were tagged as PI by the second annotator, and many tokens that we tagged as pt were tagged as pi by the second annotator.
They were presented to the second annotator as we annotated them grouped by word form (e.g. all tokens of increase together, all tokens of increases together, etc).; within a word group, ordered by UNIX sort of the source filename.
They were presented to the second annotator as we annotated them grouped by nominalization (i.e. all tokens of inhibition together, all tokens of induction together, etc)., ordered by the order in which they were retrieved from the corpus.
We then reviewed a small set of examples, and the second annotator labelled a few examples until they felt comfortable.
The second annotator is an experienced medical doctor with clinical working experience in internal medicine for more than 15 years.
