In Scotland, linkage between several centrally held datasets including cancer registries and death records is performed using probability matching (Kendrick and Clarke, 1993), which uses all available identifying information, including name, date of birth, postal code and patient reference number, to link records belonging to the same person.
The patient cohorts used are large and the clinical information is consolidated from a number of hospital databases that use different data definitions and often hold incomplete datasets.
We then averaged the true-positive rate and the false-positive rate, obtained over the 100 hold out datasets (10 sub-sampling runs and 10-fold cross-validation for each run).
The same trends hold on datasets with larger numbers of genes; in particular, ASTRAL-2 remains typically the most accurate method (or close to the most accurate method) and CA-ML is typically the least accurate.
Model validation using hold out datasets, presented in detail in the accompanying technical appendix, suggests a very high degree of correlation at the school level between the observed and expected mean Hb levels (R>0.98) and the prevalence of anaemia (R>0.95), suggesting that the approach we used to estimate anaemia prevalence (and associated attributable fractions) was appropriate.
Typical desktop hosts have 1 Gbit/s NIC connections, while modern servers holding large datasets have 10 Gbit/s or better NICs.
Column 4 shows the average test AUCs achieved by each of the three models run on the hold-out datasets.
We note that the curves for Avishkar are smoother because they have been averaged over multiple hold-out datasets during cross-validation and sub-sampling from the larger negative dataset.
Currently, the database holds 32 datasets comprising 7636 gene expression measurements extracted from 20 different published gene or protein expression studies from various pancreatic cancer types, pancreatic precursor lesions (PanINs) and chronic pancreatitis.
Together, these techniques enable a responsive system that in particular holds a dataset with more than 1.4 billion triples and over 380,000 tags.
