Partners HealthCare
Biobank Disease Challenge
Welcome to the Partners HealthCare Biobank Disease Challenge
Partners HealthCare hosted the first "Biobank Disease Challenge," an artificial intelligence and machine learning data analytics competition that was open to researchers across the United States.
The goal was to enable major translational data science players to leverage the Partners HealthCare Biobank in order to develop better phenotypic algorithms for clinical and basic research.
The competition ran from September 12 to October 10th. Prizes were awarded for the best machine learning models and visualizations.
We congratulate all of the 50 teams for their hard work and outstanding submissions. The results of this challenge advance the understanding of data in the Partners HealthCare Biobank which will inform and improve the way this data will be used for research at Partners and more generally in the global understanding and development of phenotypic algorithms for clinical and basic research involving electronic health data.
As we progress, we anticipate offering additional data challenges. Please check back to this site for future competitions
The official winners are
1st Place (Tied) - $12,500 PrizeIBM Research / Center for Computational Health
Prithwish Chakraborty, Michal Ozery-Flato, Kristen Severson, Eryu Xia
Total judging score: 56 / Mean AUC: 0.8792
1st Place (Tied) - $12,500 Prize
University of Pittsburgh / Department of Biomedical Informatics
Xinghua Lu, Degan Hao, Qiao Jin
Total judging score: 56 / Mean AUC: 0.8768
3rd Place - $5,000 Prize
Massachusetts General Hospital / Computational Pathology
Maciej Pacula, Lev Lipkin, Dandan Mo
Total judging score: 48 / Mean AUC: 0.8268
Partners Biobank Disease Challenge Description
IntroductionThe Partners HealthCare Biobank is an enterprise-wide initiative whose goal is to provide a foundation for the next generation of translational research studies of genotype, environment, gene-environment interaction, biomarker and family history associations with disease phenotypes (Karlson, 2016). The Biobank has leveraged in-person and electronic recruitment methods to enroll over 80,000 participants since 2010. All participants in the Biobank have their electronic health record (EHR) data linked to biospecimens, genotype data and self-reported health survey information (Gainer, 2016). The data is aggregated into an i2b2 environment called the Biobank Portal to facilitate querying and analysis in a web user interface and data backend (Murphy, 2010).
EHR data is available for large populations, can be quickly ascertained for research and provides a naturalistic data collection pattern. These benefits come with significant limitations, however. Namely, the data has significant data quality and completeness biases that can be difficult to use to derive new knowledge or identify interesting signals. One of the earliest use cases of leveraging this EHR data is for building computed phenotype algorithms to use for GWAS and other genomic data analyses (Kohane, 2011; Bowton, 2014; Castro, 2016). Computed phenotypes are machine learning algorithms to identify a patient’s true disease state from raw EHR data. Once true disease status is known, additional models can be derived to predict future onset of disease, quantify severity of disease and/or identify sub-populations within diseases.
The Challenge
The Biobank Disease Challenge is a two-part challenge:
Part 1: Develop Computed Phenotypes for 5 Diseases
In this challenge we are asking participants to develop computed phenotype machine learning algorithms to identify 5 disease states (to be released at the beginning of the challenge) of all patients enrolled in the Partners Biobank. Algorithms must be developed against available EHR data and limited training data. These algorithms will be assessed against gold standard labels from clinician assessment of a patient’s full clinical chart. All submissions should include an algorithm score for each patient and disease state. Scores will be ranked for each disease and an area under the ROC curve (AUC) will be calculated based on the validation dataset. The final score will be partially based on the average of the 5 disease AUCs.
Part 2: AI for Novel Insights from EHR data
The second part is an open-ended task that will invite teams to develop artificial intelligence algorithms to identify interesting patterns within one of the disease populations in part 1 of the challenge. In this challenge, users can choose any analysis/model that provides insight from the EHR data. Examples might include (but not limited to):
- Group patients into clinically meaningful clusters
- Stratify patients by disease severity
- Identify longitudinal patterns of disease or care
- Identify predictors of treatment initiation and response
All submissions must include a fully self-contained R or Jupyter Notebook in HTML format, built towards human-interpretability with explanatory visualizations.
Judging Panel
A panel of judges with expertise in medicine, machine learning and health informatics has been assembled to judge each submission. The final judging score to determine the winners of the challenge will be compiled as below.
Judging Criteria
Part 1: Algorithm Performance
Average AUC: ______ (1-20)
Interpretability of the models: _____ (1-10)
Part 2: Innovation/Insight
Design of Methods / Analysis ____ (1-5)
Innovation ____ (1-5)
Clinical Importance ____ (1-10)
Clarity of Results / Visualizations ____ (1-10)
References
Doshi-Velez, F., Ge, Y. and Kohane, I., 2014. Comorbidity clusters in autism spectrum disorders: an electronic health record time-series analysis. Pediatrics, 133(1), pp.e54-e63.
Murphy, S.N., Weber, G., Mendis, M., Gainer, V., Chueh, H.C., Churchill, S. and Kohane, I., 2010. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). Journal of the American Medical Informatics Association, 17(2), pp.124-130.
McCoy Jr, T.H., Castro, V.M., Hart, K.L., Pellegrini, A.M., Yu, S., Cai, T. and Perlis, R.H., 2018. Genome-wide association study of dimensional psychopathology using electronic health records. Biological psychiatry.
Karlson, E.W., Boutin, N.T., Hoffnagle, A.G. and Allen, N.L., 2016. Building the partners healthcare biobank at partners personalized medicine: informed consent, return of research results, recruitment lessons and operational considerations. Journal of personalized medicine, 6(1), p.2.
Gainer, V.S., Cagan, A., Castro, V.M., Duey, S., Ghosh, B., Goodson, A.P., Goryachev, S., Metta, R., Wang, T.D., Wattanasin, N. and Murphy, S.N., 2016. The Biobank Portal for Partners personalized medicine: a query tool for working with consented biobank samples, genotypes, and phenotypes using i2b2. Journal of personalized medicine, 6(1), p.11.
Kohane, I.S., 2011. Using electronic health records to drive discovery in disease genomics. Nature Reviews Genetics, 12(6), p.417.
Bowton, E., Field, J.R., Wang, S., Schildcrout, J.S., Van Driest, S.L., Delaney, J.T., Cowan, J., Weeke, P., Mosley, J.D., Wells, Q.S. and Karnes, J.H., 2014. Biobanks and electronic medical records: enabling cost-effective research. Science translational medicine, 6(234), pp.234cm3-234cm3.
Castro, V.M., Minnier, J., Murphy, S.N., Kohane, I., Churchill, S.E., Gainer, V., Cai, T., Hoffnagle, A.G., Dai, Y., Block, S. and Weill, S.R., 2015. Validation of electronic health record phenotyping of bipolar disorder cases and controls. American Journal of Psychiatry, 172(4), pp.363-372.