Presentation: Keynote: Dask & Prefect for Healthcare Machine Learning on AWS

In early 2019, SymphonyRM started a very ambitious R&D phase to develop machine learning models across many disease areas on clinical healthcare data. To achieve our vision, we needed infrastructure to give our data scientists superpowers. The combination of Dask and Prefect turned out to be incredibly powerful and productive, allowing us to achieve excellent results.

Part 1: Our path to Dask
- Business context: the challenge of helping healthcare organizations be proactive & the opportunity to develop predictive models on clinical data
- Technical context: background in machine learning in python at small scale & data ingestion pipelines in Apache Airflow
- Technical requirements going into R&D
Part 2: R&D Infrastructure & Results
- Dask for parallel execution & Prefect for parameterized machine learning pipelines
- Capabilities unlocked:
  - Run a machine learning pipeline end-to-end or only a portion: feature engineering, train & test models, hyperparameter tuning
  - Scale to 100+ workers in AWS for parallel execution
  - Run hundreds of experiments in minutes instead of days
- Achievements unlocked:
  - Infrastructure that gave data scientists superpowers
  - Dozens of successful models
Part 3: How we use Dask & Prefect today
- Highly parameterized Prefect Flows for machine learning pipelines
- Interactive use by data scientists
- Production use in scheduled jobs
Conclusions & Recommendations

21 May 13:00 – 21 May 14:00 in Talks, Tutorials / Workshops 2, Tutorials / Workshops 1

Keynote: Dask & Prefect for Healthcare Machine Learning on AWS

Joe Schmid, Jie Lou

Description

Abstract