Presentation: Dask DataFrame groupby. Why it can fail and how to compensate.

Dask DataFrame groupby operations are very common and very powerful. However due to the distributed nature of Dask DataFrames, they can fail in unexpected ways. This talk covers mitigation strategies for these problems, including using set_index to optimize data layout, and using split_out and split_every parameters to optimize computation.

When do groupby's work well out of the box
aggregations where the number of groups fits easily into memory
applys, where data is already properly partitioned and has an index
Working on aggregations with large numbers of groups.
How to use split_out and split_every to tune performance.
groupby + apply.
How to use set_index to partition your dataset
deep dive into code implementation
the apply_concat_apply function
implementation of set_index: quantile computations, and passing in divisions

21 May 15:00 – 21 May 15:30 in Talks

Dask DataFrame groupby. Why it can fail and how to compensate.

Hugo Shi

Description

Abstract