Presentation: Using Dask and many GPUs to train a neural network with PyTorch

I took a neural network I had trained on a single CPU to generate pet names and tried retraining it with tons of connected GPUs using Dask, PyTorch, and the package dask-pytorch-ddp. I learned a lot about when is the right time to use multiple GPUs and what the pitfalls can be. In this talk I'll discuss what these lessons mean for training with GPUs and Dask.

The setup - First I'll cover the basics of NLP and text-generating neural networks. I'll specifically talk though how my network runs on a dataset of pet names.
Moving to PyTorch GPU - My original neural network ran using TensorFlow (in R!), and so it took a bit of work to migrate to PyTorch.
Using Dask and DDP - Dask will be used here to coordinate training over many GPUs at once. To do this I use PyTorch DDP and the new package. dask-pytorch-ddp which helps pass data between the different workers.
Experiment results - How many GPUs did compared to a single CPU, and what that means for Dask users.
Concurrent training - An alternative use for Dask is instead of trying to train a single network faster, a lot of different networks can be trained concurrently. I'll show the usefulness of this method for the pet names data.

21 May 15:30 – 21 May 16:00 in Talks

Using Dask and many GPUs to train a neural network with PyTorch

Jacqueline Nolis

Description

Abstract