Documentation for training with multiple GPUs

Help scvi-tools
1

Is there any official tutorial or documentation around how to train with multiple GPUs? I’ve found various issues on github and here, but I didn’t find anything on scvi-tools.org outside of the API documentation.

2

Hi, multi-GPU support is still experimental so we don’t have official documentation or a tutorial yet. You may try out this functionality with the following:

model = scvi.model.SCVI(adata)
model.train(accelerator="gpu", devices=-1, strategy="ddp_find_unused_parameters_true")

but note that the behavior might be buggy.

3

Thanks this is helpful, I had not included strategy="ddp_find_unused_parameters_true" in my attempt to use multiple GPUs.

4

image
image640×370 22.4 KB

# come to train
DestVI.setup_anndata(st_adatas, layer="counts")
st_model = DestVI.from_rna_model(st_adatas, sc_model)
st_model.train(max_epochs=1200, batch_size = 512, devices=-1, accelerator = 'gpu'， strategy="ddp_find_unused_parameters_true")

i have set the strategy="ddp_find_unused_parameters_true", but it looks like nn.DataParallel rather than ddp

5

Hi , multi-GPU is still under development, we aim to reach that in the coming few weeks.
Thanks

1 Like