How to use GPU partitions (MIGs)?

Help
,
1

Hi everyone,

I am trying to run scvi as followed, but I wan to speed up by using several GPUs. However, my cluster only gives me MIG partitions from GPUs. Even tho I can request several, when feeding the MIGs to scvi it only recognizes the one in position [0], but it does not recognize the rest, is there any way around, so I can use several devices?

scvi.model.SCVI.setup_anndata(adata)
vae = scvi.model.SCVI(adata_hvg, n_hidden=128, n_layers=2, n_latent=20, )
vae.train(accelerator=“gpu”, devices=3)

2

I don’t think it’s a scvi-tools API issue, or at least not something that is solvable right now, based on how it is implemented now with torch, lightning and CUDA.

Perhaps try to define it before you run the scvi-tools pipeline (CUDA_VISIBLE_DEVICES?)

I can only suggest using devices=-1.

Try to see if the multigpu tutorial works in your case: Train SCVI model with multi-GPU support — scvi-tools