# GPU available but not using

**URL:** <https://discourse.scverse.org/t/gpu-available-but-not-using/1211>\
**Category:** scvi-tools\
**Created:** [March 9, 2023, 1:26am UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211 "2023-03-09T01:26:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![karenlawwc](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/karenlawwc/32/536_2.png) [@karenlawwc](https://discourse.scverse.org/u/karenlawwc)\
**Post date:** [March 9, 2023, 1:26am UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211/1 "2023-03-09T01:26:48Z")

</div>

Hi,

Thanks for the great tool! I am going through the tutorials and are trying to invoke GPU to run the analysis for training. My system should have the Nvidia GPU, and I should have installed a version of PyTorch that supports it. However, when I did vae.train(), it indicated that GPU is available in my environment but not using to run the training step.

I am wondering how would I be able to utilize the GPU detected in the environment? Is there any extra parameter that I should set?  
Here below are the warning messages and a screenshot of what I mentioned here. Thank you very much for the help in the community!

**INFO:pytorch\_lightning.utilities.rank\_zero:GPU available: True (cuda), used: False**  
INFO:pytorch\_lightning.utilities.rank\_zero:TPU available: False, using: 0 TPU cores  
INFO:pytorch\_lightning.utilities.rank\_zero:IPU available: False, using: 0 IPUs  
INFO:pytorch\_lightning.utilities.rank\_zero:HPU available: False, using: 0 HPUs  
/wynton/home/fong/karenlawwc/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pytorch\_lightning/trainer/setup.py:176: PossibleUserWarning: GPU available but not used. Set `accelerator` and `devices` using `Trainer(accelerator='gpu', devices=2)`.  
rank\_zero\_warn(

 ![Screen Shot 2023-03-08 at 5.25.02 PM](https://canada1.discourse-cdn.com/flex035/uploads/forum11/original/1X/10e5838e3d8f4677e666398e613f03aaf395b415.png)

---

<div class="post-metadata">

**Author:** ![martinkim0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/martinkim0/32/881_2.png) [@martinkim0](https://discourse.scverse.org/u/martinkim0)\
**Post date:** [March 9, 2023, 5:47pm UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211/2 "2023-03-09T17:47:18Z")

</div>

Hi, thank you for your question. Could you try passing in `train(use_gpu=True)` and checking if the problem still occurs?

---

<div class="post-metadata">

**Author:** ![karenlawwc](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/karenlawwc/32/536_2.png) [@karenlawwc](https://discourse.scverse.org/u/karenlawwc)\
**Post date:** [March 9, 2023, 8:40pm UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211/3 "2023-03-09T20:40:40Z")

</div>

Hi, thanks for the reply Martin!

When I tried vae.train(use\_gpu=True), the following error occurred.

MisconfigurationException: `MPSAccelerator` can not run on your system since the accelerator is not available. The following accelerator(s) is available and can be passed into `accelerator` argument of `Trainer`: [‘cpu’, ‘cuda’]

I am guessing it would be a system issue, not anything I can solve by adding/changing additional parameter?

Thank you!

---

<div class="post-metadata">

**Author:** ![martinkim0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/martinkim0/32/881_2.png) [@martinkim0](https://discourse.scverse.org/u/martinkim0)\
**Post date:** [March 9, 2023, 10:04pm UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211/4 "2023-03-09T22:04:34Z")

</div>

Hmm I see, I’ve actually come across a similar error before where Lightning tries to use MPS on Linux, so it could be on their side of things.

We have a [pull request](https://github.com/scverse/scvi-tools/pull/1950) scheduled for our next minor release (0.20.2) that allows you to directly pass in `accelerator` and `devices` into `Trainer`. We currently parse `use_gpu` into those arguments on our end, and something could have gone wrong there. This could solve your issue, but it won’t be out for another week or two.

For now, I would recommend creating a new environment with CUDA PyTorch as well as the latest scvi-tools release (0.20.1) and seeing if the issue still persists. Usually creating a fresh environment solves GPU detection issues for me, let me know if it works for you!

---

<div class="post-metadata">

**Author:** ![karenlawwc](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/karenlawwc/32/536_2.png) [@karenlawwc](https://discourse.scverse.org/u/karenlawwc)\
**Post date:** [March 9, 2023, 11:16pm UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211/5 "2023-03-09T23:16:10Z")

</div>

Thank you Martin, I will first try to install the latest release (0.20.1) to create a fresh environment and then see if it will solve the issue. If not, I will try again when the next release is out for passing accelerator into Trainer!

Another question, I tried running the training and subsequent steps without GPU and it went great. However, I got stuck at the Differential gene expression step and here is the error. I am hoping you could possibly help me identify how to modify my adata to make it work since it went fine with the tutorial PBMC dataset.

I already filtered the adata so that protein and RNA share the same cells. The full traceback are below. Somehow the index of the adata is messed up and is throwing the error?

 ![adata_demo-2](https://canada1.discourse-cdn.com/flex035/uploads/forum11/original/1X/f4a25b3bc8e3d4ace4b27e05b7f69085448613a1.jpeg)

## DE…: 0%| | 0/18 [00:39\<?, ?it/s]

KeyError Traceback (most recent call last)  
Cell In [25], line 1  
----\> 1 de\_df = vae.differential\_expression(groupby = “cell\_calls”)  
2 de\_df.head(5)

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/scvi/model/\_totalvi.py:763, in TOTALVI.differential\_expression(self, adata, groupby, group1, group2, idx1, idx2, mode, delta, batch\_size, all\_stats, batch\_correction, batchid1, batchid2, fdr\_target, silent, protein\_prior\_count, scale\_protein, sample\_protein\_mixing, include\_protein\_background, \*\*kwargs)  
749 model\_fn = partial(  
750 self.\_expression\_for\_de,  
751 scale\_protein=scale\_protein,  
(…)  
755 batch\_size=batch\_size,  
756 )  
757 col\_names = np.concatenate(  
758 [  
759 np.asarray(\_get\_var\_names\_from\_manager(adata\_manager)),  
760 self.protein\_state\_registry.column\_names,  
761 ]  
762 )  
→ 763 result = \_de\_core(  
764 adata\_manager,  
765 model\_fn,  
766 groupby,  
767 group1,  
768 group2,  
769 idx1,  
770 idx2,  
771 all\_stats,  
772 cite\_seq\_raw\_counts\_properties,  
773 col\_names,  
774 mode,  
775 batchid1,  
776 batchid2,  
777 delta,  
778 batch\_correction,  
779 fdr\_target,  
780 silent,  
781 \*\*kwargs,  
782 )  
784 return result

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/scvi/model/base/\_utils.py:267, in _de\_core(adata\_manager, model\_fn, groupby, group1, group2, idx1, idx2, all\_stats, all\_stats\_fn, col\_names, mode, batchid1, batchid2, delta, batch\_correction, fdr, silent, \*\*kwargs)  
265 res = res.sort\_values(by=sort\_key, ascending=False)  
266 if mode == “change”:  
→ 267 res[f"is\_de\_fdr_{fdr}"] = \_fdr\_de\_prediction(res[“proba\_de”], fdr=fdr)  
268 if idx1 is None:  
269 g2 = “Rest” if group2 is None else group2

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/scvi/model/base/\_utils.py:288, in \_fdr\_de\_prediction(posterior\_probas, fdr)  
286 raise ValueError(“posterior\_probas should be 1-dimensional”)  
287 sorted\_genes = np.argsort(-posterior\_probas)  
→ 288 sorted\_pgs = posterior\_probas[sorted\_genes]  
289 cumulative\_fdr = (1.0 - sorted\_pgs).cumsum() / (1.0 + np.arange(len(sorted\_pgs)))  
290 d = (cumulative\_fdr \<= fdr).sum()

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pandas/core/series.py:984, in Series. **getitem** (self, key)  
981 key = np.asarray(key, dtype=bool)  
982 return self.\_get\_values(key)  
→ 984 return self.\_get\_with(key)

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pandas/core/series.py:1019, in Series.\_get\_with(self, key)  
1015 if key\_type == “integer”:  
1016 # We need to decide whether to treat this as a positional indexer  
1017 # (i.e. self.iloc) or label-based (i.e. self.loc)  
1018 if not self.index.\_should\_fallback\_to\_positional:  
→ 1019 return self.loc[key]  
1020 else:  
1021 return self.iloc[key]

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pandas/core/indexing.py:967, in \_LocationIndexer. **getitem** (self, key)  
964 axis = self.axis or 0  
966 maybe\_callable = com.apply\_if\_callable(key, self.obj)  
→ 967 return self.\_getitem\_axis(maybe\_callable, axis=axis)

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pandas/core/indexing.py:1194, in \_LocIndexer.\_getitem\_axis(self, key, axis)  
1191 if hasattr(key, “ndim”) and key.ndim \> 1:  
1192 raise ValueError(“Cannot index with multidimensional key”)  
→ 1194 return self.\_getitem\_iterable(key, axis=axis)  
1196 # nested tuple slicing  
1197 if is\_nested\_tuple(key, labels):

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pandas/core/indexing.py:1132, in \_LocIndexer.\_getitem\_iterable(self, key, axis)  
1129 self.\_validate\_key(key, axis)  
1131 # A collection of keys  
→ 1132 keyarr, indexer = self.\_get\_listlike\_indexer(key, axis)  
1133 return self.obj.\_reindex\_with\_indexers(  
1134 {axis: [keyarr, indexer]}, copy=True, allow\_dups=True  
1135 )

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pandas/core/indexing.py:1330, in \_LocIndexer.\_get\_listlike\_indexer(self, key, axis)  
1327 ax = self.obj.\_get\_axis(axis)  
1328 axis\_name = self.obj.\_get\_axis\_name(axis)  
→ 1330 keyarr, indexer = ax.\_get\_indexer\_strict(key, axis\_name)  
1332 return keyarr, indexer

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pandas/core/indexes/base.py:5796, in Index.\_get\_indexer\_strict(self, key, axis\_name)  
5793 else:  
5794 keyarr, indexer, new\_indexer = self.\_reindex\_non\_unique(keyarr)  
→ 5796 self.\_raise\_if\_missing(keyarr, indexer, axis\_name)  
5798 keyarr = self.take(indexer)  
5799 if isinstance(key, Index):  
5800 # GH 42790 - Preserve name from an Index

File ~/miniconda3/envs/scanpy\_env/lib/python3.10/site-packages/pandas/core/indexes/base.py:5859, in Index.\_raise\_if\_missing(self, key, indexer, axis\_name)  
5856 raise KeyError(f"None of [{key}] are in the [{axis\_name}]“)  
5858 not\_found = list(ensure\_index(key)[missing\_mask.nonzero()[0]].unique())  
→ 5859 raise KeyError(f”{not\_found} not in index")

KeyError: ‘[239, 238, 237, 235, 234, 233, 236, 231, 230, 229, 232, 245, 244, 243, 241, 240, 242, 255, 260, 259, 258, 257, 256, 254, 248, 252, 251, 250, 249, 253, 247, 246, 274, 273, 272, 271, 270, 269, 268, 266, 265, 264, 263, 262, 261, 267, 286, 293, 292, 291, 290, 289, 288, 287, 285, 284, 283, 282, 281, 280, 279, 278, 277, 276, 275, 307, 306, 305, 304, 303, 302, 301, 299, 298, 297, 296, 295, 294, 300, 317, 322, 321, 320, 319, 318, 315, 316, 314, 313, 312, 311, 310, 309, 308, 337, 336, 335, 334, 333, 332, 330, 331, 328, 327, 326, 325, 324, 323, 329, 348, 349, 350, 351, 355, 353, 354, 356, 352, 347, 344, 345, 343, 342, 341, 340, 339, 338, 346, 370, 369, 368, 367, 366, 365, 364, 361, 362, 360, 359, 358, 357, 363, 384, 383, 389, 382, 386, 387, 388, 385, 381, 375, 379, 378, 377, 376, 374, 373, 372, 371, 380, 405, 406, 407, 408, 413, 410, 411, 412, 404, 409, 402, 403, 400, 401, 390, 391, 392, 394, 393, 396, 397, 398, 399, 395, 433, 430, 431, 432, 434, 440, 436, 437, 438, 441, 429, 435, 428, 439, 426, 427, 414, 416, 417, 418, 419, 415, 421, 422, 423, 424, 425, 420, 453, 461, 460, 459, 458, 457, 456, 455, 454, 452, 462, 450, 449, 448, 447, 446, 445, 444, 443, 442, 451, 474, 476, 477, 481, 479, 480, 473, 478, 472, 475, 470, 469, 468, 467, 466, 465, 464, 463, 471, 500, 501, 502, 503, 504, 505, 506, 511, 508, 509, 512, 513, 514, 499, 507, 498, 510, 496, 482, 483, 485, 497, 486, 487, 488, 484, 490, 491, 492, 493, 494, 495, 489, 529, 530, 531, 532, 536, 534, 535, 537, 528, 533, 527, 525, 517, 524, 523, 522, 521, 520, 519, 518, 516, 515, 526, 556, 555, 553, 557, 554, 558, 564, 560, 561, 562, 563, 565, 559, 552, 538, 550, 551, 539, 540, 541, 543, 542, 545, 546, 547, 548, 549, 544, 584, … 32071, 32072, 32073, 32074, 32075, 32076, 32077, 32078, 32079, 32080, 32081, 32082, 32083, 32084, 32085, 32086, 32087, 32088, 32089, 32090, 32091, 32092, 32093, 32094, 32095, 32096, 32097, 32098, 32099, 32100, 32101, 32102, 32103, 32104, 32105, 32106, 32107, 32108, 32109, 32110, 32111, 32112, 32113, 32114, 32115, 32116, 32117, 32118, 32119, 32120, 32121, 32122, 32123, 32124, 32125, 32126, 32127, 32128, 32129, 32130, 32131, 32132, 32133, 32134, 32135, 32136, 32137, 32138, 32139, 32140, 32141, 32142, 32143] not in index’

---

<div class="post-metadata">

**Author:** ![adamgayoso](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/adamgayoso/32/100_2.png) [@adamgayoso](https://discourse.scverse.org/u/adamgayoso)\
**Post date:** [March 10, 2023, 3:52am UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211/6 "2023-03-10T03:52:25Z")

</div>

> [@karenlawwc](#):
>
> I already filtered the adata so that protein and RNA share the same cells. The full traceback are below. Somehow the index of the adata is messed up and is throwing the error?

Issue will be fixed in the next release, but also check out my response here:

> [@Questions about running differential expression](https://discourse.scverse.org/t/questions-about-running-differential-expression/1137/9):
>
> Ok I understand the issue. In the notebook you shared you can replace adata.obsm["protein\_expression"] = mdata["adt"].layers["counts"].A.copy() with adata.obsm["protein\_expression"] = mdata["adt"].to\_df("counts") The issue arises due to the first line giving a numpy array, so our library generates integer based index for the protein names. This causes an issue with accessing a pandas series in your traceback. We will push a fix soon. In the meantime, the recommended workflow is to use data…

---

<div class="post-metadata">

**Author:** ![karenlawwc](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/karenlawwc/32/536_2.png) [@karenlawwc](https://discourse.scverse.org/u/karenlawwc)\
**Post date:** [March 10, 2023, 8:56pm UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211/7 "2023-03-10T20:56:27Z")

</div>

Thank you Martin and Adam!

Martin, I tried creating a new conda environment with CUDA PyTorch and the latest scvi-tools like you mentioned, GPU is now being detected and running. Thank you very much for your help!

As for the differential expression step, there is still error after modifying according to what Adam suggested, so hopefully can continue to work on making this work together. Thank you very much!

---

<div class="post-metadata">

**Author:** ![jesswhitts](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/jesswhitts/32/585_2.png) [@jesswhitts](https://discourse.scverse.org/u/jesswhitts)\
**Post date:** [March 17, 2023, 4:09pm UTC](https://discourse.scverse.org/t/gpu-available-but-not-using/1211/8 "2023-03-17T16:09:32Z")

</div>

I’m having a similar issue, but with mac M1 GPU, should this be compatible? PyTorch should run with this I believe.

I created a new conda environment as per the scvi-tools set up instructions, and ran ‘conda install pytorch torchvision torchaudio -c pytorch’ and ‘conda install jax jaxlib -c conda-forge’ too. Now it still isn’t using the GPU, and I’m getting an error.

I’m following the introductory tutorial, code used:

(scvi-env) jwhittle@BSC00783 ~ % pip install --quiet scvi-colab  
(scvi-env) jwhittle@BSC00783 ~ % python  
Python 3.9.16 | packaged by conda-forge | (main, Feb 1 2023, 21:38:11)  
[Clang 14.0.6] on darwin  
Type “help”, “copyright”, “credits” or “license” for more information.

> > > from scvi\_colab import install  
> > > install(run\_outside\_colab=True)  
> > > INFO scvi-colab: Installing scvi-tools.  
> > > INFO scvi-colab: Install successful. Testing import.  
> > > Global seed set to 0  
> > > /Users/jwhittle/opt/anaconda3/envs/scvi-env/lib/python3.9/site-packages/flax/struct.py:132: FutureWarning: jax.tree\_util.register\_keypaths is deprecated, and will be removed in a future release. Please use `register_pytree_with_keys()` instead.  
> > > jax.tree\_util.register\_keypaths(data\_clz, keypaths)  
> > > /Users/jwhittle/opt/anaconda3/envs/scvi-env/lib/python3.9/site-packages/flax/struct.py:132: FutureWarning: jax.tree\_util.register\_keypaths is deprecated, and will be removed in a future release. Please use `register_pytree_with_keys()` instead.  
> > > jax.tree\_util.register\_keypaths(data\_clz, keypaths)

> > > import scanpy as sc  
> > > import scvi  
> > > sc.set\_figure\_params(figsize=(4, 4))  
> > > adata = scvi.data.heart\_cell\_atlas\_subsampled()  
> > > INFO File data/hca\_subsampled\_20k.h5ad already downloaded  
> > > sc.pp.filter\_genes(adata, min\_counts=3)  
> > > adata.layers[“counts”] = adata.X.copy()  
> > > sc.pp.normalize\_total(adata, target\_sum=1e4)  
> > > sc.pp.log1p(adata)  
> > > adata.raw = adata  
> > > scvi.model.SCVI.setup\_anndata(  
> > > … adata,  
> > > … layer=“counts”,  
> > > … categorical\_covariate\_keys=[“cell\_source”, “donor”],  
> > > … continuous\_covariate\_keys=[“percent\_mito”, “percent\_ribo”],  
> > > … )  
> > > model = scvi.model.SCVI(adata)  
> > > model  
> > > SCVI Model with the following params:  
> > > n\_hidden: 128, n\_latent: 10, n\_layers: 1, dropout\_rate: 0.1, dispersion: gene,  
> > > gene\_likelihood: zinb, latent\_distribution: normal  
> > > Training status: Not Trained  
> > > Model’s adata is minified?: False

> > > model.train()  
> > > GPU available: True (mps), used: False  
> > > TPU available: False, using: 0 TPU cores  
> > > IPU available: False, using: 0 IPUs  
> > > HPU available: False, using: 0 HPUs  
> > > /Users/jwhittle/opt/anaconda3/envs/scvi-env/lib/python3.9/site-packages/pytorch\_lightning/trainer/setup.py:201: UserWarning: MPS available but not used. Set `accelerator` and `devices` using `Trainer(accelerator='mps', devices=1)`.  
> > > rank\_zero\_warn(  
> > > Epoch 1/400: 0%| | 0/400 [00:00\<?, ?it/s][E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > [E thread\_pool.cpp:109] Exception in thread pool task: mutex lock failed: Invalid argument  
> > > zsh: bus error python  
> > > (scvi-env) jwhittle@BSC00783 ~ % /Users/jwhittle/opt/anaconda3/envs/scvi-env/lib/python3.9/multiprocessing/resource\_tracker.py:216: UserWarning: resource\_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown  
> > > warnings.warn('resource\_tracker: There appear to be %d ’

Can anyone help with this please?