# scANVI fails and returns NaNs after few epochs

**URL:** https://discourse.scverse.org/t/scanvi-fails-and-returns-nans-after-few-epochs/920
**Category:** Help
**Tags:** scvi, scanvi
**Created:** [November 11, 2022, 6:44pm UTC](https://discourse.scverse.org/t/scanvi-fails-and-returns-nans-after-few-epochs/920 "2022-11-11T18:44:20Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![temitopeleke](https://avatars.discourse-cdn.com/v4/letter/t/7ea924/32.png) [@temitopeleke](https://discourse.scverse.org/u/temitopeleke)
#### Post date: [November 11, 2022, 6:44pm UTC](https://discourse.scverse.org/t/scanvi-fails-and-returns-nans-after-few-epochs/920/1 "2022-11-11T18:44:20Z")

</div>

Hello,

**Backstorry:**

I successfully installed scvi-tools on Apple Silicon. This successfully trained on small dataset of (20k cells, 3k genes), however performance was extremely impeded when dataset size was (20k x 17k). So, I decided to use the accelerated PyTorch training on MPS. However, the `aten::remainder.Tensor_out` operation is not yet implemented using MPS so I get the following error

```auto
NotImplementedError: Could not run 'aten::index.Tensor' with arguments from the 'MPS' backend.

```

This was described in this github issue, which suggested to set the `PYTORCH_ENABLE_MPS_FALLBACK=1` environment variable to use CPU fall back instead for the operation.

Training worked as a result, but always fails after some epochs, or even first epoch and returns the rror;

```auto
ValueError: Expected parameter loc (Tensor of shape (X, Y))) of distribution Normal(loc: torch.Size([X, Y]), scale: torch.Size([X, Y)) to satisfy the constraint Real(), but found invalid values

```

Can anyone please help with this?

---

<div class="post-metadata">

### Author: ![adamgayoso](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/adamgayoso/32/100_2.png) [@adamgayoso](https://discourse.scverse.org/u/adamgayoso)
#### Post date: [November 11, 2022, 7:58pm UTC](https://discourse.scverse.org/t/scanvi-fails-and-returns-nans-after-few-epochs/920/2 "2022-11-11T19:58:40Z")

</div>

PyTorch MPS support is not fully operational (i.e., some tensor operations fail as this happened to you). Therefore, M1 support is restricted to CPU. We do not anticipate much of a speedup for scvi models anyway on MPS.

---

<div class="post-metadata">

### Author: ![temitopeleke](https://avatars.discourse-cdn.com/v4/letter/t/7ea924/32.png) [@temitopeleke](https://discourse.scverse.org/u/temitopeleke)
#### Post date: [November 11, 2022, 8:15pm UTC](https://discourse.scverse.org/t/scanvi-fails-and-returns-nans-after-few-epochs/920/3 "2022-11-11T20:15:03Z")

</div>

☹

Thanks a lot for the quick reply!

I’m fairly new to this space. Since all my personal compute resource are apple (silicon) based, are there any platforms with GPU support that you could recommend (something like sagemaker) etc.

Thanks!