# Model complexity selection

**URL:** <https://discourse.scverse.org/t/model-complexity-selection/139>\
**Category:** scvi-tools\
**Tags:** modeling\
**Created:** [July 8, 2021, 12:17am UTC](https://discourse.scverse.org/t/model-complexity-selection/139 "2021-07-08T00:17:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anatoly\_Buchin](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/anatoly_buchin/32/50_2.png) [@Anatoly\_Buchin](https://discourse.scverse.org/u/Anatoly_Buchin)\
**Post date:** [July 8, 2021, 12:17am UTC](https://discourse.scverse.org/t/model-complexity-selection/139/1 "2021-07-08T00:17:16Z")

</div>

Hi everybody,

I am currently running the scvi-tools to perform batch correction and dimensionality reduction of the dataset consisting of multiple patients. I am not sure about the complexity of the model that has been chosen for dimensionality reduction.

Here is an example of the code I am currently running:

adata\_da\_HVG=adata\_da.copy()  
sc.pp.highly\_variable\_genes(adata\_da\_HVG, n\_top\_genes=5000, subset=True)

scvi.data.setup\_anndata(  
adata\_da\_HVG,  
batch\_key=‘donor\_id’,  
layer=“counts”,  
labels\_key=‘cell\_group’)

model\_da = scvi.model.SCVI(adata\_da\_HVG)  
model\_da.train()

adata\_da\_HVG.obsm[“X\_scVI”]=model\_da.get\_latent\_representation()

I wonder if there is a way to visualize the model complexity to look at bias-variance tradeoff for the particular dataset. I would like to check if I would need more layers to perform the dimensionality reduction more efficiently.

-Anatoly

---

<div class="post-metadata">

**Author:** ![Valentine\_Svensson](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/valentine_svensson/32/10_2.png) [@Valentine\_Svensson](https://discourse.scverse.org/u/Valentine_Svensson)\
**Post date:** [July 9, 2021, 3:42am UTC](https://discourse.scverse.org/t/model-complexity-selection/139/2 "2021-07-09T03:42:04Z")

</div>

I believe you can investigate the bias variance trade off by looking at the training curves. Your `model_da` should have an field `.history` that has a dictionary of training metrics per epoch of model training. It definitely has the reconstruction error for the training set. I can’t remember how to make the training also record the reconstruction error for the test set. @adamgayoso is this in the documentation somewhere?

When you have these curves for alternative models you can compare them. Curves with higher reconstruction error in both train and test will have more bias. Curves where the test error goes up compared to the training error will have more variance. (Caveat, I always get the concepts of bias and variance for model mixed, I’m hoping I’m getting it right now).

Now, how to quantify the complexity of the model? That is hard I think… You could try to count all the parameters, but there are also various forms of regularization both for the neural networks and the latent variables (and other parameters). Not sure how that factors in to the definition of model complexity.

/Valentine

---

<div class="post-metadata">

**Author:** ![adamgayoso](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/adamgayoso/32/100_2.png) [@adamgayoso](https://discourse.scverse.org/u/adamgayoso)\
**Post date:** [July 9, 2021, 4:27am UTC](https://discourse.scverse.org/t/model-complexity-selection/139/3 "2021-07-09T04:27:54Z")

</div>

> [@Valentine\_Svensson](#):
>
> I can’t remember how to make the training also record the reconstruction error for the test set. @adamgayoso is this in the documentation somewhere?

Two options here. Note that by default the “test” set is really called the “validation” set. This convention is due to the fact that the validation set **can** be used for early stopping. I can go into more detail on this if it’s confusing.

1. Add `check_val_every_n_epoch=1` as an argument to `train` to record all the losses on the validation set. Note, training will be slower, you can set this value to 10, or 20, if you want.
2. call `model.get_reconstruction_error(indices=model.validation_indices)`. I believe here higher is better, where the trainer history is logging the negative of this.

---

<div class="post-metadata">

**Author:** ![tu.le](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/tu.le/32/92_2.png) [@tu.le](https://discourse.scverse.org/u/tu.le)\
**Post date:** [September 1, 2021, 7:35am UTC](https://discourse.scverse.org/t/model-complexity-selection/139/4 "2021-09-01T07:35:09Z")

</div>

Hi Adam,  
Thanks for your explanation. When checking **model.history** dict, I saw several items, such as **train\_loss\_step** , **train\_loss\_epoch** , and some others. What is the difference between them? Which one should be used to assess the goodness of the model (as an alternative to the **reconstruction\_error** that you have suggested above)? Also, assuming that **early\_stopping\_monitor=‘elbo\_validation’** , which value in this dict has been related to the **loss** value written down during the training process (illustrated as below)

Epoch 40/40: 100%|██████████| 40/40 [00:44\<00:00, 1.11s/it, **loss=588** , v\_num=1]

Many thanks,

---

<div class="post-metadata">

**Author:** ![Valentine\_Svensson](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/valentine_svensson/32/10_2.png) [@Valentine\_Svensson](https://discourse.scverse.org/u/Valentine_Svensson)\
**Post date:** [January 20, 2022, 8:07am UTC](https://discourse.scverse.org/t/model-complexity-selection/139/5 "2022-01-20T08:07:00Z")

</div>

Hi tu.le,

Yes some identities are split up, and during training the trainable parts of the ‘loss’ for the optimization objective can be calculated in way that is equivalent for the sake of the gradients in the objective function. The full reconstruction error will require more calculations, but not all of factors contribute to the optimization. So for a human, ‘`reconstruction_error`’ is what matters, but a computer calculating gradients can’t tell the difference between `reconstruction_error` and e.g. `train_loss_step`.

Best,  
/Valentine

---

<div class="post-metadata">

**Author:** ![tu.le](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/tu.le/32/92_2.png) [@tu.le](https://discourse.scverse.org/u/tu.le)\
**Post date:** [February 10, 2022, 4:28am UTC](https://discourse.scverse.org/t/model-complexity-selection/139/6 "2022-02-10T04:28:18Z")

</div>

Hi Valentine,  
Thanks for your explanation. It’s clear to me now.  
Best,  
Tu,