# Interpreting validation loss curve in query to reference mapping

**URL:** <https://discourse.scverse.org/t/interpreting-validation-loss-curve-in-query-to-reference-mapping/243>\
**Category:** scvi-tools\
**Created:** [January 4, 2022, 3:20pm UTC](https://discourse.scverse.org/t/interpreting-validation-loss-curve-in-query-to-reference-mapping/243 "2022-01-04T15:20:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![emdann](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/emdann/32/254_2.png) [@emdann](https://discourse.scverse.org/u/emdann)\
**Post date:** [January 4, 2022, 3:20pm UTC](https://discourse.scverse.org/t/interpreting-validation-loss-curve-in-query-to-reference-mapping/243/1 "2022-01-04T15:20:42Z")

</div>

Hello,  
I’ve been using the scArches implementation in scvi-tools for query to reference mapping with trained SCVI models. I wanted to explore the trends of reconstruction loss on training and validation set to detect possible overfitting issues. During training of the reference scVI model, I see the expected trend where the training error underestimates the validation error. Conversely during training of the query model I find that the validation error is consistently lower than the training error and I find this puzzling.

Following the example in the [reference-mapping tutorial](https://docs.scvi-tools.org/en/stable/tutorials/notebooks/scarches_scvi_tools.html):

```auto
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import anndata
import scvi
import scanpy as sc

sc.set_figure_params(figsize=(4, 4))
scvi.settings.seed = 94705

url = "https://figshare.com/ndownloader/files/24539828"
adata = sc.read("pancreas.h5ad", backup_url=url)
print(adata)

## Define query dataset
query = np.array([s in ["smartseq2", "celseq2"] for s in adata.obs.tech])
adata_ref = adata[~query].copy()
adata_query = adata[query].copy()
sc.pp.highly_variable_genes(
    adata_ref,
    n_top_genes=2000,
    batch_key="tech",
    subset=True
)

adata_query = adata_query[:, adata_ref.var_names].copy()

## Train reference
scvi.model.SCVI.setup_anndata(adata_ref, batch_key="tech", layer="counts")
arches_params = dict(
    use_layer_norm="both",
    use_batch_norm="none",
    encode_covariates=True,
    dropout_rate=0.2,
    n_layers=2,
)

vae_ref = scvi.model.SCVI(
    adata_ref,
    **arches_params
)
vae_ref.train(check_val_every_n_epoch=1)

```

Plotting validation error for reference model training:

```auto
plt.plot(vae_ref.history['reconstruction_loss_train']['reconstruction_loss_train'], label='train');
plt.plot(vae_ref.history['reconstruction_loss_validation']['reconstruction_loss_validation'], label='validation');
plt.legend()

```

![Screenshot 2022-01-04 at 15.17.57](https://canada1.discourse-cdn.com/flex035/uploads/forum11/original/1X/421370aec2300c54f318579a5bdc98240c85f616.png)

Plotting validation error for query model training:

```auto
plt.plot(vae_q.history['reconstruction_loss_train']['reconstruction_loss_train'], label='train');
plt.plot(vae_q.history['reconstruction_loss_validation']['reconstruction_loss_validation'], label='validation');
plt.legend()

```

![Screenshot 2022-01-04 at 15.18.02](https://canada1.discourse-cdn.com/flex035/uploads/forum11/original/1X/7f7a466e8d8ffb3fb845b5f5d664318c00305fb2.png)

Any intuition on why this might be going on?

Many thanks in advance,  
Emma

---

<div class="post-metadata">

**Author:** ![adamgayoso](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/adamgayoso/32/100_2.png) [@adamgayoso](https://discourse.scverse.org/u/adamgayoso)\
**Post date:** [January 7, 2022, 12:34am UTC](https://discourse.scverse.org/t/interpreting-validation-loss-curve-in-query-to-reference-mapping/243/2 "2022-01-07T00:34:37Z")

</div>

Hi Emma,

We are a bit stumped and will think about it. We reproduced it on the pancreas example. Some initial thoughts:

1. Could be due to random sampling – e.g,. validation set is small and is biased with one cell type, perhaps you could try a query train size of 50% and see if it’s different.
2. The network during query training has very few parameters

We will get back to you if we have any more insight.

---

<div class="post-metadata">

**Author:** ![adamgayoso](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/adamgayoso/32/100_2.png) [@adamgayoso](https://discourse.scverse.org/u/adamgayoso)\
**Post date:** [January 7, 2022, 1:05am UTC](https://discourse.scverse.org/t/interpreting-validation-loss-curve-in-query-to-reference-mapping/243/3 "2022-01-07T01:05:04Z")

</div>

This thread also has some relevant points:

> <https://twitter.com/aureliengeron/status/1110839223878184960>

---

<div class="post-metadata">

**Author:** ![alexsunny123](https://avatars.discourse-cdn.com/v4/letter/a/51bf81/32.png) [@alexsunny123](https://discourse.scverse.org/u/alexsunny123)\
**Post date:** [November 15, 2022, 11:35am UTC](https://discourse.scverse.org/t/interpreting-validation-loss-curve-in-query-to-reference-mapping/243/4 "2022-11-15T11:35:35Z")

</div>

> [@adamgayoso](#):
>
> Hi Emma,
> 
> We are a bit stumped and will think about it. We reproduced it on the pancreas example. Some initial thoughts:
> 
> 1. Could be due to random sampling – e.g,. validation set is small and is biased with one cell type, perhaps you could try a query train size of 50% and see if it’s different.
> 2. The network during query training has very few parameters
> 
> We will get back to you if we have any more insight.

thanks for the awesome information.