Validation loss lower than the training loss in scvi

Help scvi-tools
,
1

Hi, I used scvi to do integration for ~260k cells; 5k HVGs with 60 batches, I have two questions:

  1. Are the parameters looks good? Should I use autotune to search hyperparameters?
  2. I found validation loss lower than training loss (<1.5% difference), is acceptable?
# preparation
adata = sc.read_h5ad('./adata.h5ad.gz')
adata.layers["counts"] = adata.X.copy()
sc.pp.normalize_total(adata, target_sum=1e4, exclude_highly_expressed=True)
sc.pp.log1p(adata)
adata.raw = adata
sc.pp.highly_variable_genes(adata, n_top_genes=5000, flavor="cell_ranger", batch_key="SampleID", n_bins=20)
adata_hvg_5k = adata[:, adata.var["highly_variable"]].copy()
# scvi
adata_scVI = adata_hvg_5k.copy()
scvi.model.SCVI.setup_anndata(adata_scVI,
                              layer="counts",
                              continuous_covariate_keys=["pct_counts_mt"],
                              batch_key="SampleID")
model = scvi.model.SCVI(adata_scVI, n_layers=2, n_latent=50, gene_likelihood="nb")
# train
model.train(max_epochs=300, early_stopping=True,
            plan_kwargs=dict(n_steps_kl_warmup=1600, n_epochs_kl_warmup=None)
            )
# plot reconstruction loss
plt.plot(model.history['reconstruction_loss_train']['reconstruction_loss_train'], label='train')
plt.plot(model.history['reconstruction_loss_validation']['reconstruction_loss_validation'], label='validation')
plt.legend()

image
image996×734 51.8 KB

Thanks!

2

If you have time then I would try autotune.

The validation reconstruction loss could be lower due to things like dropout and batch norm being frozen during the validation loop.

3

Thanks, I just worry about if I do something wrong with parameters, I’ll try autotune.

4

Hi, I found in autotune tutorial, we use model_cls.setup_anndata(adata) to register the data. So I wonder if I should set layer, continuous_covariate_keys and batch_key parameters in .setup_anndata() for my own dataset. Thanks!

5

When I used batch_key in .setup_anndata() for autotune, all tune trials will get an error ValueError: assignment destination is read-only

6

Hmm I haven’t encountered that error before with autotune, could you send the code that you are using?

7

Hi Martin, thanks for your time, here is the code:

# preparation
adata = sc.read_h5ad('./adata.h5ad.gz')
adata.layers["counts"] = adata.X.copy()
sc.pp.normalize_total(adata, target_sum=1e4, exclude_highly_expressed=True)
sc.pp.log1p(adata)
adata.raw = adata
sc.pp.highly_variable_genes(adata, n_top_genes=5000, flavor="cell_ranger", batch_key="SampleID", n_bins=20)
adata_hvg_5k = adata[:, adata.var["highly_variable"]].copy()
# scvi register
adata_scVI = adata_hvg_5k.copy()
model_cls = scvi.model.SCVI
model_cls.setup_anndata(adata_scVI,
                        layer="counts",
                        continuous_covariate_keys=["pct_counts_mt"],
                        batch_key="SampleID")
# scvi tuner
scvi_tuner = autotune.ModelTuner(model_cls)
# search space
search_space = {
    "n_latent": tune.choice([25, 30, 40, 50, 60]),
    "n_hidden": tune.choice([128, 192, 256]),
    "n_layers": tune.choice([1, 2, 3]),
    "lr": tune.loguniform(1e-4, 1e-2),
    "gene_likelihood": tune.choice(["nb", "zinb"])
}
# fit
ray.init(log_to_driver=False)
results = scvi_tuner.fit(
    adata_scVI,
    metric="validation_loss",
    search_space=search_space,
    searcher="hyperopt",
    scheduler="asha",
    num_samples=50,
    max_epochs=150,
    resources={"cpu": 24, "gpu": 4},
)

Then all tune trials will get an error like:

2023-05-02 13:46:46,790	ERROR trial_runner.py:1062 -- Trial _trainable_2fc8e5be: Error processing event.
ray.exceptions.RayTaskError(ValueError): ray::ImplicitFunc.train() (pid=31418, ip=10.80.0.35, repr=_trainable)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/ray/tune/trainable/trainable.py", line 368, in train
    raise skipped from exception_cause(skipped)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py", line 337, in entrypoint
    return self._trainable_func(
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/ray/tune/trainable/function_trainable.py", line 654, in _trainable_func
    output = fn()
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/ray/tune/trainable/util.py", line 406, in _inner
    return inner(config, checkpoint_dir=None)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/ray/tune/trainable/util.py", line 398, in inner
    return trainable(config, **fn_kwargs)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/scvi/autotune/_manager.py", line 389, in _trainable
    getattr(model_cls, setup_method_name)(adata, **setup_kwargs)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/scvi/model/_scvi.py", line 211, in setup_anndata
    adata_manager.register_fields(adata, **kwargs)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/scvi/data/_manager.py", line 174, in register_fields
    self._add_field(
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/scvi/data/_manager.py", line 217, in _add_field
    field_registry[_constants._STATE_REGISTRY_KEY] = field.register_field(
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/scvi/data/fields/_dataframe_field.py", line 189, in register_field
    categorical_mapping = _make_column_categorical(
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/scvi/data/_utils.py", line 167, in _make_column_categorical
    df[alternate_column_key] = codes
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/pandas/core/frame.py", line 3612, in __setitem__
    self._set_item(key, value)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/pandas/core/frame.py", line 3797, in _set_item
    self._set_item_mgr(key, value)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/pandas/core/frame.py", line 3756, in _set_item_mgr
    self._iset_item_mgr(loc, value)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/pandas/core/frame.py", line 3746, in _iset_item_mgr
    self._mgr.iset(loc, value)
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1078, in iset
    blk.set_inplace(blk_locs, value_getitem(val_locs))
  File "/share/home/Grape/software/install_pkg/miniconda3/envs/mainenv/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 360, in set_inplace
    self.values[locs] = values
ValueError: assignment destination is read-only

But when not set batch_key=“SampleID”, all tune trials will run well:

model_cls.setup_anndata(adata_scVI,
                        layer="counts",
                        continuous_covariate_keys=["pct_counts_mt"]
                        )
  1. I confused for the target/metric of hyperparameter optimum in scVI integration? Should I set batch_key parameters for .setup_anndata()?
  2. When not set the batch_key, tune got n_latent=60 is optimum, is plausible? (For ~260k cells; 5k HVGs; 60 batches, is this parameter set too large?). I know it’s not good to deviate far from default n_latent=10.

Thanks again!

8

oh~ I encountered this error today… also, if I drop the batch, it will run.