After scanvi model training I’m trying to check various validation metrics. However, model.history.keys() gives me only the following options:
‘train_loss_step’, ‘train_loss_epoch’, ‘elbo_train’, ‘reconstruction_loss_train’, ‘kl_local_train’, ‘kl_global_train’, ‘train_classification_loss’, ‘train_accuracy’, ‘train_f1_score’.
So, only training metrics are given. However, model.validation_indices is not empty - it’s size is 10% of adata. What does it mean? Was validation performed?
Hi, thanks for you question. By default, we don’t run a validation loop after each epoch but do reserve 10% of the data as the validation set. In order to run validation, you can pass in check_val_every_n_epoch=n to the train method.
Thanks you for the quick response!
Do I understand correctly that model.get_reconstruction_error() gives me validation loss after the last epoch? Can I have other metrics (elbo, kl, etc.) for validation data as well?
model.get_reconstruction_error should be run after training and will return the model’s reconstruction error (negative reconstruction loss) for your entire dataset. When you enable the validation loop, the model will automatically log the following components: the validation loss, validation ELBO, validation KL divergence, and validation reconstruction loss, which will all be in model.history.
Sorry, one more clarifying question… Which options should I choose if I need only the final validation loss, validation ELBO, validation KL etc. I.e. not the full log but just resulting model quality.
Depends on what you mean by model quality 
In general, taking a look at the validation ELBO will give you a good sense of whether the model is overfitting/converging, and this will be the first metric you would want to check post-training.
If you’re interested in the goodness-of-fit of the generative model, you can take a look at scvi.criticism.PosteriorPredictiveCheck. We don’t have a tutorial for this on our main documentation website yet, but we do have one in the older documentation. If you’re interested in batch correction/biological conservation in the latent space, check out our scib-metrics suite.
Hope this helps and let me know if you have any other questions.
Yes, I was taking about ELBO and other metrics which are already implemented in the current version of scvi-tools. My question was more about setting check_val_every_n_epoch. Which value of n should I choose to get final validation values? Should I choose n=1 and then take the last elements from arrays kept in history?
Yes, you can set any n <= max_epochs and take the last elements in the history.