Error running ModelTuner.fit

Dear the authors of scvi-tools,
Thank you very much for continuing support!

I have tried to run automated hyperparameter tuning using ModelTuner class functions following the instructions here:

The sample codes are written as following:

adata_sampled = scvi.data.synthetic_iid()
model_cls = scvi.model.SCVI
model_cls.setup_anndata(adata_sampled)
tuner = scvi.autotune.ModelTuner(model_cls)
results = tuner.fit(adata_sampled, metric=“validation_loss”)

However, all the trials came up with an error, stating that:

File “./scvi/autotune/_manager.py”, line 133, in _get_search_space
_type = tunables[param][“tunable_type”]
KeyError: ‘n_hidden’

I have checked the accompanying files of the ModelTuner class but could not figure out what caused the problem. Could you please tell me what I should do to fix the error and get the codes run smoothly?

Thank you very much in advance!
Kind regards,

Hi, thank you for your question. I am not able to replicate your issue using the latest version of scvi. Would you be able to provide the scvi-tools version you are using, and could you also run tuner.info() and post what that returns. Thanks!

1 Like

Hello Martin, hope this message finds you well.

Thank you very much for your feedback!
I was running scvi-tools 0.19.0 on python 3.8 and raytune 2.2.0.

I have realized that the installed scvi package did not contain the _scvi.py and _vae.py version that support autotuning. Therefore, I have updated the two (and supporting) files with their development versions retrieved from the following link: Implement a basic working ModelTuner API with Ray Tune by martinkim0 · Pull Request #1785 · scverse/scvi-tools · GitHub. However, necessary changes are still required: to _defaults.py (BaseLatentModeModuleClass was added to make VAE class eligible for model’s hyperparameter scanning), and to _vae.py to make other hyperparameters of the VAE class tunable (in the original file, only n_hidden is declared as Tunable).

Despite all those updates, still I could not figure out how train_kwargs (such as batch_size) and plan_kwargs (such as learning_rate) are passed to the function call (tuner.fit) and parsed by _get_search_space function (in the TunerManager class) in order to make them tunable. If you have any working examples for these hyperparameters could you please share in the follow-up posts?

Thank you very much in advance!
Kind regards,

Hi, thank you for your question. Sorry about the discrepancies regarding tunable hyperparameters in the code base – it is still in progress and we are working on adding support for more hyperparameters and model classes. We have a pull request that will be released soon with v0.20.0 that will contain changes that make this process easier. This will also include tutorials on how to use this module.

Regarding how train and plan keyword arguments are passed into the function, you can provide all hyperparameters (model, train, and plan) as a single dictionary in the search_space argument of ModelTuner.fit. We parse the arguments on our end so it is not necessary to provide separate dictionaries of keyword args. Hopefully this will be clear in the documentation once it is officially released.

1 Like

Hi Martin,

Thanks a lot for helping to clarify the unclear points. It’s great to know that the development of autotuning modules is almost completed. It would be of great help for the community regarding the popularity of scvi-tools. I am looking forward to its release in the near future.

Wish you and the scvi-tools development team a great year ahead!
Kind regards,

Hi Martin,

Apologies for digging up an old thread but I was wondering how exactly one would provide train_kwargs such as batch_size in the search_space argument?
Doing the following : search_space = {"batch_size": tune.choice([129, 130]), ## other arguments} is throwing an error : batch_size is invalid for SCVI.

Thanks!