Autotune failed due to "h5py objects cannot be pickled"

Help scvi-tools
1

Hi,

I am trying to tune the parameters. However it failed due to an error saying that “h5py objects cannot be pickled”. I read some related posts saying similar failure due to multiprocessing and reduced the GPU to only 1. But it still failed. Also the tutorial seems doing file with multiple CPUs or GPUs. What should I do to avoid the problem?

I also have the same error (h5py objects cannot be pickled) when I tried to save models using pickle on the cluster, but it worked in the standard python environment.

model_cls = scvi.model.SCVI
model_cls.setup_anndata(adata, layer="counts", batch_key="source", size_factor_key="size_factors")

# search space
search_space = {
    "model_params": {"n_hidden": tune.choice([64, 128, 256]), "n_layers": tune.choice([1, 2, 3]),"n_latent": tune.choice([25, 30, 40, 50, 60]),"lr": tune.loguniform(1e-4, 1e-2),"gene_likelihood": tune.choice(["nb", "zinb"])},
    "train_params": {"max_epochs": max_epochs},
}

ray.init(log_to_driver=False)
results = autotune.run_autotune(
    model_cls,
    data=adata,
    metrics="validation_loss",
    mode='min',
    search_space=search_space,
    searcher="hyperopt",
    scheduler="asha",
    num_samples=50,
    resources={"gpu": 1},
)

Error message trackback is listed below.

Traceback (most recent call last):
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/worker.py”, line 764, in put_object**
** serialized_value = self.get_serialization_context().serialize(value)**
** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/serialization.py”, line 519, in serialize**
** return self._serialize_to_msgpack(value)**
** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/serialization.py”, line 497, in _serialize_to_msgpack**
** pickle5_serialized_object = self._serialize_to_pickle5(**
** ^^^^^^^^^^^^^^^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/serialization.py”, line 444, in _serialize_to_pickle5**
** raise e**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/serialization.py”, line 439, in _serialize_to_pickle5**
** inband = pickle.dumps(**
** ^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/cloudpickle/cloudpickle.py”, line 1479, in dumps**
** cp.dump(obj)**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/cloudpickle/cloudpickle.py”, line 1245, in dump**
** return super().dump(obj)**
** ^^^^^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/h5py/_hl/base.py”, line 370, in getnewargs**
** raise TypeError(“h5py objects cannot be pickled”)**
TypeError: h5py objects cannot be pickled
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
** File “~/users/project_scvi/working/ourdata/run_autotune.py”, line 92, in **
** main(args.suffix, args.max_epochs)**
** File “~/users/project_scvi/working/ourdata/run_autotune.py”, line 81, in main**
** setup_and_tune_model(adata, max_epochs)**
** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^**
** File “~/users/project_scvi/working/ourdata/run_autotune.py”, line 47, in setup_and_tun**
e_model
** results = autotune.run_autotune(**
** ^^^^^^^^^^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/scvi/autotune/_tune.py”, line 132, in run_autotune**
** experiment.result_grid = experiment.get_tuner().fit()**
** ^^^^^^^^^^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/scvi/autotune/_experiment.py”, line 475, in get_tuner**
** trainable = with_parameters(_trainable, experiment=self)**
** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/tune/trainable/util.py”, line 107, in with_parameters**
** parameter_registry.put(prefix + k, v)**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/tune/registry.py”, line 301, in put**
** self.flush()**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/tune/registry.py”, line 313, in flush**
** self.references[k] = ray.put(v)**
** ^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/auto_init_hook.py”, line 21, in auto_init_wrapper**
** return fn(args, kwargs)
** ^^^^^^^^^^^^^^^^^^^*
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/client_mode_hook.py”, line 103, in wrapper**
** return func(args, kwargs)
** ^^^^^^^^^^^^^^^^^^^^^*
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/worker.py”, line 2772, in put**
** object_ref = worker.put_object(value, owner_address=serialize_owner_address)**
** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^**
** File “~/users/conda/envs/scvi-env/lib/python3.12/site-packages/ray/_private/worker.py”, line 773, in put_object**
** raise TypeError(msg) from e**
TypeError: Could not serialize the put value Experiment scvi_10e3c6f1-8d59-4062-ab78-fa85b99f136b:
================================================================================
Checking Serializability of Experiment scvi_10e3c6f1-8d59-4062-ab78-fa85b99f136b
================================================================================
!!! FAIL serialization: h5py objects cannot be pickled
** Serializing ‘_data’ AnnData object with n_obs × n_vars = 3967777 × 2000 backed at ‘~/users/project_scvi/working/ourdata/snRNA_atlas_merged_hvg2k.h5ad’**
** obs: ‘age’, ‘diagnosis’, ‘cell_type’, ‘sample_id’, ‘source’, ‘sex’, ‘size_factors’, ‘_scvi_batch’, ‘_scvi_labels’**
** var: ‘highly_variable’, ‘highly_variable_rank’, ‘means’, ‘variances’, ‘variances_norm’, ‘highly_variable_nbatches’**
** uns: ‘hvg’, ‘_scvi_uuid’, ‘_scvi_manager_uuid’**
** obsm: ‘X_pca’, ‘X_umap’**
** layers: ‘counts’…**
** !!! FAIL serialization: h5py objects cannot be pickled**
** Serializing ‘X’ CSRDataset: backend hdf5, shape (3967777, 2000), data_dtype float64…**
** !!! FAIL serialization: h5py objects cannot be pickled**
** Serializing ‘_abc_impl’ <_abc._abc_data object at 0x150a689b8e00>…**
** !!! FAIL serialization: cannot pickle ‘_abc._abc_data’ object**
================================================================================
Variable:

** FailTuple(_abc_impl [obj=<_abc._abc_data object at 0x150a689b8e00>, parent=CSRDataset: backend hdf5, shape (3967777, 2000), data_dtype float64])**

was found to be non-serializable. There may be multiple other undetected variables that were non-serializable.
Consider either removing the instantiation/imports of these variables or moving the instantiation into the scope of the function/class.

2

Hi, you use AnnData object in backed mode. Can you verify this?
** FailTuple(_abc_impl [obj=<_abc._abc_data object at 0x150a689b8e00>, parent=CSRDataset: backend hdf5, shape (3967777, 2000), data_dtype float64])**

Ray can‘t handle backed data. You might be more successful with the dask backend (but it’s not tested in scVI-tools and will likely require work on our end to support).

3

ah yes. I did used backed when I loaded the data. Thank you!
Now I remove the backed mode and keep everything else. I got a new error saying
“TypeError: VAE.init() got an unexpected keyword argument ‘lr’”.

Am I doing it correctly with “lr”: tune.loguniform(1e-4, 1e-2) in the search_space statement?

5

@tangxj we had similar issue posted in our github repo 2 weeks ago.
you should check something similar to:

search_space = {
    "model_params": {"n_hidden": tune.choice([64, 128, 256]), 
                     "n_layers": tune.choice([1, 2, 3, 4]),
                     "n_latent": tune.choice([10, 20, 30, 40, 50]),
                     "gene_likelihood": tune.choice(["nb", "zinb"])
                    },
    "train_params": {"max_epochs": 100,
                    "plan_kwargs": {"lr": tune.loguniform(1e-4, 1e-2)}}}

and it should work

1 Like
6

@ori-kron-wis @cane11 Thank you both for the suggestion very much! It worked and the job has been running for two days and still keeps on running. I examined the current results in the log file (experiment_state_***.json), it is interesting that the one with minimal validation_loss out of the 21 runs is:

Item 20: validation_loss = 1388.1212158203125, experiment_tag = 21_gene_likelihood=zinb,n_hidden=256,n_latent=60,n_layers=2,max_epochs=200,lr=0.0007

The validation_loss ranged from 1388 to 1459. Does that mean I should take zinb as the gene likelihood as well as other params?

It seems I was greedy and gave too many options in the search space while I had no idea how long it would take. It seems I might not be able to get the whole tuning finished (my wall time limit was set to 4 days). What would be your suggestion based on your experience? Many thanks!