Error loading previously saved scVI model after updating packages / cloud environment

Hello. I recently updated my environment due to server issues which resulted in having to install new package versions (scvi-tools, pandas, etc). I was unable to check the package versions I was using prior to updating the environment before I updated it. I do believe I was using a version of scvi-tools newer than v0.15.0 because my model has the structure ‘model.pt’ (I believe an older version would have the structure ‘model_params.pt’). In the updated environment I get an error when trying to load a previously saved scVI model using scvi.model.SCVI.load(). The error is as follows:

TypeError: Argument 'placement' has incorrect type (expected pandas._libs.internals.BlockPlacement, got slice).

I do not think that I can use convert_legacy_save to fix this error due to the format the model is saved in.

I also get the following warning, as only 1 of the 2 available GPUs is being used:

Trainer will use only 1 of 2 GPUs because it is running inside an interactive / notebook environment. You may try to set `Trainer(devices=2)` but please note that multi-GPU inside interactive / notebook environments is considered experimental and unstable. Your mileage may vary.

Has anyone experienced something similar? Any input would be appreciated.

Thank you,
Julianne

Hi, sorry you’re experiencing this issue. What library is raising the TypeError? Feel free to past the full traceback of the error.

Here’s the whole error:

TypeError                                 Traceback (most recent call last)
Cell In[13], line 5
      1 # tutorial section is called "train reference"
      2 # reference scVI model was trained in scVI with batch_key & EPI annotation notebook
      3 # here, load the trained model
----> 5 model = scvi.model.SCVI.load(working_path + "scvi_model_FGID_and_Healthy_version10x_batch_using_batchkey/",
      6                              adata, accelerator = 'gpu')
      7                              #use_gpu=True

File ~/.local/lib/python3.10/site-packages/scvi/model/base/_base_model.py:674, in BaseModelClass.load(cls, dir_path, adata, accelerator, device, prefix, backup_url)
    661 load_adata = adata is None
    662 _, _, device = parse_device_args(
    663     accelerator=accelerator,
    664     devices=device,
    665     return_device="torch",
    666     validate_single_device=True,
    667 )
    669 (
    670     attr_dict,
    671     var_names,
    672     model_state_dict,
    673     new_adata,
--> 674 ) = _load_saved_files(
    675     dir_path,
    676     load_adata,
    677     map_location=device,
    678     prefix=prefix,
    679     backup_url=backup_url,
    680 )
    681 adata = new_adata if new_adata is not None else adata
    683 _validate_var_names(adata, var_names)

File ~/.local/lib/python3.10/site-packages/scvi/model/base/_utils.py:66, in _load_saved_files(dir_path, load_adata, prefix, map_location, backup_url)
     64 try:
     65     _download(backup_url, dir_path, model_file_name)
---> 66     model = torch.load(model_path, map_location=map_location)
     67 except FileNotFoundError as exc:
     68     raise ValueError(
     69         f"Failed to load model file at {model_path}. "
     70         "If attempting to load a saved model from <v0.15.0, please use the util function "
     71         "`convert_legacy_save` to convert to an updated format."
     72     ) from exc

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1025, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
   1023             except RuntimeError as e:
   1024                 raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1025         return _load(opened_zipfile,
   1026                      map_location,
   1027                      pickle_module,
   1028                      overall_storage=overall_storage,
   1029                      **pickle_load_args)
   1030 if mmap:
   1031     f_name = "" if not isinstance(f, str) else f"{f}, "

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1446, in _load(zip_file, map_location, pickle_module, pickle_file, overall_storage, **pickle_load_args)
   1444 unpickler = UnpicklerWrapper(data_file, **pickle_load_args)
   1445 unpickler.persistent_load = persistent_load
-> 1446 result = unpickler.load()
   1448 torch._utils._validate_loaded_sparse_tensors()
   1449 torch._C._log_api_usage_metadata(
   1450     "torch.load.metadata", {"serialization_id": zip_file.serialization_id()}
   1451 )

File ~/.local/lib/python3.10/site-packages/pandas/core/internals/blocks.py:2728, in new_block(values, placement, ndim, refs)
   2716 def new_block(
   2717     values,
   2718     placement: BlockPlacement,
   (...)
   2725     # - check_ndim/ensure_block_shape already checked
   2726     # - maybe_coerce_values already called/unnecessary
   2727     klass = get_block_type(values.dtype)
-> 2728     return klass(values, ndim=ndim, placement=placement, refs=refs)

TypeError: Argument 'placement' has incorrect type (expected pandas._libs.internals.BlockPlacement, got slice)

It looks like an inconsistency in pandas versions. This is not strictly an error in scVI-tools but PyTorch unpickling expects the same version of pandas is my understanding (correct me if I’m wrong @martinkim0).

Hi Martin and Can,

I am experiencing a similar issue after updating my scvi-tools version from 0.19.0 to 1.1.2. My current version of pandas is 2.0.3 and I am not sure what my old version was. Here is the code I am trying to run:

model = scvi.model.SCVI.load(working_path + 
                             "/new_scvi_model_Epithelial_batch10x/", 
                             adata)

Here is the error I am getting:

INFO     File /home/jupyter/new_scvi_model_Epithelial_batch10x/model.pt already downloaded                         
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[70], line 1
----> 1 model = scvi.model.SCVI.load(working_path + 
      2                              "/new_scvi_model_Epithelial_batch10x/", 
      3                              adata)
      4                              #accelerator='gpu')

File ~/.local/lib/python3.10/site-packages/scvi/model/base/_base_model.py:674, in BaseModelClass.load(cls, dir_path, adata, accelerator, device, prefix, backup_url)
    661 load_adata = adata is None
    662 _, _, device = parse_device_args(
    663     accelerator=accelerator,
    664     devices=device,
    665     return_device="torch",
    666     validate_single_device=True,
    667 )
    669 (
    670     attr_dict,
    671     var_names,
    672     model_state_dict,
    673     new_adata,
--> 674 ) = _load_saved_files(
    675     dir_path,
    676     load_adata,
    677     map_location=device,
    678     prefix=prefix,
    679     backup_url=backup_url,
    680 )
    681 adata = new_adata if new_adata is not None else adata
    683 _validate_var_names(adata, var_names)

File ~/.local/lib/python3.10/site-packages/scvi/model/base/_utils.py:66, in _load_saved_files(dir_path, load_adata, prefix, map_location, backup_url)
     64 try:
     65     _download(backup_url, dir_path, model_file_name)
---> 66     model = torch.load(model_path, map_location=map_location)
     67 except FileNotFoundError as exc:
     68     raise ValueError(
     69         f"Failed to load model file at {model_path}. "
     70         "If attempting to load a saved model from <v0.15.0, please use the util function "
     71         "`convert_legacy_save` to convert to an updated format."
     72     ) from exc

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1025, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
   1023             except RuntimeError as e:
   1024                 raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1025         return _load(opened_zipfile,
   1026                      map_location,
   1027                      pickle_module,
   1028                      overall_storage=overall_storage,
   1029                      **pickle_load_args)
   1030 if mmap:
   1031     f_name = "" if not isinstance(f, str) else f"{f}, "

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1446, in _load(zip_file, map_location, pickle_module, pickle_file, overall_storage, **pickle_load_args)
   1444 unpickler = UnpicklerWrapper(data_file, **pickle_load_args)
   1445 unpickler.persistent_load = persistent_load
-> 1446 result = unpickler.load()
   1448 torch._utils._validate_loaded_sparse_tensors()
   1449 torch._C._log_api_usage_metadata(
   1450     "torch.load.metadata", {"serialization_id": zip_file.serialization_id()}
   1451 )

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1439, in _load.<locals>.UnpicklerWrapper.find_class(self, mod_name, name)
   1437         pass
   1438 mod_name = load_module_mapping.get(mod_name, mod_name)
-> 1439 return super().find_class(mod_name, name)

ModuleNotFoundError: No module named 'pandas.core.indexes.numeric'

Thank you for your help!
Kate

Ah yeah, ModuleNotFoundError: No module named 'pandas.core.indexes.numeric' is because the model was previously saved with pandas < 2.0 but loaded with pandas >= 2.0. You can fix this by downgrading to pandas < 2.0.