Error loading previously saved scVI model after updating packages / cloud environment

Help scvi-tools
1

Hello. I recently updated my environment due to server issues which resulted in having to install new package versions (scvi-tools, pandas, etc). I was unable to check the package versions I was using prior to updating the environment before I updated it. I do believe I was using a version of scvi-tools newer than v0.15.0 because my model has the structure ‘model.pt’ (I believe an older version would have the structure ‘model_params.pt’). In the updated environment I get an error when trying to load a previously saved scVI model using scvi.model.SCVI.load(). The error is as follows:

TypeError: Argument 'placement' has incorrect type (expected pandas._libs.internals.BlockPlacement, got slice).

I do not think that I can use convert_legacy_save to fix this error due to the format the model is saved in.

I also get the following warning, as only 1 of the 2 available GPUs is being used:

Trainer will use only 1 of 2 GPUs because it is running inside an interactive / notebook environment. You may try to set `Trainer(devices=2)` but please note that multi-GPU inside interactive / notebook environments is considered experimental and unstable. Your mileage may vary.

Has anyone experienced something similar? Any input would be appreciated.

Thank you,
Julianne

2

Hi, sorry you’re experiencing this issue. What library is raising the TypeError? Feel free to past the full traceback of the error.

3

Here’s the whole error:

TypeError                                 Traceback (most recent call last)
Cell In[13], line 5
      1 # tutorial section is called "train reference"
      2 # reference scVI model was trained in scVI with batch_key & EPI annotation notebook
      3 # here, load the trained model
----> 5 model = scvi.model.SCVI.load(working_path + "scvi_model_FGID_and_Healthy_version10x_batch_using_batchkey/",
      6                              adata, accelerator = 'gpu')
      7                              #use_gpu=True

File ~/.local/lib/python3.10/site-packages/scvi/model/base/_base_model.py:674, in BaseModelClass.load(cls, dir_path, adata, accelerator, device, prefix, backup_url)
    661 load_adata = adata is None
    662 _, _, device = parse_device_args(
    663     accelerator=accelerator,
    664     devices=device,
    665     return_device="torch",
    666     validate_single_device=True,
    667 )
    669 (
    670     attr_dict,
    671     var_names,
    672     model_state_dict,
    673     new_adata,
--> 674 ) = _load_saved_files(
    675     dir_path,
    676     load_adata,
    677     map_location=device,
    678     prefix=prefix,
    679     backup_url=backup_url,
    680 )
    681 adata = new_adata if new_adata is not None else adata
    683 _validate_var_names(adata, var_names)

File ~/.local/lib/python3.10/site-packages/scvi/model/base/_utils.py:66, in _load_saved_files(dir_path, load_adata, prefix, map_location, backup_url)
     64 try:
     65     _download(backup_url, dir_path, model_file_name)
---> 66     model = torch.load(model_path, map_location=map_location)
     67 except FileNotFoundError as exc:
     68     raise ValueError(
     69         f"Failed to load model file at {model_path}. "
     70         "If attempting to load a saved model from <v0.15.0, please use the util function "
     71         "`convert_legacy_save` to convert to an updated format."
     72     ) from exc

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1025, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
   1023             except RuntimeError as e:
   1024                 raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1025         return _load(opened_zipfile,
   1026                      map_location,
   1027                      pickle_module,
   1028                      overall_storage=overall_storage,
   1029                      **pickle_load_args)
   1030 if mmap:
   1031     f_name = "" if not isinstance(f, str) else f"{f}, "

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1446, in _load(zip_file, map_location, pickle_module, pickle_file, overall_storage, **pickle_load_args)
   1444 unpickler = UnpicklerWrapper(data_file, **pickle_load_args)
   1445 unpickler.persistent_load = persistent_load
-> 1446 result = unpickler.load()
   1448 torch._utils._validate_loaded_sparse_tensors()
   1449 torch._C._log_api_usage_metadata(
   1450     "torch.load.metadata", {"serialization_id": zip_file.serialization_id()}
   1451 )

File ~/.local/lib/python3.10/site-packages/pandas/core/internals/blocks.py:2728, in new_block(values, placement, ndim, refs)
   2716 def new_block(
   2717     values,
   2718     placement: BlockPlacement,
   (...)
   2725     # - check_ndim/ensure_block_shape already checked
   2726     # - maybe_coerce_values already called/unnecessary
   2727     klass = get_block_type(values.dtype)
-> 2728     return klass(values, ndim=ndim, placement=placement, refs=refs)

TypeError: Argument 'placement' has incorrect type (expected pandas._libs.internals.BlockPlacement, got slice)
4

It looks like an inconsistency in pandas versions. This is not strictly an error in scVI-tools but PyTorch unpickling expects the same version of pandas is my understanding (correct me if I’m wrong @martinkim0).

5

Hi Martin and Can,

I am experiencing a similar issue after updating my scvi-tools version from 0.19.0 to 1.1.2. My current version of pandas is 2.0.3 and I am not sure what my old version was. Here is the code I am trying to run:

model = scvi.model.SCVI.load(working_path + 
                             "/new_scvi_model_Epithelial_batch10x/", 
                             adata)

Here is the error I am getting:

INFO     File /home/jupyter/new_scvi_model_Epithelial_batch10x/model.pt already downloaded                         
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[70], line 1
----> 1 model = scvi.model.SCVI.load(working_path + 
      2                              "/new_scvi_model_Epithelial_batch10x/", 
      3                              adata)
      4                              #accelerator='gpu')

File ~/.local/lib/python3.10/site-packages/scvi/model/base/_base_model.py:674, in BaseModelClass.load(cls, dir_path, adata, accelerator, device, prefix, backup_url)
    661 load_adata = adata is None
    662 _, _, device = parse_device_args(
    663     accelerator=accelerator,
    664     devices=device,
    665     return_device="torch",
    666     validate_single_device=True,
    667 )
    669 (
    670     attr_dict,
    671     var_names,
    672     model_state_dict,
    673     new_adata,
--> 674 ) = _load_saved_files(
    675     dir_path,
    676     load_adata,
    677     map_location=device,
    678     prefix=prefix,
    679     backup_url=backup_url,
    680 )
    681 adata = new_adata if new_adata is not None else adata
    683 _validate_var_names(adata, var_names)

File ~/.local/lib/python3.10/site-packages/scvi/model/base/_utils.py:66, in _load_saved_files(dir_path, load_adata, prefix, map_location, backup_url)
     64 try:
     65     _download(backup_url, dir_path, model_file_name)
---> 66     model = torch.load(model_path, map_location=map_location)
     67 except FileNotFoundError as exc:
     68     raise ValueError(
     69         f"Failed to load model file at {model_path}. "
     70         "If attempting to load a saved model from <v0.15.0, please use the util function "
     71         "`convert_legacy_save` to convert to an updated format."
     72     ) from exc

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1025, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
   1023             except RuntimeError as e:
   1024                 raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1025         return _load(opened_zipfile,
   1026                      map_location,
   1027                      pickle_module,
   1028                      overall_storage=overall_storage,
   1029                      **pickle_load_args)
   1030 if mmap:
   1031     f_name = "" if not isinstance(f, str) else f"{f}, "

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1446, in _load(zip_file, map_location, pickle_module, pickle_file, overall_storage, **pickle_load_args)
   1444 unpickler = UnpicklerWrapper(data_file, **pickle_load_args)
   1445 unpickler.persistent_load = persistent_load
-> 1446 result = unpickler.load()
   1448 torch._utils._validate_loaded_sparse_tensors()
   1449 torch._C._log_api_usage_metadata(
   1450     "torch.load.metadata", {"serialization_id": zip_file.serialization_id()}
   1451 )

File ~/.local/lib/python3.10/site-packages/torch/serialization.py:1439, in _load.<locals>.UnpicklerWrapper.find_class(self, mod_name, name)
   1437         pass
   1438 mod_name = load_module_mapping.get(mod_name, mod_name)
-> 1439 return super().find_class(mod_name, name)

ModuleNotFoundError: No module named 'pandas.core.indexes.numeric'

Thank you for your help!
Kate

6

Ah yeah, ModuleNotFoundError: No module named 'pandas.core.indexes.numeric' is because the model was previously saved with pandas < 2.0 but loaded with pandas >= 2.0. You can fix this by downgrading to pandas < 2.0.

7

If people get here, a simple fix is adding the following to your script before loading a model (the issue are integer as pandas Index in model.history). Fixing this is not straightforward (pandas <2.0 handles integer indices differently). However, the fix does not apply to old saved models but only to newly stored models.

import sys
import pandas
sys.modules['pandas.core.indexes.numeric'] = pandas.core.indexes.base
pandas.core.indexes.base.Int64Index = pandas.core.indexes.base.Index