# Add new data to existing integration

**URL:** <https://discourse.scverse.org/t/add-new-data-to-existing-integration/3495>\
**Category:** scvi-tools\
**Created:** [February 19, 2025, 8:36pm UTC](https://discourse.scverse.org/t/add-new-data-to-existing-integration/3495 "2025-02-19T20:36:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tangxj](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@tangxj](https://discourse.scverse.org/u/tangxj)\
**Post date:** [February 19, 2025, 8:36pm UTC](https://discourse.scverse.org/t/add-new-data-to-existing-integration/3495/1 "2025-02-19T20:36:14Z")

</div>

Hi all,

I have a large integrated adata (~4M cells) through SCVI (with 2K HVG) and now I am going to add a small new single-cell data (28K cells) into the large integration. Is there a quick way to “project” the large integration to the smaller new data? Or do I have to start from the beginning, concatenate original full set and then select HVG and retraining for SCVI and SCANVI label transferring?

---

<div class="post-metadata">

**Author:** ![ori-kron-wis](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/ori-kron-wis/32/1099_2.png) [@ori-kron-wis](https://discourse.scverse.org/u/ori-kron-wis)\
**Post date:** [February 20, 2025, 1:32pm UTC](https://discourse.scverse.org/t/add-new-data-to-existing-integration/3495/2 "2025-02-20T13:32:52Z")

</div>

I think you are looking for scarches, where the ref data is your large integrated model and the query data is that small new single cell.  
please follow this tutorial:

> **[Reference mapping with scvi-tools](https://docs.scvi-tools.org/en/latest/tutorials/notebooks/scrna/scarches_scvi_tools.html)**
>
> This tutorial covers the usage of the scArches method with SCVI, SCANVI, and TOTALVI. This particular workflow is useful in the case where a model is trained on some data (called reference here) an...

---

<div class="post-metadata">

**Author:** ![tangxj](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@tangxj](https://discourse.scverse.org/u/tangxj)\
**Post date:** [March 6, 2025, 5:33pm UTC](https://discourse.scverse.org/t/add-new-data-to-existing-integration/3495/3 "2025-03-06T17:33:34Z")

</div>

I tried the scArches for scVI and it worked fine. However, I got errors when I used scArches for scANVI, it did not work. could you please see where might be wrong?

code:

```auto
# scANVI arches
# start from scvi model as trained for scArches above.  
scvi_model=scvi.model.SCVI.load("scvi-model_nb_scArches", adata_ref)

SCANVI_LABELS_KEY = "C_scANVI"

adata_ref.obs[SCANVI_LABELS_KEY] = adata_ref.obs["celltype_manual"].values

scanvi_model=scvi.model.SCANVI.from_scvi_model(scvi_model,unlabeled_category="Unknown",labels_key=SCANVI_LABELS_KEY)

scvi.settings.seed = 0
torch.set_float32_matmul_precision("high")

scanvi_model.train(max_epochs=20, n_samples_per_label=100)
scvi.model.SCANVI.prepare_query_anndata(adata_query, scanvi_model)
#INFO Found 100.0% reference vars in query data.                 
                                               
scanvi_query = scvi.model.SCANVI.load_query_data(adata_query, scanvi_model)

```

* * *

ValueError Traceback (most recent call last)  
Cell In[44], line 1  
----\> 1 scanvi\_query = scvi.model.SCANVI.load\_query\_data(adata\_query, scanvi\_model)

File /usrs/conda/envs/scvi-env/lib/python3.12/site-packages/scvi/model/base/\_archesmixin.py:123, in ArchesMixin.load\_query\_data(cls, adata, reference\_model, inplace\_subset\_query\_vars, accelerator, device, unfrozen, freeze\_dropout, freeze\_expression, freeze\_decoder\_first\_layer, freeze\_batchnorm\_encoder, freeze\_batchnorm\_decoder, freeze\_classifier)  
117 raise ValueError(  
118 "Saved model does not contain original setup inputs. "  
119 “Cannot load the original setup.”  
120 )  
122 setup\_method = getattr(cls, registry[\_SETUP\_METHOD\_NAME])  
 → 123 setup\_method(  
124 adata,  
125 source\_registry=registry,  
126 extend\_categories=True,  
127 allow\_missing\_labels=True,  
128 \*\*registry[\_SETUP\_ARGS\_KEY],  
129 )  
131 model = \_initialize\_model(cls, adata, attr\_dict)  
132 adata\_manager = model.get\_anndata\_manager(adata, required=True)

File /usrs/conda/envs/scvi-env/lib/python3.12/site-packages/scvi/model/\_scanvi.py:486, in SCANVI.setup\_anndata(cls, adata, labels\_key, unlabeled\_category, layer, batch\_key, size\_factor\_key, categorical\_covariate\_keys, continuous\_covariate\_keys, \*\*kwargs)  
484 anndata\_fields += cls.\_get\_fields\_for\_adata\_minification(adata\_minify\_type)  
485 adata\_manager = AnnDataManager(fields=anndata\_fields, setup\_method\_args=setup\_method\_args)  
 → 486 adata\_manager.register\_fields(adata, \*\*kwargs)  
487 cls.register\_manager(adata\_manager)

File /usrs/conda/envs/scvi-env/lib/python3.12/site-packages/scvi/data/\_manager.py:186, in AnnDataManager.register\_fields(self, adata, source\_registry, \*\*transfer\_kwargs)  
183 self.\_validate\_anndata\_object(adata)  
185 for field in self.fields:  
 → 186 self.\_add\_field(  
187 field=field,  
188 adata=adata,  
189 source\_registry=source\_registry,  
190 \*\*transfer\_kwargs,  
191 )  
193 # Save arguments for register\_fields.  
194 self.\_source\_registry = deepcopy(source\_registry)

File /usrs/conda/envs/scvi-env/lib/python3.12/site-packages/scvi/data/\_manager.py:221, in AnnDataManager.\_add\_field(self, field, adata, source\_registry, \*\*transfer\_kwargs)  
218 if not field.is\_empty:  
219 # Transfer case: Source registry is used for validation and/or setup.  
220 if source\_registry is not None:  
 → 221 field\_registry[\_constants.\_STATE\_REGISTRY\_KEY] = field.transfer\_field(  
222 source\_registry[\_constants.\_FIELD\_REGISTRIES\_KEY][field.registry\_key][  
223 \_constants.\_STATE\_REGISTRY\_KEY  
224 ],  
225 adata,  
226 \*\*transfer\_kwargs,  
227 )  
228 else:  
229 field\_registry[\_constants.\_STATE\_REGISTRY\_KEY] = field.register\_field(adata)

File /usrs/conda/envs/scvi-env/lib/python3.12/site-packages/scvi/data/fields/\_scanvi.py:105, in LabelsWithUnlabeledObsField.transfer\_field(self, state\_registry, adata\_target, allow\_missing\_labels, \*\*kwargs)  
103 if ec in kwargs:  
104 kwargs.pop(ec)  
 → 105 transfer\_state\_registry = super().transfer\_field(  
106 state\_registry, adata\_target, extend\_categories=False, \*\*kwargs  
107 )  
108 mapping = transfer\_state\_registry[self.CATEGORICAL\_MAPPING\_KEY]  
109 return self.\_remap\_unlabeled\_to\_final\_category(adata\_target, mapping)

File /usrs/conda/envs/scvi-env/lib/python3.12/site-packages/scvi/data/fields/\_dataframe\_field.py:219, in CategoricalDataFrameField.transfer\_field(self, state\_registry, adata\_target, extend\_categories, \*\*kwargs)  
217 mapping = np.concatenate([mapping, [c]])  
218 else:  
 → 219 raise ValueError(  
220 f"Category {c} not found in source registry. "  
221 f"Cannot transfer setup without `extend_categories = True`."  
222 )  
223 cat\_dtype = CategoricalDtype(categories=mapping, ordered=True)  
224 new\_mapping = \_make\_column\_categorical(  
225 getattr(adata\_target, self.attr\_name),  
226 self.\_original\_attr\_key,  
227 self.attr\_key,  
228 categorical\_dtype=cat\_dtype,  
229 )

ValueError: Category nan not found in source registry. Cannot transfer setup without `extend_categories = True`.

---

<div class="post-metadata">

**Author:** ![cane11](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/cane11/32/241_2.png) [@cane11](https://discourse.scverse.org/u/cane11)\
**Post date:** [March 7, 2025, 8:04am UTC](https://discourse.scverse.org/t/add-new-data-to-existing-integration/3495/4 "2025-03-07T08:04:27Z")

</div>

Hi, please set nans in your cell-type obs columm to the unlabeled category of your model or remove the column. Currently you are passing in a cell-type named „NaN“, which is not intended.