# Does processing the data always overwrite the raw counts?

**URL:** <https://discourse.scverse.org/t/does-processing-the-data-always-overwrite-the-raw-counts/1094>\
**Category:** scanpy\
**Created:** [February 2, 2023, 4:18am UTC](https://discourse.scverse.org/t/does-processing-the-data-always-overwrite-the-raw-counts/1094 "2023-02-02T04:18:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![natalia.nutella](https://avatars.discourse-cdn.com/v4/letter/n/bb73d2/32.png) [@natalia.nutella](https://discourse.scverse.org/u/natalia.nutella)\
**Post date:** [February 2, 2023, 4:18am UTC](https://discourse.scverse.org/t/does-processing-the-data-always-overwrite-the-raw-counts/1094/1 "2023-02-02T04:18:17Z")

</div>

I’m trying to understand the expected behavior in Scanpy re: what happens to different versions of the data during processing.

It’s my understanding that doing operations on the data always overwrites `.X` and one has to specifically copy the pre-modified data into a layer if you want to keep it. Is this correct?

ie If I have an anndata object that only has a raw counts matrix:

```auto
> adata = sc.read_h5ad('test.h5ad')

> adata

AnnData object with n_obs × n_vars = 4977 × 30721
    obs: 'n_genes'
    var: 'featureid', 'n_cells'
    uns: 'genome', 'modality', 'uid'

> adata.X

<4977x30721 sparse matrix of type '<class 'numpy.uint32'>'
	with 16718362 stored elements in Compressed Sparse Row format>

```

Then running normalization on it will overwrite the counts and not store them somewhere else:  
(does not create a layer to store the raw counts, does not store them in adata.raw)

```auto
> scanpy.pp.normalize_total(adata)

adata

AnnData object with n_obs × n_vars = 4977 × 30721
    obs: 'n_genes'
    var: 'featureid', 'n_cells'
    uns: 'genome', 'modality', 'uid'
 # no layers created

# .X is now float, so presumably normalized
adata.X

<4977x30721 sparse matrix of type '<class 'numpy.float32'>'
	with 16718362 stored elements in Compressed Sparse Row format>

> adata.raw.X
AttributeError: 'NoneType' object has no attribute 'X'

```

This is different from `pegasus` where sequential operations create new matrices in the object without the user having to specify:

Example, an object that was log-normalized with pegasus:

```auto
adata_pg

MultimodalData object with 1 UnimodalData: 'GRCh38-rna'
    It currently binds to UnimodalData object GRCh38-rna

UnimodalData object with n_obs x n_vars = 3538 x 36484
    UID: GRCh38-rna; Genome: GRCh38; Modality: rna
    It contains 2 matrices: 'counts', 'counts.log_norm'
    It currently binds to matrix 'counts.log_norm' as X

```

And different from Seurat, where default behavior is to store the different versions of the data in `@counts,` `@data`, `@scaled`.

Am I correct in my interpretation? And why is this not the case in Scanpy?

---

<div class="post-metadata">

**Author:** ![wbrett87](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/wbrett87/32/517_2.png) [@wbrett87](https://discourse.scverse.org/u/wbrett87)\
**Post date:** [February 5, 2023, 1:27pm UTC](https://discourse.scverse.org/t/does-processing-the-data-always-overwrite-the-raw-counts/1094/2 "2023-02-05T13:27:23Z")

</div>

This is also not completely clear to me either!

---

<div class="post-metadata">

**Author:** ![Jingheng-ZHANG-1999](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/jingheng-zhang-1999/32/684_2.png) [@Jingheng-ZHANG-1999](https://discourse.scverse.org/u/Jingheng-ZHANG-1999)\
**Post date:** [June 1, 2023, 5:24pm UTC](https://discourse.scverse.org/t/does-processing-the-data-always-overwrite-the-raw-counts/1094/3 "2023-06-01T17:24:13Z")

</div>

I think you’re right.  
I also discovered using `adata=adata.raw.to_adata()` not only restores anndata.X but also erases layers on it, making it a less preferred approach to store raw count matrix.  
Now I use:

```auto
adata=sc.read_h5ad('xxx.h5ad')
adata.layers['counts']=adata.X.copy() 
sc.pp.normalize_total(adata,exclude_highly_expressed=False) 
adata.layers['cpm']=adata.X.copy() 
sc.pp.log1p(adata)
adata.layers['data']=adata.X.copy() 
sc.pp.highly_variable_genes(adata,batch_key='sample',flavor='seurat_v3',layer="counts",n_top_genes=2500) 
sc.pp.regress_out(adata, ['pct_counts_mt'])
sc.pp.scale(adata, max_value=10)
adata.layers['scaled']=adata.X.copy()

```

A bit clumsey, but makes me feel safe.

---

<div class="post-metadata">

**Author:** ![cdpolt](https://yyz1.discourse-cdn.com/flex035/user_avatar/discourse.scverse.org/cdpolt/32/628_2.png) [@cdpolt](https://discourse.scverse.org/u/cdpolt)\
**Post date:** [April 22, 2024, 6:52pm UTC](https://discourse.scverse.org/t/does-processing-the-data-always-overwrite-the-raw-counts/1094/4 "2024-04-22T18:52:31Z")

</div>

Does this still work for you? When I try and do this now, the operations also work on all layers so by the end you would have a bunch of layers that all look identical…