Clustering with X_scVI after removing doublets

I am following instructions in SOLO to find doublets when having two batches. When I don’t care about doublets I use X-scVI latent space to cluster cells. Now that I have doublets I was wondering if I just want to cluster singlets, should I use another latent space to cluster? Since X-scVI is based on the info of all cells regardless of doublets or singlets (right?) and it doesn’t make sense to filter out cells (doublets) and still use the old X-scVI for clustering rest of cells (Singlets). What is the best strategy here? Thank you.