Error running mu.tl.mofa

Hi there,
I am using muon to run MOFA on my multi-omics data. The data is first passed to MuData and mu.pp.intersect_obs. When I run mu.tl.mofa , MOFA just gets interrupted:

Attempting to save the model at the current iteration…
Saving model in mu_brain_interrupted.hdf5…
Note: the model to be saved is not trained.
Do you know why is that happening?

my mudata looks like that:
MuData object with n_obs × n_vars = 2781 × 276086
2 modalities
rna: 2781 x 6421
atac: 2781 x 269665

Thank you very much in advance!

Hey @nelkazwi!

Do you think you could provide some stack trace with the error, either here or on GitHub? MOFA can save a (not fully trained) model when the training is interrupted but that usually means it receives an external signal to interrupt the training.

Another useful way to investigate the issue might be to check if it works on a smaller dataset.

Hi,
I tried smaller datasets and I am getting this error:

Loaded view=‘atac’ group=‘group1’ with N=4032 samples and D=8179 features…
Loaded view=‘rna’ group=‘group1’ with N=4032 samples and D=862 features…

Model options:

  • Automatic Relevance Determination prior on the factors: True
  • Automatic Relevance Determination prior on the weights: True
  • Spike-and-slab prior on the factors: False
  • Spike-and-slab prior on the weights: True
    Likelihoods:
  • View 0 (atac): poisson
  • View 1 (rna): poisson

GPU mode is activated

######################################

Training the model with seed 1

######################################

Exiting now without saving the partially trained model. To save a partially trained model, set save_interrupted in the training options to true.

TypeError Traceback (most recent call last)
File ~/miniconda3/lib/python3.9/site-packages/mofapy2/run/entry_point.py:49, in keyboardinterrupt_saver..saver(self, *args, **kwargs)
48 try:
—> 49 func(self, *args, **kwargs)
50 # Internal methods will raise TypeError when interrupted

File ~/miniconda3/lib/python3.9/site-packages/mofapy2/run/entry_point.py:1020, in entry_point.run(self)
1019 # Train the model
→ 1020 train_model(self.model)

File ~/miniconda3/lib/python3.9/site-packages/mofapy2/build_model/train_model.py:28, in train_model(model)
26 print (“\n”)
—> 28 model.iterate()
30 print(“\n”)

File ~/miniconda3/lib/python3.9/site-packages/mofapy2/core/BayesNet.py:224, in BayesNet.iterate(self)
223 converged = False; convergence_token = 1
→ 224 elbo.iloc[0] = self.precompute()
225 number_factors[0] = self.dim[‘K’]

File ~/miniconda3/lib/python3.9/site-packages/mofapy2/core/BayesNet.py:193, in BayesNet.precompute(self)
192 for n in self.nodes:
→ 193 self.nodes[n].precompute(self.options)
195 # Precompute ELBO

File ~/miniconda3/lib/python3.9/site-packages/mofapy2/core/nodes/multiview_nodes.py:105, in Multiview_Node.precompute(self, options)
104 for m in self.activeM:
→ 105 self.nodes[m].precompute(options)

File ~/miniconda3/lib/python3.9/site-packages/mofapy2/core/nodes/nongaussian_nodes.py:189, in Poisson_PseudoY.precompute(self, options)
188 self.updateParameters()
→ 189 self.updateExpectations()

File ~/miniconda3/lib/python3.9/site-packages/mofapy2/core/nodes/nongaussian_nodes.py:202, in Poisson_PseudoY.updateExpectations(self)
201 tau = self.markov_blanket[“Tau”].getValue()
→ 202 self.E = self.params[“zeta”] - sigmoid(self.params[“zeta”])*(1-self.obs/self.ratefn(self.params[“zeta”])) / tau
203 self.E[self.mask] = 0.

File cupy/_core/core.pyx:1676, in cupy._core.core._ndarray_base.array_ufunc()

File cupy/_core/_kernel.pyx:1222, in cupy._core._kernel.ufunc.call()

File cupy/_core/_kernel.pyx:138, in cupy._core._kernel._preprocess_args()

File cupy/_core/_kernel.pyx:124, in cupy._core._kernel._preprocess_arg()

TypeError: Unsupported type <class ‘numpy.ndarray’>

During handling of the above exception, another exception occurred:

SystemExit Traceback (most recent call last)
[… skipping hidden 1 frame]

Input In [4], in <cell line: 1>()
----> 1 mu.tl.mofa(mdata,n_factors=20, gpu_mode=True,save_interrupted=False)

File ~/miniconda3/lib/python3.9/site-packages/muon/_core/tools.py:540, in mofa(data, groups_label, use_raw, use_layer, use_var, use_obs, likelihoods, n_factors, scale_views, scale_groups, center_groups, ard_weights, ard_factors, spikeslab_weights, spikeslab_factors, n_iterations, convergence_mode, gpu_mode, use_float32, smooth_covariate, smooth_warping, smooth_kwargs, save_parameters, save_data, save_metadata, seed, outfile, expectations, save_interrupted, verbose, quiet, copy)
539 logging.info(f"[{datetime.now().strftime(‘%Y-%m-%d %H:%M:%S’)}] Running the model…")
→ 540 ent.run()
542 if (
543 smooth_kwargs is not None
544 and “new_values” in smooth_kwargs
545 and smooth_kwargs[“new_values”]
546 and smooth_covariate
547 ):

File ~/miniconda3/lib/python3.9/site-packages/mofapy2/run/entry_point.py:64, in keyboardinterrupt_saver..saver(self, *args, **kwargs)
63 sys.stdout.flush()
—> 64 sys.exit()

SystemExit:

During handling of the above exception, another exception occurred:

AssertionError Traceback (most recent call last)
[… skipping hidden 1 frame]

File ~/miniconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py:1983, in InteractiveShell.showtraceback(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)
1980 if exception_only:
1981 stb = ['An exception has occurred, use %tb to see ’
1982 ‘the full traceback.\n’]
→ 1983 stb.extend(self.InteractiveTB.get_exception_only(etype,
1984 value))
1985 else:
1986 try:
1987 # Exception classes can customise their traceback - we
1988 # use this in IPython.parallel for exceptions occurring
1989 # in the engines. This should return a list of strings.

File ~/miniconda3/lib/python3.9/site-packages/IPython/core/ultratb.py:585, in ListTB.get_exception_only(self, etype, value)
577 def get_exception_only(self, etype, value):
578 “”“Only print the exception type and message, without a traceback.
579
580 Parameters
(…)
583 value : exception value
584 “””
→ 585 return ListTB.structured_traceback(self, etype, value)

File ~/miniconda3/lib/python3.9/site-packages/IPython/core/ultratb.py:443, in ListTB.structured_traceback(self, etype, evalue, etb, tb_offset, context)
440 chained_exc_ids.add(id(exception[1]))
441 chained_exceptions_tb_offset = 0
442 out_list = (
→ 443 self.structured_traceback(
444 etype, evalue, (etb, chained_exc_ids),
445 chained_exceptions_tb_offset, context)
446 + chained_exception_message
447 + out_list)
449 return out_list

File ~/miniconda3/lib/python3.9/site-packages/IPython/core/ultratb.py:1118, in AutoFormattedTB.structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
1116 else:
1117 self.tb = tb
→ 1118 return FormattedTB.structured_traceback(
1119 self, etype, value, tb, tb_offset, number_of_lines_of_context)

File ~/miniconda3/lib/python3.9/site-packages/IPython/core/ultratb.py:1012, in FormattedTB.structured_traceback(self, etype, value, tb, tb_offset, number_of_lines_of_context)
1009 mode = self.mode
1010 if mode in self.verbose_modes:
1011 # Verbose modes need a full traceback
→ 1012 return VerboseTB.structured_traceback(
1013 self, etype, value, tb, tb_offset, number_of_lines_of_context
1014 )
1015 elif mode == ‘Minimal’:
1016 return ListTB.get_exception_only(self, etype, value)

File ~/miniconda3/lib/python3.9/site-packages/IPython/core/ultratb.py:865, in VerboseTB.structured_traceback(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)
856 def structured_traceback(
857 self,
858 etype: type,
(…)
862 number_of_lines_of_context: int = 5,
863 ):
864 “”“Return a nice text document describing the traceback.”“”
→ 865 formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
866 tb_offset)
868 colors = self.Colors # just a shorthand + quicker name lookup
869 colorsnormal = colors.Normal # used a lot

File ~/miniconda3/lib/python3.9/site-packages/IPython/core/ultratb.py:799, in VerboseTB.format_exception_as_a_whole(self, etype, evalue, etb, number_of_lines_of_context, tb_offset)
796 assert isinstance(tb_offset, int)
797 head = self.prepare_header(etype, self.long_header)
798 records = (
→ 799 self.get_records(etb, number_of_lines_of_context, tb_offset) if etb else
800 )
802 frames =
803 skipped = 0

File ~/miniconda3/lib/python3.9/site-packages/IPython/core/ultratb.py:854, in VerboseTB.get_records(self, etb, number_of_lines_of_context, tb_offset)
848 formatter = None
849 options = stack_data.Options(
850 before=before,
851 after=after,
852 pygments_formatter=formatter,
853 )
→ 854 return list(stack_data.FrameInfo.stack_data(etb, options=options))[tb_offset:]

File ~/miniconda3/lib/python3.9/site-packages/stack_data/core.py:546, in FrameInfo.stack_data(cls, frame_or_tb, options, collapse_repeated_frames)
530 @classmethod
531 def stack_data(
532 cls,
(…)
536 collapse_repeated_frames: bool = True
537 ) → Iterator[Union[‘FrameInfo’, RepeatedFrames]]:
538 “”"
539 An iterator of FrameInfo and RepeatedFrames objects representing
540 a full traceback or stack. Similar consecutive frames are collapsed into RepeatedFrames
(…)
544 and optionally an Options object to configure.
545 “”"
→ 546 stack = list(iter_stack(frame_or_tb))
548 # Reverse the stack from a frame so that it’s in the same order
549 # as the order from a traceback, which is the order of a printed
550 # traceback when read top to bottom (most recent call last)
551 if is_frame(frame_or_tb):

File ~/miniconda3/lib/python3.9/site-packages/stack_data/utils.py:98, in iter_stack(frame_or_tb)
96 while frame_or_tb:
97 yield frame_or_tb
—> 98 if is_frame(frame_or_tb):
99 frame_or_tb = frame_or_tb.f_back
100 else:

File ~/miniconda3/lib/python3.9/site-packages/stack_data/utils.py:91, in is_frame(frame_or_tb)
90 def is_frame(frame_or_tb: Union[FrameType, TracebackType]) → bool:
—> 91 assert_(isinstance(frame_or_tb, (types.FrameType, types.TracebackType)))
92 return isinstance(frame_or_tb, (types.FrameType,))

File ~/miniconda3/lib/python3.9/site-packages/stack_data/utils.py:172, in assert_(condition, error)
170 if isinstance(error, str):
171 error = AssertionError(error)
→ 172 raise error

AssertionError:
Do you know what is the problem?
Thank you so much!