Bug report [BUG] - pycisTopic QC

[TheeJonas]

Hey!

Describe the bug
I get a polars.exceptions.ColumnNotFoundError: CB error when running the pycistopic qc. I tried it both with my own data but also the provided tutorial with the same outcome.

To Reproduce
the code that throws the error is the following

!pycistopic qc \
    --fragments data/fragments.tsv.gz \
    --regions outs/consensus_peak_calling/consensus_regions.bed \
    --tss outs/qc/tss.bed \
    --output outs/qc/10x_multiome_brain

I run pycistopic on a cluster with an environment that i specifically created for it, following the tutorial code.
so i call pycistopic qc from that environment path directly

Error output

Traceback (most recent call last):
  File "/cfs/klemming/projects/snic/naiss2024-23-187/python/pycistopic/bin/pycistopic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/cfs/klemming/projects/supr/naiss2024-23-187/python/pycistopic/pycisTopic/src/pycisTopic/cli/pycistopic.py", line 26, in main
    args.func(args)
  File "/cfs/klemming/projects/supr/naiss2024-23-187/python/pycistopic/pycisTopic/src/pycisTopic/cli/subcommand/qc.py", line 233, in run_qc
    qc(
  File "/cfs/klemming/projects/supr/naiss2024-23-187/python/pycistopic/pycisTopic/src/pycisTopic/cli/subcommand/qc.py", line 144, in qc
    ) = compute_qc_stats(
        ^^^^^^^^^^^^^^^^^
  File "/cfs/klemming/projects/supr/naiss2024-23-187/python/pycistopic/pycisTopic/src/pycisTopic/qc.py", line 569, in compute_qc_stats
    .collect()
     ^^^^^^^^^
  File "/cfs/klemming/projects/snic/naiss2024-23-187/python/pycistopic/lib/python3.11/site-packages/polars/_utils/deprecation.py", line 97, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cfs/klemming/projects/snic/naiss2024-23-187/python/pycistopic/lib/python3.11/site-packages/polars/lazyframe/opt_flags.py", line 331, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cfs/klemming/projects/snic/naiss2024-23-187/python/pycistopic/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 2300, in collect
    return wrap_df(ldf.collect(engine, callback))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: CB

Resolved plan until failure:

	---> FAILED HERE RESOLVING 'sink' <---
DF ["by", "total_fragments_in_peaks_count", "unique_fragments_in_peaks_count"]; PROJECT */3 COLUMNS

Version
-python 3.11
-pycisTopic 2.0a0
-polars 1.30.0

Additional context
Since i got the same error with my data & the tutorial data i am just wondering if its a version issue or has to do with polars.

Let me know if you need more info

I would be happy for any help!

Jonas

[ghuls]

It could be a Polars version issue.

Can you post the output of your fragments file?

zcat data/fragments.tsv.gz | grep -v '^#' | head

[ghuls]

With the pycistopic_v3 branch and Polars 1.31.0 I don't get an error.
https://github.com/aertslab/pycisTopic/commits/pycistopic_v3/

[TheeJonas]

Hey!
Thanks for getting back to me! I will check the fragment file again but there did not seem to be an issue.
Regarding your second post, you suggest to try use the pycistopic_v3? Is that a new version in development? Sorry i am still new to github.
Or should i try to update my Polars to 1.31.0 with pycisTopic 2.0a0?

Thanks for the clarification in advance!

Best, Jonas

[TheeJonas]

With:
zcat data/fragments.tsv.gz | grep -v '^#' | head

I get:

chr1	10066	10198	GATGAGGGTGACATGC-1	1
chr1	10072	10303	TACGTACAGGAAGCTA-1	1
chr1	10073	10186	AACCGCTCAGCACCAT-1	2
chr1	10073	10272	GGTCATTAGCACAGGG-1	1
chr1	10090	10204	ATCACAATCCTCCTAA-1	1
chr1	10091	10327	GCGGTTGGTTAAGCCA-1	1
chr1	10095	10279	TGCTTGCTCGAGCTAT-1	2
chr1	10096	10284	ACTTAGTCACAATTAC-1	1
chr1	10103	10327	CAGGGCTTCCAGGTTG-1	1
chr1	10138	10346	TTTGAGTCATAAAGCA-1	1
grep: write error: Broken pipe

gzip: stdout: Broken pipe

[ghuls]

The fragments file looks fine.

For pycisTopic 2.0a0, you probably will have to downgrade Polars to a lower version (1.10.0 was released around that time, so that version might work).

The pycistopic_v3 branch is much faster than pycisTopic 2.0a0, but there is no tutorial yet how to use the new functions.

[beihzb]

Hi,

I also encountered the same error when using pycisTopic:

---> FAILED HERE RESOLVING 'sink' <---
DF ["by", "total_fragments_in_peaks_count", "unique_fragments_in_peaks_count"]; PROJECT */3 COLUMNS

After checking the package versions, I found that the issue is caused by an incompatible (too high) version of polars.

I suggest updating the pyproject.toml file to pin the dependencies as follows:
"numpy >= 1.20.3, < 2"
"polars >= 0.18.3, < 1"

Please make sure that subsequent installations of other packages do not unintentionally upgrade these versions. Be cautious when using conda, as it might alter pip dependencies and break the environment.

[LongpanICR]

Hi @ghuls ,
Would you mind telling us the plan about the approximate date of the relase of pycistopic v3?
It seems that a lot of error has been fix in the v3. Also, I saw you are continuously developing and refining the v3.
Best wishes,
Long