Issue running experiments on Norman

[kpetrovicc]

Hi,

Hope you are doing well! Thank you for open-sourcing your work on scDFM 😄

I believe there is an issue with pre-processing pipeline, I downloaded data from Zenodo and I ran the following

python src/script/run.py --batch_size=48 --devices='0' --model_type=origin --lr=5e-5 --steps=200000 --data_name=norman --d_model=128 --eta_min=1e-6 --fusion_method=differential_perceiver --infer_top_gene=1000 --n_top_genes=5000 --result_path=./result/additive --perturbation_function=crisper --noise_type=Gaussian --mode=predict_y --gamma=0.5 --split_method=additive --use_mmd_loss --fold=1 --topk=30 --use_negative_edge

I believe there is a data preprocessing pipeline that isn't explained in the readme or missing as ['condition'] column doesn't exist in the Norman h5ad file.

`import scanpy as sc
a = sc.read_h5ad('./data/norman.h5ad')
print('shape:', a.shape)
print('obs columns:', list(a.obs.columns))
print('first 3 rows:')
print(a.obs.head(3))
"
shape: (27658, 2000)
obs columns: ['guide_identity', 'read_count', 'UMI_count', 'coverage', 'gemgroup', 'good_coverage', 'number_of_cells', 'guide_merged', 'gene_program']
first 3 rows:
guide_identity read_count UMI_count ... number_of_cells guide_merged gene_program
cell_barcode ...
AAACCTGAGGCCCTTG-1 KLF1_MAP2K6__KLF1_MAP2K6 1037.0 59.0 ... 1.0 KLF1+MAP2K6 Pro-growth
AAACCTGCACGAAGCA-1 NegCtrl10_NegCtrl0__NegCtrl10_NegCtrl0 958.0 39.0 ... 1.0 ctrl Ctrl
AAACCTGCAGACGTAG-1 CEBPE_RUNX1T1__CEBPE_RUNX1T1 244.0 14.0 ... 1.0 CEBPE+RUNX1T1 Granulocyte/apoptosis

[3 rows x 9 columns]`

Could you please list all preprocessing steps to run the script?

Best,
Katarina

[oYCLo]

Hi Katarina,

Hope you are doing well, and thank you very much for your interest in scDFM!

Sorry for the confusion. We did not apply an additional preprocessing pipeline to the Norman dataset ourselves. The version used in our experiments is the preprocessed Norman dataset released by scFoundation, rather than the raw Norman .h5ad file from Zenodo.

The scFoundation processed dataset can be downloaded here:

https://figshare.com/articles/dataset/scFoundation_Large_Scale_Foundation_Model_on_Single-cell_Transcriptomics_-_processed_datasets/24049200?file=44477939

In this processed version, the metadata fields required by our script, including the perturbation/condition-related annotations, are already available or formatted consistently with our dataloader. This is why the raw Norman file you downloaded does not contain the condition column expected by the code.

We have also provided a Google Drive download link for the exact processed data used in our experiments, together with the corresponding split files used for evaluation:

https://drive.google.com/drive/folders/1cNpYAt9jVWZN82miNZtkP10YeSo7hufL

We apologize that this was not clearly explained in the README.

Best,
Chenglei Yu