Skip to content

README.md

Latest commit

 

History

History
215 lines (158 loc) · 5.46 KB

README.md

File metadata and controls

215 lines (158 loc) · 5.46 KB
Raw

ClairS Model Training Guide

This directory contains template workflows for training ClairS long-read somatic models. The scripts use local command names and local input file names, and write outputs under a local output directory.

ClairS uses two training stages:

  1. SS training: synthetic somatic training from two unrelated high-confidence samples.
  2. SSRS training: real-sample refinement using tumor/normal benchmark samples and existing ClairS SS checkpoints.

Run SS first, then run SSRS.

Prerequisites

  • clairs.py available in PATH, or update CLAIRS in the scripts.
  • python3, pypy3, parallel, samtools, bedtools, mosdepth, tabix, and whatshap available in PATH.
  • Position-sorted and indexed BAM files.
  • Reference FASTA and index files.
  • Truth VCFs and benchmark BED files.
  • A GPU for model training.
  • Enough disk space for phased BAMs, synthetic BAMs, tensors, bins, candidate files, and model checkpoints.

Contents

  • clairs_ss_model_training_workflow.sh

    • SS workflow.
    • Phases and haplotags two unrelated samples.
    • Generates synthetic normal/tumor BAMs at multiple coverage and purity settings.
    • Trains SNV pileup, SNV full-alignment, INDEL pileup, and INDEL full-alignment models.

  • clairs_ssrs_model_training_workflow.sh

    • SSRS workflow.
    • Uses real tumor/normal benchmark samples.
    • Haplotags tumor reads with ClairS.
    • Initializes training from installed ClairS SS checkpoints.
    • Writes refined SSRS checkpoints under output/train_ssrs.

I. SS Training Workflow

Setup variables

Edit the top of clairs_ss_model_training_workflow.sh if your tool names or input file names differ:

CLAIRS='clairs.py'
PYTHON='python3'
PYPY='pypy3'
WHATSHAP='whatshap'
PARALLEL='parallel'
SAMTOOLS='samtools'
BEDTOOLS='bedtools'
MOSDEPTH='mosdepth'
TABIX='tabix'
PLATFORM='ont'

ROOT_FOLDER="output"
THREADS=36

The workflow expects these local files by default:

HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz
HG004_GRCh38_1_22_v4.2.1_benchmark.vcf.gz
HG003_GRCh38_1_22_v4.2.1_benchmark.bed
HG004_GRCh38_1_22_v4.2.1_benchmark.bed
HG003.bam
HG004.bam
GRCh38_no_alt.fasta

The BAM files should be position-sorted and indexed.

Run

bash clairs_ss_model_training_workflow.sh

Main steps

  1. Phase truth VCFs and haplotag input BAMs with WhatsHap.
  2. Calculate raw BAM coverage with mosdepth.
  3. Split phased BAMs into chunks.
  4. Mix chunks into synthetic normal/tumor BAMs.
  5. Intersect high-confidence BED regions between paired samples.
  6. Extract normal and tumor candidate sites.
  7. Build SNV and INDEL candidate lists.
  8. Create pileup and full-alignment tensors.
  9. Convert tensors to training bins.
  10. Train four models: SNV pileup, SNV full-alignment, INDEL pileup, and INDEL full-alignment.

Outputs

The SS workflow writes to:

output/

Important outputs:

output/train/pileup_snv/
output/train/full_alignment_snv/
output/train/pileup_indel/
output/train/full_alignment_indel/

After each training run, choose the checkpoint epoch with the lowest validation loss from the corresponding model directory.

II. SSRS Training Workflow

SSRS should be run after SS training or with installed ClairS SS model checkpoints available under the active environment.

Setup variables

Edit the top of clairs_ssrs_model_training_workflow.sh if your tool names or input file names differ:

CLAIRS='clairs.py'
PYTHON='python3'
PYPY='pypy3'
PARALLEL='parallel'
SAMTOOLS='samtools'
BEDTOOLS='bedtools'
PLATFORM='ont'

ROOT_FOLDER="output"
THREADS=36

By default, SSRS initializes from:

${CONDA_PREFIX}/bin/clairs_models/ont_r10_dorado_sup_5khz/pileup.pkl
${CONDA_PREFIX}/bin/clairs_models/ont_r10_dorado_sup_5khz/full_alignment.pkl
${CONDA_PREFIX}/bin/clairs_models/ont_r10_dorado_sup_5khz/indel/pileup.pkl
${CONDA_PREFIX}/bin/clairs_models/ont_r10_dorado_sup_5khz/indel/full_alignment.pkl

The workflow expects these local tumor/normal BAM files by default:

1437_Tumor_ONT.GRCh38.sorted.bam
1937_Tumor.ONT.GRCh38.sorted.bam
1954_Tumor_ONT.GRCh38.sorted.bam
2009_Tumor_ONT.GRCh38.sorted.bam

1437_Normal_ONT.GRCh38.sorted.bam
1937_Normal_ONT.GRCh38.sorted.bam
1954_Normal_ONT.GRCh38.sorted.bam
2009_Normal_ONT.GRCh38.sorted.bam

It also expects these benchmark VCFs and BED files:

H1437_DeepSomatic_multicancer-model.vcf.gz
HCC1937_DeepSomatic_multicancer-model.vcf.gz
HCC1954_DeepSomatic_multicancer-model.vcf.gz
H2009_DeepSomatic_multicancer-model.vcf.gz

H1437_high-confidence-regions_DeepSomatic-multicancer-model.bed
HCC1937_high-confidence-regions_DeepSomatic-multicancer-model.bed
HCC1954_high-confidence-regions_DeepSomatic-multicancer-model.bed
H2009_high-confidence-regions_DeepSomatic-multicancer-model.bed

The reference FASTA is:

GRCh38_no_alt_analysis_set.fasta

Run

bash clairs_ssrs_model_training_workflow.sh

Main steps

  1. Subsample tumor and normal BAMs and create full-depth symbolic links.
  2. Haplotag tumor reads with ClairS.
  3. Extract candidate sites from real tumor/normal benchmark data.
  4. Build SNV and INDEL candidate lists.
  5. Create pileup and full-alignment tensors.
  6. Convert tensors to training bins.
  7. Train four SSRS models initialized from the ClairS SS checkpoints.

Outputs

The SSRS workflow writes to:

output/

Important outputs:

output/train_ssrs/pileup_snv/
output/train_ssrs/full_alignment_snv/
output/train_ssrs/pileup_indel/
output/train_ssrs/full_alignment_indel/