AdapTIGER: Adaptive Spectral Feature Compression for Efficient Speech Separation

AdapTIGER extends TIGER (ICLR 2025) by replacing its fixed band-split encoder/decoder with a lightweight Spectral Feature Compression (SFC) cross-attention mechanism, enabling input-adaptive spectral grouping while reducing parameters by 15%.

Based on:

• TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction (Xu et al., ICLR 2025)
• SFC: Spectral Feature Compression - input-adaptive frequency band learning

---

Overview

TIGER achieves competitive speech separation with only 822K parameters using hand-crafted frequency band splits (67 fixed sub-bands). However, these fixed boundaries cannot adapt to input characteristics - different speaker pitches, noise profiles, or acoustic conditions would benefit from different spectral groupings.

AdapTIGER replaces TIGER's 134 independent per-band modules (67 encoder + 67 decoder) with a single SFC encoder-decoder pair based on cross-attention. This:

• Reduces parameters from 822K to 696K (15% reduction)
• Enables input-adaptive spectral grouping - attention weights vary per input, allowing the model to dynamically adjust how frequency bins are grouped into bands
• Discovers harmonic structure - mid-frequency bands learn to adaptively attend to the fundamental frequency (F0) region based on speaker content

What's Changed vs Original TIGER

New files (our contribution)

File	Description
look2hear/models/sfc.py	SFC encoder/decoder - cross-attention replacing fixed band-split modules
look2hear/models/sigreg.py	SIGReg anti-collapse regularizer
graphs/plot_sfc_interpretability.py	SFC attention analysis and visualization
analysis_outputs/sfc_interpretability/	Generated attention heatmaps, adaptivity plots
paper_figures/	Publication-ready figures
results/table_test.csv, results/table_validation.csv	Aggregated experiment results
train-student.slurm	SLURM script for studentkillable partition
run_eval_after_train.py	Helper called by audio_train.py for post-training eval
configs/tiger-sfc-*.yml	All SFC experiment configs

Modified files

File	What changed
look2hear/models/tiger.py	Added use_sfc flag - when enabled, replaces 67 per-band encoder/decoder modules with SFC
look2hear/system/audio_litmodule.py	Added SFC regularization loss forwarding, W&B logging of SFC metrics
look2hear/metrics/wrapper.py	Extended with additional evaluation metrics
audio_train.py	Added auto-resubmit on SLURM wall-time, gradient checkpointing, post-training eval
audio_test.py	Added efficiency benchmarking (MACs, inference time, memory), CSV result logging
train.slurm	Adapted for our cluster (8 GPUs, local /tmp venv fallback, NCCL P2P disabled)
eval.slurm	Evaluation SLURM script
configs/tiger-baseline-3s.yml	Baseline config tuned for our setup (3s segments, 100 epochs)

Unchanged from original TIGER

• look2hear/models/tiger.py - core separator architecture (FFI blocks, F3A, MSA modules) is untouched
• look2hear/losses/ - PIT wrapper and loss functions (neg SI-SDR, neg SNR)
• look2hear/datas/ - EchoSet and Libri2Mix data modules
• inference_speech.py, inference_dnr.py - inference scripts
• download_echoset.py - dataset download
• assets/ - original TIGER figures and logo

Key Contributions

1. SFC Encoder/Decoder: Replaces 134 fixed per-band modules with a single cross-attention mechanism. Learnable band queries (K=67) attend to frequency bins (F=321), producing soft, input-dependent spectral groupings.

2. G1 Gated Attention: Per-head sigmoid gate applied after scaled dot-product attention, breaking the low-rank bottleneck (rank <= d_head) inherent in standard multi-head attention.

3. SVD Query Initialization: Band queries initialized deterministically from the top singular vectors of the positional bias matrix - eliminates 3dB seed sensitivity observed with random initialization.

4. Four-Part Anti-Collapse Regularization (AdapTIGER-Reg variant):

   • Cosine-similarity log-barrier (prevents band collapse)
   • Smoothness penalty (prevents sharp jumps in similarity matrix)
   • SIGReg loss (prevents dimensionality collapse)
   • Adaptivity reward (encourages input-dependent attention patterns)

Architecture

AdapTIGER preserves TIGER's core separator (8 FFI blocks with F3A and MSA modules) and modifies only the encoder/decoder:

Input Mixture
    |
   STFT (win=640, hop=160) --> 321 freq bins x T frames
    |
  [SFC Encoder]  <-- NEW: cross-attention (67 band queries attend to 321 freq keys)
    |               with G1 gating, positional bias, SwiGLU FFN
    |
  Separator (8 FFI blocks, unchanged from TIGER)
    |
  [SFC Decoder]  <-- NEW: cross-attention (321 freq queries attend to 67 band keys)
    |
  Complex Mask x iSTFT
    |
  Separated Sources (x2)

Results

Performance on EchoSet (two-speaker separation):

Model	Params	Val SI-SDRi (dB)	Test SI-SDRi (dB)	Adaptivity	Eff. Rank	Band Cosine Sim
Baseline TIGER	822K	13.67	13.14	0	56.3/67	0.15
AdapTIGER-clean	696K	13.31	11.45	0.001	2.5/67	0.71
AdapTIGER-reg	696K	12.69	10.79	0.002	20.4/67	0.08

• AdapTIGER-clean: No regularization. Learns input-adaptive attention but suffers band-feature collapse (all bands converge to similar representations).
• AdapTIGER-reg: Trades 0.66dB for structured band diversity via four-part regularization.

Installation

git clone <this-repo>
cd TIGER-plus
pip install -r requirements.txt

Usage

Training

On a SLURM cluster (recommended):

sbatch train.slurm

Or directly with PyTorch Lightning:

python audio_train.py --conf_dir configs/tiger-sfc-clean-v2.yml

Available configurations:

• tiger-baseline-3s.yml - Baseline TIGER (no SFC)
• tiger-sfc-clean-v2.yml - AdapTIGER-clean (SFC, no regularization)
• tiger-sfc-v11-3s.yml - AdapTIGER with SIGReg regularization

Evaluation

python audio_test.py --conf_dir configs/tiger-sfc-clean-v2.yml

Results are appended to results/table_test.csv and results/table_validation.csv.

Inference

python inference_speech.py --audio_path path/to/mixture.wav

Dataset

EchoSet (20,268 train / 4,604 val / 2,650 test samples at 16kHz):

sbatch download.slurm
# or
python download_echoset.py

Project Structure

audio_train.py                    # Training entry point (Lightning CLI)
audio_test.py                     # Evaluation entry point
inference_speech.py               # Single-file inference

configs/
  tiger-baseline-3s.yml           # Baseline TIGER config
  tiger-sfc-clean-v2.yml          # AdapTIGER-clean config
  tiger-sfc-v11-3s.yml            # AdapTIGER with regularization

look2hear/
  models/
    tiger.py                      # TIGER model (use_sfc flag toggles SFC)
    sfc.py                        # SFC encoder/decoder (cross-attention)
    sigreg.py                     # SIGReg anti-collapse regularizer
  system/
    audio_litmodule.py            # PyTorch Lightning training module
  losses/
    matrix.py                     # Loss functions (neg SI-SDR, neg SNR)
    pit_wrapper.py                # Permutation Invariant Training
  datas/
    echosetdatamodule.py          # EchoSet data module
  metrics/
    wrapper.py                    # Evaluation metrics

results/
  table_test.csv                  # Test set results across experiments
  table_validation.csv            # Validation results across experiments

graphs/
  plot_sfc_interpretability.py    # SFC attention visualization

analysis_outputs/
  sfc_interpretability/           # Attention heatmaps, adaptivity plots

paper_figures/                    # Publication-ready figures

train.slurm                       # SLURM training script (8 GPUs, DDP)
eval.slurm                        # SLURM evaluation script
download.slurm                    # SLURM dataset download script

Experiments/                      # Training outputs (checkpoints, logs)
DataPreProcess/EchoSet/           # Downloaded dataset

Citation

@article{xu2024tiger,
  title={TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation},
  author={Xu, Mohan and Li, Kai and Chen, Guo and Hu, Xiaolin},
  journal={arXiv preprint arXiv:2410.01469},
  year={2024}
}

Acknowledgments

This project builds on TIGER by Mohan Xu, Kai Li, Guo Chen, and Xiaolin Hu (Tsinghua University), and incorporates ideas from the Spectral Feature Compression framework.