Skip to content

Backend Services Machine Learning Services Machine Learning Services

Jump to bottom
github-actions[bot] edited this page May 2, 2026 · 4 revisions

Machine Learning Services

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document explains the machine learning services powering ChordMiniApp’s audio analysis pipeline. It covers beat detection, chord recognition, and audio processing capabilities, detailing detector implementations (Beat-Transformer, madmom, Librosa, Chord-CNN-LSTM), model management, orchestration, integration with external libraries (Spleeter, librosa), and performance optimization strategies. It also documents model configuration, training data requirements, evaluation metrics, and troubleshooting guidance for common ML issues.

Project Structure

The ML services are organized under a clear separation of concerns:

  • Orchestrators: BeatDetectionService and ChordRecognitionService manage detector selection, file-size-aware routing, and fallback strategies.
  • Detectors: Individual services wrap external libraries and models with a normalized interface.
  • Audio Utilities: Provide audio validation, silence trimming, duration estimation, and resampling.
  • Model Management: Availability checks, device info, and model directory introspection.
  • Configuration: YAML-based model configuration for BTC models; settings for Chord-CNN-LSTM.
graph TB
subgraph "Orchestrators"
BDS["BeatDetectionService"]
CRS["ChordRecognitionService"]
end
subgraph "Detectors"
BTD["BeatTransformerDetectorService"]
MDS["MadmomDetectorService"]
LSD["LibrosaDetectorService"]
CCLDS["ChordCNNLSTMDetectorService"]
end
subgraph "Audio & Model Utils"
SU["SpleeterService"]
AU["audio_utils"]
MU["model_utils"]
CM["chord_mappings"]
end
subgraph "Configs"
BTC_CFG["btc_config.yaml"]
CCL_SETTINGS["Chord-CNN-LSTM/settings.py"]
end
BDS --> BTD
BDS --> MDS
BDS --> LSD
CRS --> CCLDS
CRS --> SU
CRS --> CM
MU --> BDS
MU --> CRS
BTC_CFG --> CRS
CCL_SETTINGS --> CRS
Loading

Diagram sources

Section sources

Core Components

  • BeatDetectionService: Central orchestrator for beat detection with automatic detector selection, file-size-aware routing, and fallback strategies. It supports Beat-Transformer, madmom, and Librosa detectors.
  • ChordRecognitionService: Central orchestrator for chord recognition with detector selection, chord dictionary validation, optional Spleeter-based vocal separation, and metadata enrichment.
  • Detector Wrappers: Normalized interfaces for Beat-Transformer, madmom, Librosa, and Chord-CNN-LSTM, returning consistent result schemas.
  • Audio Utilities: Validation, silence trimming, duration estimation, resampling, and format handling.
  • Model Management: Availability checks, device info, and model directory introspection for Spleeter, Beat-Transformer, Chord-CNN-LSTM, BTC, and deep learning frameworks.
  • Chord Dictionary Management: Centralized mapping of chord vocabularies and normalization utilities.

Section sources

Architecture Overview

The ML services follow a layered architecture:

  • Orchestrators accept requests, validate inputs, and select appropriate detectors based on availability and file size.
  • Detectors encapsulate model-specific logic and return normalized results.
  • Audio utilities provide pre/post-processing steps.
  • Model management utilities provide runtime availability and device information.
  • Configuration files define model hyperparameters and paths.
sequenceDiagram
participant Client as "Client"
participant CRS as "ChordRecognitionService"
participant Detector as "ChordCNNLSTMDetectorService"
participant Spleeter as "SpleeterService"
Client->>CRS : "recognize_chords(file_path, detector, chord_dict, use_spleeter)"
CRS->>CRS : "validate_audio_file()"
alt use_spleeter and available
CRS->>Spleeter : "extract_vocals(file_path)"
Spleeter-->>CRS : "vocals_path or error"
end
CRS->>Detector : "recognize_chords(vocals_path or original)"
Detector-->>CRS : "normalized chord results"
CRS-->>Client : "structured analysis results"
Loading

Diagram sources

Detailed Component Analysis

Beat Detection Service

  • Responsibilities:
    • Detector selection among Beat-Transformer, madmom, and Librosa.
    • File-size-aware routing with configurable limits.
    • Fallback strategies when requested detector is unavailable or file exceeds limits.
    • Metadata enrichment (file size, detector used, processing time, duration).
  • Selection logic:
    • Preference order varies by file size; favors madmom for small files, Beat-Transformer for medium, and madmom/librosa for large files.
    • Enforces size limits per detector; falls back to alternatives if needed.
  • Output normalization:
    • Consistent keys: success flag, beats, downbeats, total counts, BPM, time signature, duration, model identifiers, processing time.
flowchart TD
Start(["detect_beats(file_path, detector, force)"]) --> Validate["Validate audio file"]
Validate --> Size["Compute file size (MB)"]
Size --> Select["select_detector(requested, size, force)"]
Select --> Avail{"Available?"}
Avail --> |No| Fallback["Fallback to best alternative"]
Avail --> |Yes| UseDet["Use requested detector"]
Fallback --> UseDet
UseDet --> Run["detector.detect_beats(file_path)"]
Run --> Meta["Add metadata (size, duration, timing)"]
Meta --> Log["Log results and diagnostics"]
Log --> End(["Return normalized result"])
Loading

Diagram sources

Section sources

Chord Recognition Service

  • Responsibilities:
    • Detector selection among Chord-CNN-LSTM, BTC-SL, and BTC-PL.
    • Chord dictionary validation and default selection per model.
    • Optional Spleeter-based vocal separation to improve recognition accuracy.
    • Metadata enrichment and cleanup of temporary files.
  • Spleeter integration:
    • Uses 2-stem separation by default; cleans up generated stems after inference.
  • Output normalization:
    • Consistent keys: success flag, chord events with start/end times, total count, duration, model identifiers, processing time, and chord dictionary used.
sequenceDiagram
participant Client as "Client"
participant CRS as "ChordRecognitionService"
participant Detector as "ChordCNNLSTMDetectorService"
participant Spleeter as "SpleeterService"
Client->>CRS : "recognize_chords(file_path, detector, chord_dict, use_spleeter)"
CRS->>CRS : "validate_audio_file()"
alt use_spleeter and available
CRS->>Spleeter : "extract_vocals(file_path)"
Spleeter-->>CRS : "vocals_path or error"
end
CRS->>Detector : "recognize_chords(vocals_path or original, chord_dict)"
Detector-->>CRS : "normalized chord results"
CRS->>CRS : "cleanup_stems() if used"
CRS-->>Client : "structured analysis results"
Loading

Diagram sources

Section sources

Detector Implementations

Beat-Transformer Detector

  • Provides a normalized interface to the BeatTransformerDetector.
  • Availability checked via model import and checkpoint presence.
  • Returns normalized beat and downbeat arrays, BPM, time signature, duration, and processing time.

Section sources

Madmom Detector

  • Implements RNN beat activation and DBN beat tracking.
  • Heuristic downbeat candidates for 3/4 and 4/4 meters.
  • Returns normalized results with candidate downbeats and metadata.

Section sources

Librosa Detector

  • Uses librosa beat tracking with tempo estimation and simple downbeat heuristic.
  • Returns normalized results with BPM, time signature placeholder, and duration.

Section sources

Chord-CNN-LSTM Detector

  • Dynamically imports model code from the model directory.
  • Generates a temporary LAB file and parses labeled chord segments.
  • Supports multiple chord dictionaries; returns normalized chord events with durations.

Section sources

Model Management and Configuration

  • Availability checks:
    • Spleeter, Beat-Transformer, Chord-CNN-LSTM, Genius, BTC, PyTorch, and TensorFlow availability are checked independently.
    • BTC availability includes model file existence and required dependencies (PyTorch, NumPy).
  • Device information:
    • PyTorch availability and device selection (CUDA/MPS/CPU) with device names and counts.
  • Model directory introspection:
    • Lists files, subdirectories, and sizes for model directories.
  • BTC configuration:
    • Feature extraction parameters, model hyperparameters, and model paths.
  • Chord-CNN-LSTM settings:
    • Sample rates, hop lengths, and default chord dictionary.

Section sources

Audio Processing Pipeline

End-to-end pipeline from raw audio to structured analysis:

  1. Input validation and optional silence trimming.
  2. Detector selection based on file size and availability.
  3. Model inference with normalized output.
  4. Metadata enrichment (duration, processing time, detector info).
  5. Optional Spleeter separation for chord recognition.
  6. Cleanup of temporary files and stems.
flowchart TD
A["Raw Audio File"] --> B["Validate Audio"]
B --> C{"Detector Selection"}
C --> |Beat Detection| D["Run Beat Detector"]
C --> |Chord Recognition| E["Run Chord Detector"]
D --> F["Normalize Results"]
E --> G{"Use Spleeter?"}
G --> |Yes| H["Extract Vocals"]
G --> |No| I["Use Original Audio"]
H --> J["Run Chord Detector on Vocals"]
I --> J
J --> K["Parse LAB/Annotations"]
K --> L["Normalize Results"]
F --> M["Add Metadata"]
L --> M
M --> N["Return Structured Output"]
Loading

Diagram sources

Section sources

Dependency Analysis

External libraries and integrations:

  • librosa: Core audio loading, duration, and beat tracking.
  • Spleeter: Audio separation into stems (vocals/accompaniment/instruments).
  • madmom: Neural beat detection with RNN and DBN processing.
  • PyTorch/TensorFlow: Deep learning frameworks for BTC and Chord-CNN-LSTM.
  • Pretty MIDI, mir_eval, mido: Music analysis and evaluation utilities.
  • Requests, httpx: HTTP clients for external services.
graph TB
App["Flask App (app.py)"] --> BDSvc["BeatDetectionService"]
App --> CRSvc["ChordRecognitionService"]
App --> MU["model_utils.py"]
BDSvc --> BTDet["BeatTransformerDetectorService"]
BDSvc --> MadDet["MadmomDetectorService"]
BDSvc --> LibDet["LibrosaDetectorService"]
CRSvc --> CCLDet["ChordCNNLSTMDetectorService"]
CRSvc --> Svc["SpleeterService"]
CRSvc --> CM["chord_mappings.py"]
MU --> Req["requirements.txt"]
BTDet --> BTCfg["btc_config.yaml"]
CCLDet --> CCLSet["Chord-CNN-LSTM/settings.py"]
Loading

Diagram sources

Section sources

Performance Considerations

  • Detector selection by file size:
    • Small files: madmom preferred for speed.
    • Medium files: madmom or Beat-Transformer.
    • Large files: madmom or Librosa to reduce memory footprint.
  • Spleeter separation:
    • Optional; improves chord recognition accuracy but adds overhead; cleaned up after use.
  • Memory management:
    • Spleeter creates a new separator per request to avoid memory leaks.
    • Temporary files are removed after processing.
  • Batch processing:
    • Current orchestrators operate on single files; batch processing can be introduced at the API layer by iterating over lists and parallelizing with worker pools.
  • GPU acceleration:
    • PyTorch availability and device selection are checked; CUDA/MPS devices are leveraged when available.
  • Logging and diagnostics:
    • Extensive logging for processing times, detector usage, and beat-per-measure distributions to aid tuning.

[No sources needed since this section provides general guidance]

Troubleshooting Guide

Common issues and resolutions:

  • Beat-Transformer not available:
    • Ensure model import succeeds and checkpoint path is valid; check availability via model_utils.
  • madmom import failures:
    • pkg_resources compatibility issue; pin setuptools to a compatible version as indicated in logs.
  • Librosa detection errors:
    • Verify librosa installation and audio file accessibility; fallback to validation checks.
  • Chord-CNN-LSTM recognition failures:
    • Confirm model directory and required files exist; temporarily mock data is used for response format validation.
  • Spleeter separation errors:
    • Validate model availability and output directory permissions; ensure temporary directories are cleaned up on failure.
  • Model configuration mismatches:
    • Verify btc_config.yaml and Chord-CNN-LSTM settings align with training data and evaluation metrics.

Section sources

Conclusion

The ChordMiniApp ML services provide a robust, modular framework for beat detection and chord recognition. Through detector orchestration, model management, and integration with librosa, Spleeter, and deep learning frameworks, the system balances accuracy, performance, and reliability. The normalized interfaces and comprehensive availability checks ensure graceful fallbacks and predictable behavior across diverse environments.

[No sources needed since this section summarizes without analyzing specific files]

Appendices

Model Configuration and Training Data

  • BTC configuration:
    • Feature extraction parameters, model hyperparameters, and model paths are defined in btc_config.yaml.
  • Chord-CNN-LSTM settings:
    • Sample rates, hop lengths, and default chord dictionary are defined in settings.py.
  • Training data requirements:
    • BTC training is orchestrated via Kubernetes Jobs with dataset symlinks and environment-controlled hyperparameters.
  • Evaluation metrics:
    • mir_eval and related libraries are available for evaluation; integration points can be added at the API layer.

Section sources

ChordMiniApp Wiki

General

API Reference

Architecture and Design

Audio Processing and Analysis

Backend Services

Database and Storage

Deployment and Operations

Experimental Features

Frontend Application

Lyrics and Text Processing

Machine Learning Models

Project Overview

Visualization and User Interface

Clone this wiki locally