Skip to content

Project Overview Feature Showcase

Jump to bottom
github-actions[bot] edited this page May 26, 2026 · 5 revisions

Feature Showcase

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

ChordMiniApp is an open-source music analysis platform that delivers:

  • Intuitive homepage with YouTube search and recent analyses
  • Beat and chord analysis with grid visualization and segmentation overlays
  • Interactive guitar diagrams with accurate fingering patterns
  • Real-time piano visualizer with falling notes and MIDI export
  • Interactive music theory and practice games in /games and inside extraction wait states
  • Experimental melody transcription powered by Sheet Sage
  • Synchronized lyrics with an AI chatbot assistant

The platform targets musicians, educators, and music enthusiasts who want fast, accurate chord and beat analysis, plus expressive playback and export capabilities.

Project Structure

High-level structure relevant to features:

  • Frontend Next.js app under src/app and src/components
  • Backend Python Flask app under python_backend
  • Experimental services (Sheet Sage) under sheetsage
  • Standalone ML models under SongFormer
graph TB
FE["Next.js Frontend<br/>src/app, src/components"] --> BE["Python Backend<br/>python_backend/app.py"]
FE --> SHEETSAGE["Sheet Sage Service<br/>sheetsage/"]
FE --> FIREBASE["Firebase Storage/DB"]
BE --> MODELS["Beat/Chord Models<br/>Beat-Transformer, Chord-CNN-LSTM"]
FE --> MIDI["MIDI Export Utils<br/>src/utils/midiExport.ts"]
Loading

Diagram sources

Section sources

Core Components

  • Homepage: integrated search, hero demos, recent analyses, and feature highlights
  • Analysis pipeline: beat detection, chord recognition, key/signature detection, lyrics, segmentation
  • Visualization: chord grid, guitar diagrams, piano visualizer with falling notes and sheet music
  • Playback: chord playback, metronome, pitch shift, MIDI export
  • Mini games: theory quiz, ear training, guitar chord reading, and X/O practice with session review
  • Experimental melody transcription: Sheet Sage integration with caching and playback
  • AI assistant: Gemini-powered chatbot with song context

Benefits by user type:

  • Musicians: precise chord grids, interactive guitar diagrams, piano visualizer, MIDI export
  • Educators: synchronized lyrics, Roman numeral overlays, segmentation, chatbot Q&A
  • Enthusiasts: homepage demos, recent analyses, YouTube search, mini games while waiting, Sheet Sage melody overlays

Section sources

Architecture Overview

End-to-end flow from audio to visualization and export.

sequenceDiagram
participant U as "User"
participant FE as "Frontend"
participant API as "Python Backend"
participant DET as "Detectors<br/>Beat/Chord"
participant LYR as "Lyrics Service"
participant SG as "Sheet Sage"
participant FB as "Firebase"
U->>FE : "Upload or select YouTube"
FE->>API : "POST /api/extract-audio"
API->>DET : "Run beat/chord/key detection"
DET-->>API : "Analysis results"
API-->>FE : "Analysis JSON"
FE->>LYR : "Fetch lyrics (Genius/LRCLIB)"
LYR-->>FE : "Lyrics + timing"
FE->>SG : "Optional melody transcription"
SG-->>FE : "Melody note events"
FE->>FB : "Persist results (cache)"
FE-->>U : "Chord grid, piano visualizer, lyrics, chatbot"
U->>FE : "Export MIDI"
FE->>FE : "Generate MIDI from chord events"
Loading

Diagram sources

Detailed Component Analysis

Homepage Interface

  • Integrated search with sticky behavior and recent videos carousel
  • Hero demos showcasing beat/chord grid, piano visualizer, and synchronized lyrics
  • Responsive design with theme-aware backgrounds and animations

User interactions:

  • Enter YouTube URL or search query, select a result to analyze
  • Browse recent analyses to discover songs
  • Toggle tabs to explore features

Technical capabilities:

  • Dynamic imports for heavy components
  • Scroll-based animations and skeleton loaders
  • Theme-aware gradients and decorative elements

Section sources

Interactive Music Theory & Games

  • Standalone /games page with Quiz, Ear, Guitar, and X/O tabs.
  • Embedded wait-state games through ExtractionWaitPanel so users can practice while browser extraction, queueing, or inference is in progress.
  • Quiz generation covers scale degrees, chord labels, missing chord tones, Roman numeral degree questions, and secondary dominant recognition.
  • Ear training plays reference tones, progressions, and chord qualities with piano, guitar, violin, flute, or composite timbres; review rows for ear questions include playback.
  • Session history is stored in browser sessionStorage and records overall score plus each answered question, selected answers, correct answer, and correctness for later review.

Section sources

Beat & Chord Analysis with Grid Visualization

  • Chord progression grid with beat-aligned cells
  • Time signature, key signature, and pickup beats support
  • Segmentation overlays for intro/verse/chorus/bridge/outro
  • Roman numeral analysis and key modulation markers
  • Edit mode for chord corrections with sequence-aware mapping

User interactions:

  • Click cells to jump to beat
  • Toggle Roman numerals and segmentation overlays
  • Enable loop playback between selected beats

Technical capabilities:

  • Memoized rendering to minimize re-renders
  • Accidental preference for consistent spelling
  • Beat-to-chord mapping for accurate audio sync
  • Editable chords with persistent corrections
flowchart TD
Start(["Load Analysis"]) --> Build["Build Measures<br/>and Rows"]
Build --> Overlay{"Show Segmentation?"}
Overlay --> |Yes| Color["Assign Section Colors"]
Overlay --> |No| Style["Apply Styles"]
Color --> Render["Render Cells<br/>with Labels/Modulations"]
Style --> Render
Render --> Interact["Beat Click / Loop Selection"]
Interact --> Playback["Chord Playback Sync"]
Loading

Diagram sources

Section sources

Interactive Guitar Diagrams with Accurate Fingering Patterns

  • Animated chord diagrams with shape/sound labeling modes
  • Capo position control with automatic suggestions
  • Multiple voicings per chord with position selector
  • Guitar-only playback synchronized to chord grid
  • Tablature view for notation

User interactions:

  • Switch between animated, summary, and tab views
  • Adjust capo and voicing position
  • Toggle label mode (shape vs. sound)
  • Play guitar voicings with dynamics and segmentation cues

Technical capabilities:

  • Chord database integration for accurate fingering
  • Transpose chords for capo positions
  • Build chord timelines for playback
  • Segment-aware coloring for structural sections

Section sources

Real-time Piano Visualizer with Falling Notes and MIDI Export

  • Piano roll visualization with falling notes synchronized to playback
  • Scrolling chord strip with chord labels and Roman numerals
  • Sheet music display with notation rendering
  • Multi-instrument mixer with instrument-specific voicings
  • MIDI export supporting multiple tracks and melody overlays

User interactions:

  • Switch between piano roll and sheet music displays
  • Control playback speed and toggle chord playback
  • Export MIDI with instrument-specific tracks
  • Toggle melody overlay from Sheet Sage

Technical capabilities:

  • Timeline model builds chord events and notation
  • Instrument voicings: piano (full), guitar (arpeggiated), violin/flute (root), bass (single)
  • Dynamics-aware velocity shaping for realistic playback
  • Export to standard MIDI Type 1 with tempo/time signature
sequenceDiagram
participant UI as "PianoVisualizerTab"
participant TL as "Timeline Model"
participant SM as "Sheet Music Model"
participant MIX as "Audio Mixer"
participant MIDI as "MIDI Export"
UI->>TL : "Resolve chords + build events"
TL-->>UI : "Merged chord events + notation"
UI->>SM : "Build sheet music model"
UI->>MIX : "Configure instruments + settings"
UI->>MIDI : "Export MIDI (tracks + tempo)"
MIDI-->>UI : "Blob + download"
Loading

Diagram sources

Section sources

Experimental Melody Transcription with Sheet Sage Integration

  • Request melody transcription from Sheet Sage backend or API
  • Offload upload path for large files
  • Cache results and enable playback with visual notes
  • Export melody as an additional MIDI track

User interactions:

  • Enable melodic transcription playback
  • Toggle melody overlay in the piano visualizer
  • Export melody track alongside chord tracks

Technical capabilities:

  • Resolve audio source from uploaded file or proxied URL
  • Use offload upload service when appropriate
  • Build note events for visualization and MIDI export

Section sources

Synchronized Lyrics with AI Chatbot Assistant

  • Enhanced lyrics display with chord positioning above words
  • Auto-scroll to current line with throttled smooth scroll
  • AI chatbot with Gemini integration for contextual Q&A
  • Conversation history truncation and error handling

User interactions:

  • Click lyric lines to seek in playback
  • Ask questions about chords, structure, and lyrics
  • Clear conversation and close chat panel

Technical capabilities:

  • Word-position mapping for chords
  • Time-based highlighting and auto-scroll
  • Truncated conversation history for efficient prompts

Section sources

Dependency Analysis

Frontend-to-backend and service dependencies.

graph LR
FE["Frontend Pages<br/>src/app/analyze/page.tsx"] --> API["Python Backend<br/>python_backend/app.py"]
FE --> SG["Sheet Sage Client<br/>sheetSageTranscriptionClient.ts"]
FE --> MIDI["MIDI Export<br/>midiExport.ts"]
API --> DET["Beat/Chord Detectors"]
API --> LYR["Lyrics Services"]
API --> FB["Firebase"]
Loading

Diagram sources

Section sources

Performance Considerations

  • Frontend
    • Memoized chord grid rendering reduces re-renders to affected cells
    • Dynamic imports defer heavy components until needed
    • Throttled auto-scroll prevents layout thrashing
  • Backend
    • Deferred model availability checks for faster startup
    • Rate limiting and CORS configured for production
  • Experimental features
    • Sheet Sage inference is slower than core pipeline; results cached
    • MIDI export computes velocities from dynamics and segmentation for realism

[No sources needed since this section provides general guidance]

Troubleshooting Guide

  • Backend connectivity
    • Ensure backend runs on port 5001 and responds to health checks
    • Verify environment variables for CORS and Flask limits
  • Model availability
    • Confirm Beat-Transformer and Chord-CNN-LSTM models are available
    • Install system dependencies (FluidSynth) for MIDI synthesis
  • Sheet Sage
    • Check backend availability and network access
    • Use offload upload for large files
  • Firebase
    • Verify storage rules and temp folder cleanup configuration
    • Ensure anonymous auth is enabled for local development

Section sources

Conclusion

ChordMiniApp combines robust music analysis with expressive visualization and playback. Its modular architecture enables seamless integration of beat/chord detection, guitar diagrams, piano visualization, lyrics, and an experimental melody transcription pipeline, all culminating in professional MIDI export. The AI chatbot enhances discovery and learning by providing contextual insights grounded in the analysis.

[No sources needed since this section summarizes without analyzing specific files]

Appendices

Typical Workflows

  • Musician workflow
    • Upload audio or select YouTube
    • Review chord grid and segmentation
    • Export MIDI for DAW integration
    • Practice with guitar diagrams and piano visualizer
  • Educator workflow
    • Analyze a song, review Roman numerals and modulations
    • Show synchronized lyrics and structure
    • Use chatbot to explain harmonic concepts
  • Enthusiast workflow
    • Browse recent analyses
    • Try Sheet Sage melody overlay
    • Share results and export MIDI

[No sources needed since this section provides general guidance]

ChordMiniApp Wiki

General

API Reference

Architecture and Design

Audio Processing and Analysis

Backend Services

Database and Storage

Deployment and Operations

Experimental Features

Frontend Application

Lyrics and Text Processing

Machine Learning Models

Project Overview

Visualization and User Interface

Clone this wiki locally