-
-
Notifications
You must be signed in to change notification settings - Fork 48
Audio Processing and Analysis Audio Playback System
Changes Made
- Updated synchronization architecture to reflect the new unified master clock system with enhanced scrub detection
- Added comprehensive documentation for scrub detection thresholds and seek operation improvements
- Enhanced pitch shift service documentation with new feedback loop prevention mechanisms
- Updated synchronization mechanisms with 0.75-second scrub jump threshold
- Added documentation for comprehensive debugging framework for pitch-sync issues
- Introduction
- System Architecture
- Core Components
- Audio Context Management
- Pitch-Shifted Audio Playback
- Chord Playback Services
- Metronome Integration
- Audio Mixing and Volume Control
- Unified Master Clock Architecture
- Synchronization and Timing
- Playback State Management
- Comprehensive Debugging Framework
- Mobile Optimization
- Troubleshooting Guide
- Best Practices
The ChordMiniApp audio playback system is a sophisticated Web Audio API-based solution designed for synchronized musical analysis and playback. The system integrates multiple audio sources including YouTube video playback, pitch-shifted audio from Firebase Storage, synthesized chord playback, and metronome clicks. It provides precise timing control, seamless synchronization between visual and audio elements, and comprehensive volume management across all audio components.
Updated The system now features a revolutionary unified master clock architecture that replaces the previous fragmented approach with three independent time sources. This new architecture ensures perfect synchronization through a single source of truth with slave re-anchor functionality and enhanced scrub detection mechanisms.
The audio playback system follows a modular architecture with clear separation of concerns, now centered around the unified master clock:
graph TB
subgraph "Presentation Layer"
UI[React Components]
Hooks[Custom Hooks]
Debug[Debug Utilities]
EndSeekFence[User Seek Fence]
ScrubDetection[Scrub Detection]
end
subgraph "Control Layer"
Store[Playback Store]
MasterClock[Youtube Master Clock]
Mixer[Audio Mixer]
end
subgraph "Audio Services"
ContextMgr[Audio Context Manager]
PitchShift[GrainPlayer Pitch Shift]
Soundfont[Soundfont Chord Playback]
LightWeight[Lightweight Chord Playback]
Metronome[Metronome Service]
end
subgraph "External Dependencies"
YouTube[YouTube Player API]
Tone[Tone.js Library]
WebAudio[Web Audio API]
Firebase[Firebase Storage]
end
UI --> Hooks
Hooks --> Store
Store --> MasterClock
Store --> Mixer
MasterClock --> PitchShift
MasterClock --> EndSeekFence
MasterClock --> ScrubDetection
Store --> EndSeekFence
Store --> ScrubDetection
MasterClock --> YouTube
ContextMgr --> PitchShift
ContextMgr --> Soundfont
ContextMgr --> Metronome
PitchShift --> Tone
Soundfont --> WebAudio
LightWeight --> WebAudio
Metronome --> WebAudio
YouTube --> MasterClock
Firebase --> PitchShift
Debug --> MasterClock
Debug --> PitchShift
Diagram sources
The AudioContextManager serves as the central hub for Web Audio API lifecycle management. It provides singleton access to the shared AudioContext, handles browser autoplay policy compliance, and manages context suspension/resume cycles.
classDiagram
class AudioContextManager {
-AudioContext _ctx
-boolean _isInitializing
+getContext() AudioContext
+resume() Promise<void>
+suspend() Promise<void>
+close() Promise<void>
+getCurrentTime() number
-attachAutoResumeListeners() void
-_autoResumeHandler() void
}
class Singleton {
<<pattern>>
-AudioContextManager _instance
+instance AudioContextManager
}
AudioContextManager --> Singleton : "implements"
Diagram sources
Section sources
The GrainPlayerPitchShiftService leverages Tone.js for advanced pitch-shifting capabilities using granular synthesis. This service provides independent pitch control and playback rate manipulation without affecting the other.
Updated The service now operates as a pure slave to the unified master clock, eliminating the previous independent time source conflicts and featuring enhanced seek operations with scrub detection.
classDiagram
class GrainPlayerPitchShiftService {
-GrainPlayer grainPlayer
-GainNode gainNode
-Filter lowPassFilter
-Limiter limiter
-boolean _isPlaying
-number _currentTime
-number _duration
-number _playbackRate
-private _playAnchorWallTime number
-private _playAnchorPosition number
-private pendingSeekTime number
+initialize() Promise<void>
+loadAudio(url, semitones, initialRate) Promise<void>
+setPitch(semitones) void
+setVolume(volume) void
+play() void
+pause() void
+seek(time) void
+syncAnchor(positionSec, wallSec) void
+setPlaybackRate(rate) void
+getState() PitchShiftPlaybackState
+getCurrentTimeLive() number
+dispose() void
}
class SignalChain {
<<composition>>
GrainPlayerPitchShiftService --> GainNode
GrainPlayerPitchShiftService --> Filter
GrainPlayerPitchShiftService --> Limiter
}
Diagram sources
Section sources
The system provides two complementary chord playback approaches:
The SoundfontChordPlaybackService delivers realistic instrument sounds using high-quality samples from various musical instruments including piano, guitar, violin, flute, saxophone, and bass.
The LightweightChordPlaybackService generates synthetic chord sounds using Web Audio API oscillators, providing efficient playback without external dependencies.
Section sources
The AudioContextManager implements a robust singleton pattern that ensures proper Web Audio API lifecycle management across the entire application. It addresses critical browser autoplay policies by automatically resuming the audio context on the first user interaction.
- Lazy Initialization: Creates the AudioContext only when first accessed
- Browser Compatibility: Falls back to webkitAudioContext for Safari support
- Autoplay Policy Compliance: Automatically resumes context on user interaction
- State Management: Tracks context state (running, suspended, closed, interrupted)
- Memory Management: Properly closes and recreates contexts when needed
flowchart TD
Start([Application Start]) --> CheckContext{"AudioContext Exists?"}
CheckContext --> |No| CreateContext["Create New AudioContext"]
CheckContext --> |Yes| CheckState["Check Context State"]
CreateContext --> AttachListeners["Attach Auto-Resume Listeners"]
AttachListeners --> Ready[Ready]
CheckState --> |Closed| Recreate["Recreate Context"]
CheckState --> |Running| Ready
CheckState --> |Suspended| Resume["Resume Context"]
CheckState --> |Interrupted| Recreate
Recreate --> AttachListeners
Resume --> Ready
Diagram sources
Section sources
The pitch-shifted audio system provides seamless audio replacement for YouTube playback with configurable pitch shifting and independent volume control.
Updated The system now uses the unified master clock architecture with comprehensive slave re-anchor functionality and enhanced scrub detection mechanisms.
sequenceDiagram
participant UI as User Interface
participant Hook as usePitchShiftAudio
participant Service as GrainPlayerPitchShiftService
participant Master as YoutubeMasterClock
participant Store as PlaybackStore
participant YouTube as YouTube Player
UI->>Hook : Toggle Pitch Shift
Hook->>Service : initializePitchShift()
Service->>Service : loadAudio(firebaseAudioUrl)
Service->>Service : setPitch(semitones)
Service->>Service : setPlaybackRate(rate)
Hook->>Master : setReAnchorListener()
Master->>Service : syncAnchor(position, wall)
UI->>Hook : User Seek
Hook->>Service : seek(time)
Service->>YouTube : seekTo(time) [when pitch shift disabled]
UI->>Hook : Change Rate
Hook->>Store : setPlayerPlaybackRate(rate)
Store->>Master : onRateChange(rate)
Master->>Service : syncAnchor() + setPlaybackRate(rate)
Diagram sources
The system uses Tone.js GrainPlayer for pitch-shifting, which employs granular synthesis to achieve independent pitch and tempo control. The implementation includes:
- Independent Parameter Control: Pitch shifts don't affect playback speed and vice versa
- Adaptive Filtering: Dynamic low-pass filtering to prevent aliasing during upward pitch shifts
- Smooth Transitions: Proper ramping for volume and filter changes
- Buffer Management: Efficient loading and caching of audio buffers
- Slave Re-Anchor Loop: 40ms interval re-synchronization with master clock
- Scrub Detection: 0.75-second threshold for detecting scrub operations
- Feedback Loop Prevention: Mechanisms to prevent seek feedback loops
Section sources
The chord playback system offers two distinct approaches for generating musical chord sounds.
The SoundfontChordPlaybackService provides professional-grade instrument sounds through high-quality sampled instruments:
classDiagram
class SoundfontChordPlaybackService {
-AudioContext audioContext
-boolean isInitialized
-Map~InstrumentName, ActiveScheduledNote~ activeNotes
-SoundfontInstrumentRegistry instrumentRegistry
+playChord(chordName, duration, bpm) Promise<void>
+playChordInstrument(instrument, chord, duration, bpm) Promise<void>
+updateOptions(options) void
+prepareForPlayback() Promise<boolean>
+softStopInstruments(instruments) void
+stopInstruments(instruments) void
}
class InstrumentRegistry {
-Map~InstrumentName, Instrument~ loadedInstruments
+ensureLoaded(instrument) Promise<void>
+getInstrument(instrument) Instrument
+hasLoadedInstrument(instrument) boolean
+scheduleUnload(instrument) void
+reset() void
}
SoundfontChordPlaybackService --> InstrumentRegistry : "manages"
Diagram sources
The LightweightChordPlaybackService generates synthetic chord sounds using Web Audio API oscillators, providing efficient playback without external dependencies:
- Synthetic Generation: Creates chord sounds using oscillator combinations
- Realistic Timbre: Implements proper instrument characteristics for piano and guitar
- Dynamic Envelopes: Applies realistic attack, decay, sustain, and release curves
- Efficient Memory Usage: Minimal resource consumption for simpler scenarios
Section sources
The MetronomeService provides precise timing control for musical synchronization, supporting multiple sound styles and drum patterns.
classDiagram
class MetronomeService {
-AudioContext audioContext
-boolean isInitialized
-boolean isEnabled
-number volume
-string soundStyle
-string trackMode
-GainNode metronomeMasterGain
-Map~SoundStyle, BufferPair~ audioBuffers
+scheduleClick(relativeTime, isDownbeat) void
+setEnabled(enabled) Promise<void>
+toggleMetronome() Promise<boolean>
+clearScheduledClicks() void
+setVolume(volume) void
+setSoundStyle(style) Promise<void>
+setTrackMode(mode) Promise<void>
+testClick(isDownbeat) Promise<void>
}
class DrumRenderer {
+renderKick(context, time, volume, bpm, destinationNode) void
+renderSnare(context, time, volume, bpm, destinationNode) void
+renderHiHat(context, time, volume, bpm, destinationNode) void
}
MetronomeService --> DrumRenderer : "uses"
Diagram sources
Section sources
The AudioMixerService provides centralized volume management for all audio components in the application, ensuring consistent audio levels across different sources and instruments.
graph LR
subgraph "Master Controls"
MasterVolume[Master Volume 0-100]
YouTubeVolume[YouTube Volume 0-100]
PitchShiftVolume[Pitch-Shifted Audio 0-100]
MetronomeVolume[Metronome Volume 0-100]
end
subgraph "Chord Playback Controls"
ChordBus[Chord Playback Bus 0-100]
PianoVolume[Piano Volume 0-100]
GuitarVolume[Guitar Volume 0-100]
ViolinVolume[Violin Volume 0-100]
FluteVolume[Flute Volume 0-100]
SaxVolume[Saxophone Volume 0-100]
BassVolume[Bass Volume 0-100]
end
subgraph "Effective Volume Calculation"
MasterCalc[Master Volume Effect]
ChordCalc[Chord Bus Effect]
InstrumentCalc[Individual Instrument Effect]
end
MasterVolume --> MasterCalc
YouTubeVolume --> MasterCalc
PitchShiftVolume --> MasterCalc
MetronomeVolume --> MasterCalc
ChordBus --> ChordCalc
PianoVolume --> InstrumentCalc
GuitarVolume --> InstrumentCalc
ViolinVolume --> InstrumentCalc
FluteVolume --> InstrumentCalc
SaxVolume --> InstrumentCalc
BassVolume --> InstrumentCalc
MasterCalc --> ChordCalc
ChordCalc --> InstrumentCalc
Diagram sources
The system uses carefully calibrated default volumes to ensure optimal listening experience:
- Master Volume: 80% (overall application volume)
- YouTube Volume: 70% (video audio level)
- Pitch-Shifted Audio: 20% (balanced with video)
- Chord Playback: 70% (primary musical content)
- Individual Instruments: 45-50% (balanced instrument mix)
Section sources
New Section The ChordMiniApp now features a revolutionary unified master clock architecture that replaces the previous fragmented approach with three independent time sources.
The previous system had three competing time sources that fought each other:
- YouTube iframe's internal clock (via onProgress/getCurrentTime)
- GrainPlayer's 50ms setInterval counter
- A drift-correction loop that sought one from the other
At non-1× rates this created a positive feedback loop leading to "keeps refreshing like crazy" and "beat click jumps to wrong position at 2×".
The new architecture establishes a clear hierarchy of clock authority:
- YouTube iframe is the permanent MASTER of {position, rate}
- GrainPlayer becomes a pure SLAVE that follows the master
- Beat Grid Animation reads from master clock for timing accuracy
- Visual Elements synchronized through the master clock position
stateDiagram-v2
[*] --> Initializing
Initializing --> Running : Context Ready
Running --> Paused : Pause Event
Paused --> Running : Play Event
Running --> Seeking : User Seek
Seeking --> Running : Seek Complete
Running --> RateChange : Speed Change
RateChange --> Running : Rate Applied
state Running {
[*] --> Anchored
Anchored --> DriftCorrection : Position Drift Detected
DriftCorrection --> Anchored : Correction Complete
}
state Paused {
[*] --> Frozen
}
state Seeking {
[*] --> FenceActive
FenceActive --> [*] : Fence Expired
}
state RateChange {
[*] --> CounterSnap
CounterSnap --> [*] : Rate Applied
}
Diagram sources
- Single Source of Truth: YouTube iframe provides the authoritative position and rate
- Slave Re-Anchor Loop: 40ms interval re-synchronization prevents drift accumulation
- Counter-Snap on Rate Changes: Prevents position jumps during speed changes
- User-Seek Fence: Blocks stale progress updates after user-initiated seeks
- Hysteresis Threshold: Prevents excessive re-anchoring due to measurement noise
- Scrub Detection Threshold: 0.75-second threshold for detecting scrub operations
Section sources
Updated The synchronization system now operates through the unified master clock architecture with comprehensive slave re-anchor functionality and enhanced scrub detection mechanisms.
The system establishes a clear hierarchy of clock authority:
- YouTube Master Clock: Primary source for position and rate information
- Pitch-Shifted Audio Slave: Secondary clock that follows the master
- Beat Grid Animation: Reads from master clock for timing accuracy
- Visual Elements: Synchronized through the master clock position
The system implements intelligent drift correction to maintain synchronization:
- Hysteresis Threshold: Prevents excessive re-anchoring due to measurement noise
- Rate-Scaled Tolerance: Adjusts tolerance based on current playback rate
- Counter-Snap on Rate Changes: Prevents position jumps during speed changes
- User-Seek Fence: Blocks stale progress updates after user-initiated seeks
- Scrub Detection: 0.75-second threshold for detecting scrub operations
The system now includes sophisticated scrub detection to handle user-initiated seeks:
sequenceDiagram
participant Master as YoutubeMasterClock
participant Slave as GrainPlayer Service
participant User as User Interaction
Master->>Slave : setReAnchorListener(callback)
loop Every 40ms
Slave->>Master : getLivePosition()
Slave->>Slave : measureJump()
Slave->>Slave : detectScrub(jump >= 0.75s)
alt Scrub Detected
Slave->>Slave : service.seek(positionSec)
else Normal Drift
Slave->>Slave : service.syncAnchor(positionSec, wallSec)
end
end
Diagram sources
The system includes multiple layers of protection against feedback loops:
- Time Update Flag: Prevents service time updates from triggering seeks
- Seek Threshold: Only seeks when difference exceeds 0.5 seconds
- Pending Seek Queue: Queues seek operations during buffer loading
- Seek Token System: Prevents race conditions during user-initiated seeks
Section sources
The playback system uses a centralized store pattern to manage all playback state, ensuring consistency across the entire application.
classDiagram
class PlaybackStore {
+boolean isPlaying
+number currentTime
+number duration
+number playbackRate
+number currentBeatIndex
+number currentDownbeatIndex
+number seekToken
+number lastUserSeekAt
+RefObject~HTMLAudioElement~ audioRef
+setIsPlaying(playing) void
+setCurrentTime(time) void
+setDuration(duration) void
+setPlaybackRate(rate) void
+play() void
+pause() void
+seek(time) void
+setPlayerPlaybackRate(rate) void
+noteUserSeek() void
+onBeatClick(beatIndex, timestamp) void
}
class StoreAdapter {
+noteUserSeek() void
+setCurrentTime(time) void
}
PlaybackStore --> StoreAdapter : "uses"
Diagram sources
The store implements comprehensive state synchronization mechanisms:
- Multi-Source Coordination: Synchronizes YouTube, pitch-shifted audio, and chord playback
- Seek Token System: Prevents race conditions during user-initiated seeks
- Rate Fan-Out: Distributes playback rate changes across all audio sources
- Real-time Updates: Provides immediate state updates to all subscribers
- User Seek Fence: Protects against stale progress updates after seeks
Section sources
New Section The system now includes a comprehensive debugging framework specifically designed for pitch-sync issues.
The debugging framework provides runtime visibility into the synchronization process:
-
Enable/Disable: Controlled via
window.__PITCH_SYNC_DEBUG__orlocalStorage.debug:pitch-sync - Throttled Logging: High-frequency loops use throttled logging to prevent console flooding
- Tagged Events: Structured logging with pitchSync prefixes for easy filtering
- Runtime Flags: Dynamic enabling/disabling without component re-mount
classDiagram
class DebugFramework {
+isPitchSyncDebugEnabled() boolean
+pitchSyncLog(tag, data) void
+pitchSyncLogThrottled(tag, intervalMs, data) void
+__resetPitchSyncDebugThrottles() void
}
class ThrottleState {
<<Map>>
+THROTTLE_STATE Map<string, number>
}
DebugFramework --> ThrottleState : "uses"
Diagram sources
-
Rate Changes:
[pitchSync.rate.change]- tracks playback rate transitions -
Seek Operations:
[pitchSync.seek]- monitors user-initiated seeks -
Drift Corrections:
[pitchSync.drift]- logs slave re-anchor operations -
Gate Decisions:
[pitchSync.gate]- records synchronization decisions -
Performance Metrics:
[pitchSync.perf]- tracks timing and performance
// Enable debugging in console
window.__PITCH_SYNC_DEBUG__ = true;
// or
localStorage.setItem('debug:pitch-sync','1');
// Throttled logging for high-frequency events
pitchSyncLogThrottled('slave.reAnchor', 100, {
drift: drift.toFixed(4),
masterPos: masterPos.toFixed(4),
rate: rate
});Section sources
The audio playback system is designed with mobile device constraints in mind, implementing several optimization strategies:
- Lazy Loading: Tone.js library loads only when needed (150KB reduction)
- Memory Management: Proper cleanup of audio nodes and buffers
- Background Tab Handling: Adapts chord playback for reduced CPU usage
- Battery Efficiency: Minimizes audio processing when not actively needed
- Touch-Friendly Controls: Optimized for touch interaction
- Reduced Latency: Minimizes audio processing delays
- Network Optimization: Efficient audio loading from Firebase Storage
- Cross-Platform Compatibility: Works across iOS Safari, Chrome, and Android browsers
Problem: Audio fails to play on mobile devices or after page reload Solution: The AudioContextManager automatically handles autoplay by resuming on first user interaction
Problem: Audio stops when browser tab becomes inactive Solution: The system detects context suspension and automatically resumes when needed
Problem: Different behavior across browsers (Safari vs Chrome) Solution: The system uses webkitAudioContext fallback and standardized Web Audio API usage
Problem: Audio quality degradation during pitch shifts Solution: Adaptive low-pass filtering prevents aliasing during upward pitch shifts
Problem: Audio desynchronization between YouTube and chord playback Solution: Unified master clock architecture with drift correction maintains perfect synchronization
Problem: Scrubbing causes audio glitches or desynchronization Solution: 0.75-second scrub detection threshold prevents premature seeks during scrub operations
Problem: Continuous seeking between audio sources Solution: Time update flags and seek thresholds prevent feedback loops
Problem: Difficulty diagnosing synchronization problems Solution: Comprehensive debugging framework with runtime logging and throttled events
Section sources
- Always Use AudioContextManager: Never create AudioContext instances directly
- Handle Asynchronous Operations: Audio loading and initialization are asynchronous
- Manage Resources Properly: Always dispose of audio services when no longer needed
- Respect User Controls: Honor user-initiated seeks and rate changes
- Test Across Devices: Verify functionality on different browsers and devices
- Use Debug Framework: Leverage pitch-sync debugging for troubleshooting
- Lazy Load Tone.js: Only load when pitch-shifting is enabled
- Optimize Buffer Sizes: Use appropriate buffer sizes for different audio sources
- Minimize DOM Interactions: Keep audio processing in Web Audio threads
- Monitor Memory Usage: Dispose of unused audio nodes and buffers
- Hook Composition: Use custom hooks for audio functionality
- Store Integration: Leverage the playback store for state management
- Component Wrappers: Use PitchShiftAudioManager for proper initialization
- Error Handling: Implement comprehensive error handling for audio operations
- Debug Usage: Enable debugging framework during development and testing
- Master Clock First: Always update the master clock before slave components
- Rate Verification: Use playback store's rate verification for YouTube iframe alignment
- Seek Coordination: Implement seek token system to prevent race conditions
- Scrub Detection: Use 0.75-second threshold for detecting scrub operations
- Feedback Prevention: Implement time update flags and seek thresholds
- Slave Re-Anchor: Monitor 40ms intervals for drift correction
- Debug Logging: Use pitch-sync debugging for real-time synchronization monitoring
-
Backend Architecture
- Blueprint Organization
- Machine Learning Integration
- Service Layer Architecture
- Backend Architecture
- Error Handling and Logging
- Flask Application Factory
- Frontend Architecture
- Architecture and Design
- Deployment Architecture
- Audio Pipeline
- Audio Playback System
- Audio Processing and Analysis
- Real-time Audio Analysis
- YouTube Integration
- Blueprint Services
- Machine Learning Services
- Backend Services
- External Integrations
- Flask Application Architecture
- Melody Transcription
- Song Segmentation
- Experimental Feature Management
- Experimental Features
- API Integration and Service Layer
-
Component Library and UI System
- Analysis Interface Components
- Chatbot Interface Component
- Chord Analysis Components
- Chord Playback Components
- Common Components
- Component Library and UI System
- Homepage and Landing Components
- Layout and Utility Components
- Lyrics Display Components
- Piano Visualizer Components
- Settings and Configuration Components
- State Management and Data Flow
- Frontend Application
- Next.js Application Architecture
- Beat Detection Models
- Chord Recognition Models
- Adding New Models
- Machine Learning Models
- Model Management
- Model Training and Evaluation