Source separation is the process of isolating individual sounds in an auditory mixture of multiple sounds, and has a variety of applications ranging from speech enhancement and lyric transcription to digital audio production for music. Furthermore, Automatic Music Transcription (AMT) is the process of converting raw music audio into sheet music that musicians can read. Historically, these tasks have faced challenges such as significant audio noise, long training times, and lack of free-use data due to copyright restric- tions. However, recent developments in deep learning have brought new promising approaches to building low-distortion stems and generating sheet music from audio signals. Using spectrogram masking, deep neural networks, and the MuseScore API, we attempt to create an end-to-end pipeline that allows for an initial music audio mixture (e.g...wav file) to be separated into instrument stems, converted into MIDI files, and transcribed into sheet music for each component instrument.
Created as a final project for CS5100, Foundations of Artificial Intelligence, at Northeastern University.
Our project can be split in three primary directories - midi-to-note, stem-separation, and transcriber. All directories have their own demo videos for showing their desired outputs, as well as any relevant examples or documentation for generating files from your own inputs.
This directory contains our approach for converting MIDI data into sheet music via a MuseScore CLI tool. Sheet music can be generated via altering the input_file and output_file variables in midi_conversion_example.py.
This directory contains python files for creating, training, deploying, and evaluating a stem separation model using 7-second clips from the MUSDB18 dataset. Models can be trained from your own MUSDB18 samples, or you can use our pre-trained model for stem separation.
Our stem separation model works for separating vocals only; if more accurate results or more stems are desired, you will likely want to train a model with the entirety of the MUSDB18 dataset rather than our subset.
An end-to-end pipeline for stem separation can be ran from StemSeparationModel.py. All relevant dependencies can be found in environment.yml.
This directory contains python files for creating, training, and deploying an Automatic Music Transcription (AMT) model.
Our AMT model works best with piano, as that was the majority of its training data. For more accurate results with the above stem separation model, it is recommend to retrain the model on note-labelled vocal data.
- MAESTRO Dataset: 47 audio-MIDI pairs were used due to resource constraints.
| Parameter | Value |
|---|---|
Sample Rate (sr) |
22050 Hz |
Hop Length (hop_length) |
512 |
Number of Frequency Bins (n_bins) |
84 |
Bins Per Octave (bins_per_octave) |
12 |
| Standard Audio Duration | 180 seconds (3 minutes) |
| Parameter | Value |
|---|---|
Frame Rate (fs) |
Computed as sr / hop_length |
| Pitch Range | 21 (A0) to 108 (C8) (88 piano keys) |
| Piano Roll Resolution | Binary values (1 for active, 0 for inactive) |
| Parameter | Value |
|---|---|
| Optimizer | Adam |
| Learning Rate | 1e-3 |
| Loss Function | Custom Focal Loss |
| Alpha | 0.35 |
| Gamma | 4.0 |
| Metrics | Precision, Recall, F1 Score (Note-wise and Frame-wise) |
| Batch Size | 32 |
| Epochs | 50 |
| Early Stopping | Patience: 5 |
| Parameter | Value |
|---|---|
| Program | 0 (Acoustic Piano) |
| Velocity | 100 |
Frame Rate (fs) |
43.06 frames/second |
A pipeline for midi generation can be ran from midi_generation.py. All relevant dependencies can be found in requirements.txt.
Without an abundance of accessible online resources, none of our work would have been possible.
- Source Separation Tutorial Landing Page
- ArXiv Paper on Source Separation
- ScienceDirect Article on Source Separation
- IJERT Research on Audio Stems Separation
- MUSDB Dataset for Source Separation
- Seetharaman Paper on 2D Fourier Transform
- Another ArXiv Paper on Source Separation
- NUSSL GitHub Repository
- Tutorial on TF and Masking
- Cal Poly Thesis on Source Separation
- Stanford CS230 Project Report
- Wav2Mid GitHub Repository
- Final ArXiv Paper on Source Separation