How AI Stem Separation Works

A finished song is a single audio signal with everything already blended together. Separation models learn to estimate what each ingredient sounded like before the blend.

Why this is a hard problem

Mixing is like pouring several colors of paint into one bucket. The sound waves from the voice, the drums, and the bass are added together into one waveform, and there is no simple way to subtract them back out. The same frequencies often overlap too: a kick drum and a bass guitar both live in the low end, and a snare and a vocal consonant can occupy similar ranges. A model cannot just filter by pitch.

Step one: look at the sound as a picture

Many separation models start by converting audio into a spectrogram, an image-like view with time on one axis, frequency on the other, and brightness showing loudness. In a spectrogram, different instruments have recognizable shapes. Drum hits show up as short vertical bursts. Sustained synth pads look like long horizontal bands. A voice shows a stack of evenly spaced lines for its harmonics, shifting as the pitch changes.

Step two: learn from examples

To teach a model what these shapes mean, researchers train it on large sets of multitrack recordings where the individual parts are known. The training loop is simple in principle: mix the parts together, give the model only the mix, and ask it to predict each part. The prediction is compared with the real stem, and the model is adjusted to reduce the error. After many millions of examples, the model becomes good at recognizing what a vocal, a drum kit, or a bass line tends to look like and sound like.

Waveform models, spectrogram models, and hybrids

• Spectrogram models predict a mask over the spectrogram that keeps the parts belonging to one instrument and suppresses the rest.
• Waveform models work on the raw audio directly, which can preserve phase and the sharp attack of drum hits.
• Hybrid models do both and combine the results. Meta's open-source Demucs family is a well-known example of this approach.

Why results are good but not perfect

The model is estimating, not recovering. The original stems are gone for good once a song is mixed, so the output is the model's best reconstruction. That is why you may hear:

• Bleed: a faint trace of drums in the vocal stem, or a ghost of the vocal in the "other" stem.
• Artifacts: a slightly watery or phasey quality, most noticeable on isolated vocals and cymbals.
• Misplaced sounds: a keyboard part landing in the "other" stem when you expected it in synth, because the model has to decide where a sound belongs.

Dense mixes, heavy reverb, and instruments that sound alike are the hardest cases.

Why it takes a while

Separation models are large and do a lot of math for every second of audio. Longer songs take longer, and processing is faster on a graphics card (GPU) than on a regular processor. That is why splitting a full track can take a minute or more rather than a moment.

What to expect in practice

Modern separation is good enough for DJ sets, practice, remixing, and sampling, especially when the stems sit in a full mix with other parts. It is less suited to situations where you need studio-clean isolated tracks. If you want to get the best results, read our tips on getting cleaner stems.

Try splitting a song in Stemline101.