Honest evaluation: how we measured a music filter, and where it still fails

It is easy to remove music by removing everything. The first model we trained on real recordings did a milder version of that. On 48 held-out mixtures it improved SI-SDR by 3.2 to 5.6 dB, a respectable number, and it turned the voice down by 4.1 to 6.4 dB in every cell. SI-SDR is scale-invariant, so a quieter but cleaner voice cost it nothing.

Two later runs were worse: they trained to the end with healthy-looking logs and did nothing at all. Their output equalled their input.

nomus is a real-time filter that removes background music and keeps every voice, speech and singing, on the device. This post describes how we measure it now, which failure each rule answers, and where the model it ships, vocal8, still fails.

Three numbers, not one

Every table we use has three columns, because each answers a question the others cannot.

Voice level is measured where the clean voice is active; music removed, where it is silent (more than 40 dB below its loudest 20 ms segment).

SI-SDR is computed in Rust, in f64:

pub fn si_sdr(estimate: &[f32], reference: &[f32]) -> f32 {
    assert_eq!(estimate.len(), reference.len());
    let dot: f64 = estimate
        .iter()
        .zip(reference)
        .map(|(&e, &r)| e as f64 * r as f64)
        .sum();
    let ref_energy: f64 = reference.iter().map(|&r| (r as f64) * (r as f64)).sum();
    if ref_energy <= 0.0 {
        return -120.0;
    }
    let alpha = dot / ref_energy;
    let mut target_energy = 0.0f64;
    let mut noise_energy = 0.0f64;
    for (&e, &r) in estimate.iter().zip(reference) {
        let t = alpha * r as f64;
        let n = e as f64 - t;
        target_energy += t * t;
        noise_energy += n * n;
    }
    // Silence (or an orthogonal estimate) is not perfect reconstruction.
    if target_energy <= 0.0 {
        return -120.0;
    }
    if noise_energy <= 0.0 {
        return 120.0;
    }
    (10.0 * (target_energy / noise_energy).log10()).clamp(-120.0, 120.0) as f32
}

Two details. It does not subtract the mean first, as the textbook definition does, so its absolute values should not be set beside published tables. And a silent estimate against a silent reference now scores −120 dB. It used to score +120, “perfect”, and inflated averages over clips with no voice.

The fix for the quiet-voice model was in the loss, not the metric: −(0.5·SI-SDR + 0.5·SNR). The SNR term is not scale-invariant, so turning the voice down costs something. In the next run the voice level sat between −1.0 and −0.1 dB in every cell.

Every metric ignores something, including voice level. A model that passes its input through untouched reads as keeping the voice perfectly, or better: a collapsed run read +6.21 dB of “voice level” at −5 dB SNR, because its output was the whole mixture. The column only means something beside the gain.

Two runs that did nothing

vocal1 (2,145,315 parameters, 40,000 steps) and vocal2 (1,037,475 parameters, 30,000 steps) both collapsed to a constant mask of 1.0. On the held-out set every cell read +0.0 dB gain. vocal2’s training log read SI-SDR 8.499 at step 29,975. The logged number was the absolute SI-SDR of the output on the training batch, and a passthrough output scores exactly its input’s SI-SDR, which is positive on average. The log measured the data, not the model.

The parity check between PyTorch and the Rust engine passed for both. vocal1’s mask difference was exactly 0.000e0: the engine reproduced a useless model faithfully. Parity says two implementations agree; it says nothing about whether the thing they agree on works.

At the time, export wrote straight into the extension’s package. The collapsed weights never reached a release, but only because someone looked.

What that left:

• The checkpoint that ships is the best one on a fixed held-out validation batch, never the last one.
• A collapse gate. No checkpoint can win unless its validation gain is positive and its mask actually varies:

def selection_score(gain: float, mask_std: float) -> float:
    return gain if math.isfinite(gain) and math.isfinite(mask_std) and gain > 0 and mask_std > 0.05 else -math.inf

• No script promotes a model. Training and export write only into the run’s own folder and print the copy command a person would run.

Test sets built for the failures

Almost every held-out set was added because the sets before it could not see a problem.

The rule underneath: a test built like the training data measures generalisation to new files, not to new conditions, and new conditions are what users bring. Nobody plays clean stem sums. They play mastered, compressed, re-encoded uploads.

Two cautions we keep in view. The sets are small: 24 or 48 mixtures, six per cell, with no confidence intervals, so a difference of a few hundredths of a decibel is noise. And they are reused. Every model since vocal4 has been judged on the same files, and the research notes call them “reused development sets” rather than pretend otherwise.

Score the code that ships

The voice sets are scored through the Rust engine, not PyTorch. The CLI runs the model file through the same engine every app uses, in 128-sample blocks like Web Audio, and removes the engine’s fixed 960-sample delay. The scorer then checks alignment with a guarded cross-correlation, because a latency mistake of one 10 ms hop makes a perfect output look like a broken model. A healthy run prints “sample-aligned” on every clip.

RNNoise, which nobody trained for this task, is in every table as a baseline. A positive gain only means something if it is clearly above what a generic denoiser does. Here is vocal8 on the hardest set, unseen singers with degraded audio:

voice      SNR mode        n  SI-SDR in    out   gain  music att.  voice lvl
singing    +10 rnnoise     6        7.4    6.5   -0.8       -10.8      -2.25
singing    +10 voicemask   6        7.4   12.3   +5.0        -8.6      -0.39
singing     +0 rnnoise     6       -1.6    1.7   +3.3       -17.3      -2.38
singing     +0 voicemask   6       -1.6    8.2   +9.8       -16.8      -0.54
singing     -5 rnnoise     6       -6.1   -8.1   -2.0       -18.8      -2.37
singing     -5 voicemask   6       -6.1   -4.2   +1.9       -21.4      -4.31

At +10 dB, vocal8 gains 5.0 dB and costs 0.39 dB of voice; RNNoise makes the output worse. At −5 dB vocal8 still gains, but the voice comes out 4.31 dB quieter. Hold on to that cell.

Worst cells, not only means

Each comparison is summarised per set with the mean gain, the worst cell, music removed and voice level:

testset-gts-degraded  candidate gain +5.88  worst cell +1.9  music removed 14.7 dB  voice -1.41
testset-gts-degraded  shipped   gain +5.90  worst cell +3.7  music removed 13.6 dB  voice -1.18

The means say vocal8 (candidate) and vocal6 (the model it replaced) are equal on this set. The worst cell says they are not. We learned to read that column the hard way, as the next sections show.

A competitor benchmark we partly lost

NoMusic is the only other extension that filters live tab audio on the device. We read its source, reproduced its chain offline in Node with its own model files (a speech-enhancement network, RNNoise twice, then a gate that mutes anything RNNoise does not classify as speech), and scored it on the same 193 held-out clips as nomus, each run three ways: the mix, the voice alone, the music alone. The nomus side was vocal4, the model shipping at the time.

NoMusic’s gate is a speech detector, and singing is not speech to it. The same gate makes it much quieter when nobody talks, and users who want no music at all during intros and breaks will hear that.

The ablation found a second loss. NoMusic’s network alone, without the gate, beat nomus by about 3.5 dB on clean speech mixtures (+9.04 against +5.84 dB on MUSDB speech). The note concluded against us: our speech quality has room to improve, and it is not only a matter of training data.

Its caveats, stated in the note: an offline reproduction, not the extension in Chrome; and test sets built for nomus’s target, so “a fair basis for this product claim but not a neutral benchmark”.

The gate that did not work

A gate is cheap, so we simulated three on vocal4’s output: one driven by the model’s own voice reading, one by output level against a running voice peak, and one requiring both. Energy-weighted music removal stayed between 17.3 and 17.8 dB in every design. The one real improvement in pauses, about 7 dB, cut roughly one voiced moment in nine.

The reason was in the data: 96% of the music energy that leaked on music-alone inputs came from frames where the model’s voice reading was above −10 dB, and 66% from frames above −3 dB. The music that leaks is music the model believes is voice: trumpet, flute, violin, saxophone. A gate driven by the model’s own signals can only close where the model already hears no voice, and those frames were already quiet. Verdict: do not ship. The fix had to be in training.

Fixing it in training, and paying for it

vocal7 warm-started from vocal6 with solo-instrument hard negatives from Medley-solos-DB and more music-only examples with a silent target (20%, up from 8%). It did what was asked: solo trumpet removal went from 5.2 to 23.2 dB. Its voice-set means moved little. But in one cell it failed badly. On unseen singers, degraded, with music 5 dB louder than the voice, gain fell from +3.7 to +0.7 dB and the voice from −3.2 to −5.4 dB. Music-only training taught the model that silence is sometimes right, and with a buried, distorted voice that became a bias toward silence.

Selection had not seen it. At the chosen step, the general validation voice level had lost 0.13 dB, inside the 0.5 dB tolerance. Fully degraded singing at −5 dB was a small slice of the validation mixtures.

vocal8 made the three changes the vocal7 note asked for: 13% music-only examples, two more saxophone recordings moved into training, and a fixed validation set of 32 degraded singing mixtures at −5 dB with its own terms in the selection score:

        if "hard_singing_gain" in extra:
            # Reward gain on buried singing, and charge any voice lost there
            # beyond 0.5 dB of the starting model, as for the general mixtures.
            hard_loss = max(0.0, baseline["hard_singing_voice_db"] - extra["hard_singing_voice_db"] - 0.5)
            score += args.hard_singing_weight * (extra["hard_singing_gain"] - baseline["hard_singing_gain"]) - 2.0 * hard_loss

The penalties are relative to the starting model. Every run warm-starts from the shipped model, and its untrained step 0 competes for best, so a run can only replace its parent by beating it. At step 500 vocal8’s score was 3.199 against the parent’s 5.834, and the parent stayed best until training earned its way past it.

What vocal8 bought and what it cost

Music alone, energy removed, same 40 clips per group for all three models:

The cells where the trade lives, all degraded at −5 dB:

On every voice set, vocal8’s mean gain is equal to vocal6’s or better, within 0.05 dB where it is not ahead. It removes 4 to 16 dB more of the voice-like solo instruments. It recovered part of vocal7’s loss on buried singing, not all of it.

Where it still fails

• Tenor saxophone is still mostly passed: 3.8 dB removed, the lowest of any instrument. Five recordings are not enough. The research note says it needs “a real source of sax material, not more steps”.
• Buried, degraded singing. With music 5 dB louder than a degraded voice, vocal8 leaves the voice about 1 to 1.5 dB quieter than vocal6 did, and keeps 1.8 dB less gain on unseen singers.
• The validation set under-reads that gap. At the chosen step, the 32 buried-singing mixtures showed a voice loss inside the 0.5 dB tolerance, so no penalty applied, while the test cell showed a 1.1 dB loss for the same model. Adding a term to a score moves the blind spot; it does not remove it.
• Silence when nobody sings. NoMusic’s gate removed 40.5 dB of music alone against vocal4’s 17.3. vocal8 removes far more of solo instruments than vocal6 did, but we have not re-run the NoMusic benchmark with vocal8, and its music-only clips differ from the benchmark’s, so the numbers cannot be set side by side.
• Clean speech. A dedicated speech-enhancement network beat nomus by about 3.5 dB on clean speech mixtures.
• Real codecs. Degradation is simulated by quantisation. Real AAC or Opus round trips have not been tested.
• No listening panel. Every number here is a signal metric. No study has run blind listening, and each note says so.
• Two scorers. The voice sets run through the Rust engine; the music-only scores run the PyTorch checkpoint. Parity checks connect the two, but they are not the same code path.

A person decides

The life of one experiment:

No threshold would have made the past decisions correctly. vocal4 was promoted although it lost up to 0.6 dB per MUSDB cell, for 2.6 to 4.7 dB more music removed on unfamiliar music. vocal6 was promoted although its MUSDB voice level fell from −0.30 to −0.80 dB, because unseen singers went from +2.58 to +5.85 dB. vocal7 raised two voice-set means and was rejected for its hardest cell. vocal8’s research note said “not promoted automatically; the best candidate so far”. A person weighed its buried-singing cost against the instrument gains and chose it. Its model card states the trade, the weights’ hash and the one-line command that restores vocal6.

If you are evaluating a model of your own

• Report what the metric ignores beside it: level next to SI-SDR, voice kept next to music removed.
• Put an untrained baseline in every table.
• Report the worst cell next to the mean.
• Give “is the model doing anything at all” its own check, separate from parity and from the training log.
• Build each new test set to look like what the model has not learned yet.
• Benchmark competitors with your own yardstick, publish the axis you lose on, and state the bias.
• Do not let a script promote a model.

You can hear vocal8 on the live demo and judge the trade yourself. The Chrome extension, which ships vocal8, is on the nomus page.

Agrohi builds production systems like this for startups and teams. Tell us what you are building.