nomusA real-time music filter on four platforms.

Client: our own product. Role: everything: product, audio engine, machine learning, four native hosts, site. Live: agrohi.com/nomus · try the demo in your browser

nomus takes the background music out of live audio and keeps the voices, speech and singing. It runs in a Chrome tab, system-wide on macOS and Linux, and on Android. It is free, needs no account, and the audio never leaves the device.

We built it to answer a hard product question with engineering, and to show what we build for clients.

The problem

People want to hear what is being said without the music under it: a lecture with a soundtrack, a podcast with a music bed, a short video with a trending track over someone talking. Some people avoid music for personal or religious reasons. Others find it tiring or distracting.

The only other browser extension we found that filters live audio on the device uses a speech detector: it mutes everything it does not recognise as speech. That removes the music, and it also removes singing and cuts into the voice. We wanted to keep every voice, sung or spoken, and still run on the device.

Constraints

The last row is the one that is easy to get wrong. A model that halves the voice while cutting music by 20 dB scores well on a naive metric and is useless to the listener.

Architecture: one Rust core, four hosts

Everything that decides what you hear lives in one crate. The hosts contain no audio logic; each one is a few hundred lines that turn one platform’s audio into calls on the engine. Every host loads the same model file, so one fix or one new model reaches all four at once.

The engine’s timing is a constant, and every mode is padded to it:

// crates/nomus-core/src/engine.rs
/// Processor latency every mode is padded to, so switching modes never
/// shifts time. Raise this when adding a processor with more lookahead.
pub const PROCESSOR_LATENCY_BUDGET: usize = FRAME_SIZE;

/// Total engine latency in samples (20 ms at 48 kHz).
pub const LATENCY: usize = FRAME_SIZE + Self::PROCESSOR_LATENCY_BUDGET;

The browser. An AudioWorklet has no fetch and no JavaScript glue, so the engine is compiled to WebAssembly with zero imports and exposed as a flat C ABI: one call per block per channel. It cannot reach the network or the disk, and the extension’s content security policy (connect-src 'self') blocks any request to another server.

// crates/nomus-wasm/src/lib.rs
pub unsafe extern "C" fn nomus_process(
    engine: *mut Engine,
    channel: u32,
    input: *const f32,
    output: *mut f32,
    n: u32,
) -> u32 {

Android. Android lets an app capture other apps’ media audio, but does not mute the originals, so a naive filter plays both. nomus captures the audio, attaches a silencing effect to each captured app, and plays the filtered copy through its own track, which is excluded from capture. Audio crosses into Rust through direct buffers, with no copy. Android keeps a mute effect alive after the process that created it dies, so the app has three layers of crash safety to make sure no other app is ever left muted. The app has no INTERNET permission.

macOS. A Core Audio process tap hears every other process and mutes it at the source, with no driver. nomus filters the tap and writes the result to the real output in the same callback. If nomus crashes, the OS removes the tap and sound returns to normal.

// crates/nomus-system/src/macos.rs
let d = CATapDescription::initStereoGlobalTapButExcludeProcesses(CATapDescription::alloc(), &exclude);
d.setName(&NSString::from_str("nomus"));
d.setPrivate(true);
d.setMuteBehavior(CATapMuteBehavior::Muted);

Linux. nomus creates a PipeWire virtual sink, makes it the default output, and plays its filtered contents to the real device. No root, no kernel module.

Windows is not supported. Its loopback capture cannot silence the original sound, so a system-wide filter there needs a signed kernel audio driver. We said no rather than ship something that plays both.

The model and how we trained it

The network does not generate audio. Every 10 ms it predicts one gain between 0 and 1 for each of 481 frequency bins, and the engine multiplies the spectrum by it. The worst a bad prediction can do is turn something down.

Every step of the training pipeline has a Rust twin: the same window, the same framing, the same GRU gate order. The model is trained in the exact form it runs in, and every exported model must pass a parity test natively and through the WebAssembly ABI before it can ship.

Training data: read speech (LibriSpeech), studio singing in nine languages (GTSinger), music stems (MUSDB18), thousands of instrumental tracks (Free Music Archive), and solo instruments that sound like voices, such as trumpet, flute and saxophone (Medley-solos-DB). Mixtures are built on the fly and degraded to sound like real uploads: EQ, reverb, compression and quantisation.

The lesson we would pass to any client: data moved the numbers, architecture did not. Four model generations share one architecture. A more complex frequency-local network was built, tested and not shipped; it gained at most 0.09 dB. Adding a multilingual singing corpus moved unseen singers from +2.58 to +5.85 dB.

Evaluation, including where it fails

No script promotes a model. A checkpoint is rejected if its output mask collapses to a constant, and the best one is chosen on held-out audio with penalties for voice lost. We test on sets built for specific failures: degraded real-world audio, singers the model never heard, music with nobody singing, and singing buried under loud music.

Held-out results for the current model (vocal8), mean over each set:

Against NoMusic, the only other extension that filters live tab audio on-device, on the same 193 held-out clips (this benchmark ran on our earlier vocal4 model):

Where it still falls short, in our own measurements:

• Silence between words. NoMusic mutes everything that is not speech, so it is much quieter in intros and pauses (about 40 dB against our 17 dB). Our choice keeps singing; it also means music is reduced, not silenced.
• Clean speech. The speech-enhancement model inside NoMusic, run on its own, beats ours by about 3.5 dB on clean speech mixtures.
• Saxophone. Solo tenor saxophone is only reduced by 3.8 dB. We need real saxophone recordings, not more training steps.
• Singing buried under loud music. On degraded audio with music 5 dB louder than an unseen singer, vocal8 keeps +1.9 dB of gain, against +3.7 dB for the previous model. We accepted that trade for 4 to 16 dB more removal of voice-like solo instruments.
• Leakage the model thinks is voice. 96% of the music that leaks through is in frames the model scores as voice, so no output gate can fix it. Only training can.
• No listening panel yet. All numbers are objective metrics on held-out audio.

Results

Cost on the audio thread: about 48 µs per 128-sample stereo block in WebAssembly, 1.8% of the real-time budget (measured in Node/V8 on an Apple M4 Pro, not in a live Chrome tab).

What it proves for clients

• Real-time systems. We can hit a fixed latency budget on a real-time thread, with no allocation, no locks and no garbage collector, and prove it with tests.
• One core, many platforms. A single Rust engine behind a browser, a JVM, Core Audio and PipeWire, with every host checked against the same offline reference.
• Machine learning that ships. Data work, training, evaluation that looks for failure, and inference we wrote ourselves, small enough for a phone and an AudioWorklet.
• Privacy as a property of the system. No server to trust. The browser and the OS enforce it.
• Honest reporting. We publish the weaknesses with the wins, because that is what you need to decide.

A note on licences

nomus is free. Its model is trained partly on research datasets, and its weights are licensed CC BY-NC-SA 4.0. Agrohi does not sell nomus or its model; this page describes the work.