One Rust engine, four hosts: browser, Android, macOS and Linux

nomus removes background music from live audio and keeps the voices, speech and singing. It runs in a Chrome tab, on an Android phone, and system-wide on macOS and Linux. Every one of those hosts runs the same Rust engine and loads the same model file. Each native host has been checked against the offline engine on the same input, with residuals of −76 dB or lower, and the WebAssembly build matches the PyTorch reference to within 8.6e-7.

The engine is one crate with two dependencies and no I/O. Each host is a few hundred lines that turn one platform’s audio into calls on it. This post is about that split: what the engine promises, what each host has to provide, and the parts of each platform that no shared library could solve.

The constraints came first

The product request is one sentence: let me hear the people, not the music. Turned into engineering terms, it became a short list of constraints, and each one ruled something out.

The model is a GRU that predicts one gain between 0 and 1 for each of 481 frequency bins every 10 ms. It has 1,037,475 parameters and ships as a 4,150,261-byte file. The engine multiplies the spectrum by the mask and resynthesises, so the worst a bad prediction can do is attenuate.

Why Rust

The first decision was Go or Rust, and the constraints made it.

• WebAssembly with no JavaScript glue. Rust’s wasm32-unknown-unknown target produces a module with zero imports. That module can be instantiated inside an AudioWorklet, where there is no fetch and no TextDecoder.
• No runtime and no garbage collector. A pause on the audio thread is an audible click.
• A C ABI everywhere. The same extern functions serve JavaScript, the JVM and Core Audio callbacks.

Go can target WebAssembly, but it brings its runtime and scheduler along, and neither belongs on a real-time audio thread.

The same reasoning ruled out an ML runtime. The network is a LayerNorm, two linear layers and two GRU layers of width 256, which comes to a few hundred lines of hand-written Rust. A runtime would have brought megabytes of code, its own threads, imports the worklet cannot satisfy, and a model format we do not control. The price is that every architecture change is written twice, in PyTorch and in Rust, and must pass a parity test before it ships.

The engine’s contract

nomus_core::Engine takes 48 kHz float audio for one channel at a time, in blocks of any size up to 4,096 samples, and returns the same number of samples. Internally it works in 480-sample frames (10 ms). The output is the input, filtered, exactly 960 samples (20 ms) later, in every mode: bypass, RNNoise or the trained mask. Because the delay never changes, a host can switch modes or models mid-stream without the audio jumping in time.

    /// Largest block a host may pass to [`Engine::process_channel`].
    pub const MAX_BLOCK: usize = 4096;

    /// Processor latency every mode is padded to, so switching modes never
    /// shifts time. Raise this when adding a processor with more lookahead.
    pub const PROCESSOR_LATENCY_BUDGET: usize = FRAME_SIZE;

    /// Total engine latency in samples (20 ms at 48 kHz).
    pub const LATENCY: usize = FRAME_SIZE + Self::PROCESSOR_LATENCY_BUDGET;

The operations a host uses are the same on every platform. Only the spelling changes:

The mode numbers are part of every ABI: bypass 0, RNNoise 1, voice mask 2.

The shape of the system

Everything that decides what you hear lives in nomus-core, which depends on realfft and nnnoiseless and nothing else. The host crates contain no audio logic.

Planar in the engine, interleaved at the edge

Audio APIs disagree on layout. Web Audio hands a worklet one Float32Array per channel (planar). Android’s AudioRecord, Core Audio buffers and the PipeWire format nomus negotiates are interleaved: L, R, L, R. The engine is planar, because each channel owns its own buffers and processor state.

Three hosts need the same conversion, so it lives in the core as nomus_core::Interleaved. Its whole hot path:

    pub fn process(&mut self, input: &[f32], output: &mut [f32], frames: usize) -> bool {
        let ch = self.channels;
        let samples = frames * ch;
        if input.len() < samples || output.len() < samples {
            return false;
        }
        let mut start = 0;
        while start < frames {
            let n = (frames - start).min(Engine::MAX_BLOCK);
            for c in 0..ch {
                for i in 0..n {
                    self.planar_in[i] = input[(start + i) * ch + c];
                }
                self.engine
                    .process_channel(c, &self.planar_in[..n], &mut self.planar_out[..n]);
                for i in 0..n {
                    output[(start + i) * ch + c] = self.planar_out[i];
                }
            }
            start += n;
        }
        true
    }

The scratch buffers are allocated once in new, so nothing allocates on the audio thread. A test runs 5,000 stereo frames through it and compares each channel bit for bit with a separate engine fed the same channel planar.

Interleaved started inside the Android crate. When the desktop host needed the same thing, copying it would have left two implementations to keep in step, and depending on the Android crate would have pulled JNI into a desktop build. So it moved into the core, and the Android crate kept a one-line alias. That became the rule for the whole design: anything two hosts need belongs in the core.

One thread, one engine

Every engine method that changes state takes &mut self, and there is no lock inside. The engine holds a Box<dyn Processor> whose trait has no Send bound, so Engine is neither Send nor Sync. In safe Rust you cannot even move it to another thread.

The foreign-function hosts re-establish that rule by hand, and they all do it the same way: the UI never calls the engine. It publishes a desired state, and the audio thread applies it between blocks.

• Browser: the worklet’s process and its message handler both run on the audio rendering thread.
• Android: one Kotlin thread creates the engine, runs every process call and destroys it. The UI writes @Volatile fields.
• macOS: the pipeline belongs to the Core Audio IOProc thread. Linux: it is created on the PipeWire thread and never leaves it. On both, the UI writes relaxed atomics that the audio callback reads.

No locks sit between the UI and the audio thread on any platform.

Host 1: the browser

The browser was the first host to ship and the most restrictive. A Manifest V3 extension cannot hold an AudioContext in its service worker, so an offscreen document owns the audio graph and runs the captured tab through an AudioWorkletNode hosting the WebAssembly engine. While a tab is captured, Chrome plays its audio only through the capture, so the user hears only the filtered copy.

The module is 815,166 bytes with no imports, instantiated with an empty import object. Processing one block of one channel is one exported call. The second post in this series covers that host in detail.

Host 2: Android, where capture is a copy

Android 10 added playback capture: with the user’s consent, an app can record what other apps play. The captured app keeps playing through the speaker. For a filter that is a problem, because you would hear the original with its music and, about 200 ms later, the filtered copy.

So nomus does two jobs. It captures media audio, filters it through the engine over JNI, and plays the result on its own track, which is marked uncapturable. It also mutes each captured app’s original by attaching a DynamicsProcessing effect at −200 dB to that app’s audio session. Session ids are not exposed to other apps; nomus reads them from the audio policy service’s dump, which needs the DUMP permission, granted once from a computer with adb. That technique comes from RootlessJamesDSP; the implementation is our own. Apps that refuse capture are left alone and listed as “Not filtered”, because muting what capture cannot hear would silence it.

The JNI boundary copies no audio. AudioRecord fills a direct ByteBuffer, Rust reads it and writes a second one, and AudioTrack drains that. Before building slices, Rust checks capacity, alignment and overlap:

    let samples = frames as usize * h.channels();
    let bytes = samples * std::mem::size_of::<f32>();
    let aligned = (ip as usize) % std::mem::align_of::<f32>() == 0
        && (op as usize) % std::mem::align_of::<f32>() == 0;
    let overlaps = (ip as usize).abs_diff(op as usize) < bytes;
    if icap < bytes || ocap < bytes || !aligned || overlaps {
        return JNI_FALSE;
    }

The hardest problem was not audio. Testing showed that an effect created on another app’s session outlives the process that created it: after kill -9, the other app stayed silent. So nomus carries three layers of crash safety: a ledger of muted sessions written to disk before each mute, a watchdog in a second process that restores the ledger when the main process dies, and a restore on the next launch for a force-stop that kills both. The first version failed one kill -9 round in three, because of a race between the service being destroyed and the death notification. After the fix, 5 of 5 rounds restored every session.

On an API 36 arm64 emulator the engine costs about 0.4 to 0.7 ms per 10 ms stereo block, and the whole capture-to-speaker path is about 180 to 200 ms: roughly 80 ms of capture, 95 ms of playback and the engine’s 20. Most of that is the platform’s capture path, not ours. These are emulator numbers; real phones are the next measurement.

Hosts 3 and 4: the desktop, where muting decides everything

Capturing “what the computer is playing” is easy on every desktop OS. Keeping the original off the speakers is the hard part, and it decides which platforms nomus supports. Without it, the listener hears the full original plus a voice copy 20 ms or more late. Two copies of a voice 20 ms apart form a comb filter with notches every 50 Hz, starting at 25 Hz. That is worse than doing nothing.

macOS 14.2 and later has Core Audio process taps. A global tap that excludes nomus and uses the Muted behaviour captures every other process and keeps it off the hardware, with no driver:

        let me = own_process_object()?;

        // SAFETY: plain Objective-C object construction and setters.
        let description = unsafe {
            let exclude = NSArray::from_retained_slice(&[NSNumber::new_u32(me)]);
            let d = CATapDescription::initStereoGlobalTapButExcludeProcesses(CATapDescription::alloc(), &exclude);
            d.setName(&NSString::from_str("nomus"));
            d.setPrivate(true);
            d.setMuteBehavior(CATapMuteBehavior::Muted);
            d
        };

nomus reads the tap through a private aggregate device whose clock master is the real output, and runs the engine in the same I/O callback that writes the output. Teardown runs in reverse order in Drop, and destroying the tap unmutes everything. If the process dies, the OS does the same: a kill test recorded 39 mutes and 39 unmutes in each of 3 rounds. One trap shaped the code. Without the “System Audio Recording” permission, the OS still creates the tap and mutes every app but never runs the callback, so the Mac goes silent. nomus refuses to create a tap until the permission is granted.

Linux has no equivalent of a muting tap, and does not need one. nomus creates a PipeWire stream that is itself a sink named nomus, makes it the configured default output, and plays the filtered result to the real device through a second stream. Apps never send audio to the hardware, so nothing leaks. No root and no driver are involved; it is an ordinary client of the user’s session, the same pattern EasyEffects uses. A kill test found one leftover: after a crash, the configured default still named the vanished sink, and the next normal stop cleared the user’s original choice. The fix works like the Android ledger. Before taking over, nomus saves the user’s configured default to a small state file. It hands that value back on a normal stop, on the next start, or on an explicit recover. In a kill test run in Docker containers, all 7 scenarios restored the user’s output. Tests on real Linux hardware are next.

Windows is not supported. Its capture APIs, WASAPI loopback and process loopback, cannot silence the original. The route that works is a kernel-mode virtual audio driver, signed through Microsoft’s attestation process with an EV certificate. The engine would not change; the cost is the driver and the certificate.

Both desktop backends feed one shared pipeline that resamples only when the device is not at 48 kHz, one supervisor thread that rebuilds the session when the output device changes, and one Tauri app. The native part is one module name: lib.rs compiles macos.rs or linux.rs and imports it as platform.

Proving it is the same engine everywhere

A filter that sounds plausible can still be wrong: a byte-order slip in a buffer, swapped channels, a stale model in a package. The strongest check is to run the same input through the host and through nomus-cli, and subtract. The CLI is the reference host: it streams WAV files through the engine in 128-sample blocks, like Web Audio, and trims the 960-sample delay.

The Android floor is the CLI’s own 16-bit output: the largest sample difference is about 1.25 steps of a 16-bit sample. The Android host feeds 480-frame blocks and the CLI feeds 128-sample blocks, and the outputs still agree. The engine’s output does not depend on the host’s block size.

End to end on the desktop, the live filter removed 18.2 dB (macOS) and 20.0 dB (Linux) of music in the pauses between words on a held-out clip.

What the design costs

• Hosts add latency the engine cannot remove. The estimate on macOS is about 44 ms with 512-frame buffers. On Linux the reported 31 ms leaves out the PipeWire quantum, so the real figure is higher. On Android, video apps cannot compensate for the added delay.
• The boundaries are unsafe. Each JNI and WebAssembly entry point carries its own checks for alignment, capacity and aliasing. A mistake there is a memory error, not an exception.
• Android muting needs a one-time adb step and reads a dump format that is not an API. If it changes, nomus fails safe: doubled audio, never silence.
• Some audio cannot be filtered. Android apps that block capture (Chrome, Spotify and Netflix among them) play unfiltered. On Linux, streams pinned to a specific device bypass the default and bypass nomus.

Where it stands

If you are building a product that has to run on more than one platform, the lessons that carried over are small. Put anything two hosts need in the core. Give every boundary a written contract and a test on the real binary, not only on the library. Keep one simple reference host and compare every other host against it by subtraction. And find out early what each platform forbids, because that, not the shared code, sets the schedule.

You can hear the engine in your browser on the live demo, and get the Chrome extension from the nomus page. The desktop and Android apps are coming soon.

Agrohi builds production systems like this for startups and teams. Tell us what you are building.