Blog
Engineering notes.
How we build, measure and ship. Written from real projects, with the numbers and the parts that did not work.
How we test an AI SRE: a fake cluster, 17 scenarios, and no vibes
The harness behind Berth's AI SRE evals: a fixture cluster, the product's real prompt and tools, deterministic scoring, stub models that prove the scorer, and what none of it can tell you.
The eval scoring we got wrong, and what we did about it
Three rounds of scoring problems in an agent evaluation: regexes too narrow, a credential check that fired on a redacted password, and the three lines we drew for changing criteria after reading answers.
Fewer tools, better agent? What merging 18 tools into 15 did and did not buy
We measured how a small model used our agent's tools, found three that overlapped, merged them without changing the security boundary, and measured again. A smaller prompt, no change in pass rate, and a problem the merge did not touch.
Where a small local model still fails at SRE work: a taxonomy from 276 runs
Real failures from repeated runs of a local model on scripted Kubernetes incidents: skipped evidence, a misread unit, backwards capacity reasoning, shallow discovery, over-exploration, and omissions that every model made.
Your dbt descriptions are your prompt
How dbt persist_docs turns YAML descriptions into Postgres COMMENTs and then into an AI analyst's data dictionary, the stale mart description that told the agent its USD columns did not exist, and the deploy chain that decides when a fix reaches it.
When the agent runs out of tool calls: two small fixes that mattered
An agent loop with a tool-call cap can end without an answer. How we found it in a local-model agent and a cloud one, what we changed (a forced final answer, an explicit instruction, one retry), and how much it did and did not prove.
One Rust engine, four hosts: browser, Android, macOS and Linux
How nomus runs one real-time audio engine in a Chrome extension, an Android app and desktop apps for macOS and Linux, and what each platform demanded that the engine could not provide.
20 ms end to end: real-time audio in an AudioWorklet with zero-import WASM
How nomus runs a million-parameter recurrent network on the browser's audio thread: a Rust engine compiled to a WebAssembly module with no imports, a flat C ABI, and a worklet that never throws.
Honest evaluation: how we measured a music filter, and where it still fails
The metrics, test sets and selection rules behind nomus's voice-preserving music filter, the runs that collapsed, the competitor benchmark we lost on one axis, and the cases where the shipping model still fails.
Follow new posts with the RSS feed.