Blog
Writing on benchmarks.
How we test a model before we trust it, with the numbers.
Posts
Every post, newest first.
EngineeringWe Benchmarked Jev Against Our Own Classifiers. Every Router Stalled at 70%.
We set out to replace five LLM classifiers with a model built for the job. It was 15x faster and up to 44x cheaper - and no more accurate. Tracing all 507 decisions showed why, and the answer was sixteen text files none of us had read.
EngineeringEight Rules for Benchmarking a Model You're About to Trust
One request, ten runs, two defensible answers eleven seconds apart. Every rule below is here because skipping it would have let us publish something flattering and wrong - including the one that nearly made us reject a model that worked.