Measured routing vs. spend-share routing

Spend-share routing answers “what is everyone else using for tasks shaped like this?” Measured routing answers “which model did a benchmark on this client's own tasks prove passes?” The two never agree on the model — divergence is 100% in every run we've done — and which one wins on cost has now gone both ways. In July our measured picks were 2.4× cheaper in aggregate. In this August refresh, auto-beta routed most turns to gemini-2.5-flash-lite-class models and came out cheaper on all three bots. The crowd's favourite moved down-market, and popularity happened to land on models that also pass. That is the honest current state, and it is on this page because the comparison re-runs — a July screenshot would still be claiming the opposite.

What stayed constant across both rounds: quality is a wash (ties dominate every run — here 50 of 57 divergent turns), and latency is not— our routed answers come back 2.4× faster (1161ms vs 2798ms average), because measured routing answers from a small pinned pool instead of a marketplace hop. For an interactive chatbot, seconds of first-token delay are a product defect, not a rounding error. And spend-share still carries the risk measurement exists to catch: the crowd can move to a model that passes a replay and fails your real traffic — the failure mode we document, with receipts, in the real-traffic veto. Our next move, since the router is measured: re-bench the newer down-market models against each client's suites, and let the routing take the price cut only where they pass.

Method: each chatbot's own turns, run through both routers, both answers graded against the known-correct answer by the same reference judge. The rival arm is openrouter/auto-beta, OpenRouter's task-aware router — a different model id from the older openrouter/auto, which routes on a different basis and gives a much larger gap. A small, dated sample — directional, not a guarantee. Generated 2026-08-16.