We ran our router against OpenRouter's for 65 real turns. Here is what actually happened.

Across 65 turns on three chatbots, our routing cost $0.03310 against auto-beta's $0.08000 — 2.4× cheaper in aggregate. On quality, of 65 turns where the two routers picked different models, 56 were graded a tie, we scored higher on 8, and auto-beta scored higher on 1. The two routers disagreed on the model for every single turn.

The aggregate hides the most interesting thing, so look at the last column: the cost gap runs from 6.0× down to 1.0×. On one bot the advantage nearly disappears.

On the DLR back-office assistant the two arms cost within a few percent of each other. Auto-beta happened to route those turns to genuinely cheap models that were also good enough — which is exactly what a router is supposed to do. There was no cost story to tell. What remained was a quality gap (4 divergent turns to us, 0 to them) and a latency gap: we answered roughly three times faster.

We could have left that bot out. The aggregate looks better without it, and nobody would have known. But a comparison you only publish when it flatters you isn't a measurement, it's an advertisement — and the whole argument for measured routing is that we publish the number even when the number is inconvenient. The honest read: cost is the headline only when popularity happens to favour an expensive model.When it doesn't, our edge is quality and speed, which is a narrower claim and a truer one.

The version of this comparison we published on 22 July claimed 7.4× cheaper. That number was real, but it was measured against the wrong opponent. OpenRouter ships two routers: openrouter/auto, a general meta-router that has existed since 2023, and openrouter/auto-beta, the task-aware one that launched in July 2026. Our harness called auto while every comment, log line and published page described the auto-beta mechanism. They are different model ids that route differently — on an identical prompt, one picked a model costing about twenty times what the other picked.

Re-running against the router we had been describing cut our advantage from 7.4× to 2.4×. That is the correction, and it is the kind of thing that only surfaces if you check your own claim before someone else does. The failure was ours: the harness was written before auto-beta existed, the constant was never revisited, and the label on the chart drifted away from the thing being measured. It now records which rival each run used, so a future run can't quietly compare against a different opponent.

Spend-share routing answers “what is everyone else buying for prompts shaped like this?” It is a good question and a genuinely useful default — it needs no setup, no benchmark, and no knowledge of your workload. But popularity is a proxy. It tells you what the crowd bought, not what passes yourtasks. Measured routing answers a narrower question: on a benchmark of this client's own work, which is the least expensive model that actually scored a pass? When those two answers coincide, the routers tie — which is what 56 of 65 divergent turns look like. When they diverge, measurement wins on cost, on quality, or both.

That is also why the advantage is uneven rather than uniform. We are not claiming a better model; both arms draw from the same pool. We are claiming a better selection signal— and a selection signal only pays when the crowd's default is wrong for your workload. Sometimes it isn't.

65 turns across three bots is a small, dated sample. It is directional, not a guarantee, and a single run carries real variance — the reference judge is our chosen best judgment, not an oracle. The honest summary is three sentences: the two routers almost never pick the same model; quality is close to a wash with a consistent edge to us (8 wins to 1 across 65 divergent turns); and the cost advantage is real but varies by an order of magnitude depending on the bot.

Every figure on this page is read from the archived snapshot of that run (2026-07-28), committed alongside the page — so the article can't drift from the measurement it describes. The full comparison re-runs and carries the current numbers. For what the two routing philosophies are actually arguing about, see spend share vs. measured quality.

We re-ran the identical harness three weeks later, and the cost result flipped: auto-beta now routes most turns to gemini-2.5-flash-lite-class models and came out cheaper than our measured picks on all three bots — roughly 10× on one of them. Quality stayed a wash (50 ties across 57 divergent turns) and our latency edge held at 2–5× faster. The crowd moved down-market, and this time popularity landed on models that also pass.

We're publishing that the same way we published the numbers that flattered us — the live page now shows the August data. It is also the whole argument of this article played back at us: a comparison is a photograph, and any one-shot number — theirs or ours— goes stale the moment the other side ships. The response of a measured router isn't to argue with the photograph; it's to re-bench the newer cheap models on each client's own suites and take the price cut wherever they actually pass — with the real-traffic veto standing between a passing suite and production, because we've already caught a cheap model this month that passed every suite and failed the real turns.

We ran that re-bench the same day. The flash-lite class passed outright on two of the three bots and per-action on the third, our routing moved onto it wherever it earned the slot, and our serving cost dropped roughly 55% run-over-run — with auto-beta's remaining lead narrowing to 2.3–3.7×. The re-bench also caught a second lesson the hard way: one candidate briefly ranked “cheapest” because its price was unknown and a sentinel turned unknown into $0 — unknown is not zero, and the fix now keeps unpriced models out of the cost ranking entirely.