Loading benchmark data…
EyesInAI·Loading live benchmark data
Every night our Mac mini runs Qwen3.6-35B-A3B on its own hardware and answers the same graded tests as 4 paid cloud models (Qwen3.6-35B-A3B (OpenRouter), gpt-oss-120b (Together), Haiku 5.5, Sonnet 5.5). Each answer is checked against explicit rules by code, not by another model. This page is rebuilt each morning from the latest night, so it is a running record, not a one-off. See also serving local models.
Latest run: November 10, 2026 (UTC) · 9 nights published · cloud cost of the run $0.19, under a $5 nightly cap
Qwen3.6-35B-A3B (Q4 quantization) on a Mac mini with an Apple M5 Pro and 64 GB of memory. Loaded, the model holds about 22 GB. The median answer took 16.5 s, generating about 66 tokens a second (its own server’s clock).
13 tests, 3 attempts each. A failed call, or a skipped attempt in a test a model was scored on, counts as a fail. The exception is attempts classed as Not run, because a fault in our test setup kept them from reaching the model: those are left out of the score.
| Model | Runs on | Passed | Errors | Not run | Cost of the run | List price per 1M tokens (in / out) |
|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B (hub, Q4) | Our Mac mini | 39/39 (100%) | 0 | 0 | $0 API fees | — |
| Qwen3.6-35B-A3B (OpenRouter) | Cloud API | 38/39 (97%) | 0 | 0 | $0.06 | $0.1 / $0.9 |
| gpt-oss-120b (Together) | Cloud API | 39/39 (100%) | 0 | 0 | $0.01 | $0.15 / $0.6 |
| Haiku 5.5 | Cloud API | 38/39 (97%) | 0 | 0 | $0.0063 | $0.1 / $0.5 |
| Sonnet 5.5 | Cloud API | 38/39 (97%) | 0 | 0 | $0.11 | $2 / $10 |
Cloud costs are at list price. A call whose cost the host did not report is counted at an estimated reserve, so these are estimates, not invoices.
Cloud response times are not shown. Our timer for cloud calls has been stopping when the response headers arrived instead of the full answer, which made one host look about 15× faster than it is, so those times are wrong. The Mac mini times above come from its own server and are not affected.
| Kind | Qwen3.6-35B-A3B (hub, Q4) | Qwen3.6-35B-A3B (OpenRouter) | gpt-oss-120b (Together) | Haiku 5.5 | Sonnet 5.5 |
|---|---|---|---|---|---|
| Reasoning under pressure (5) | 15/15 | 15/15 | 15/15 | 14/15 | 14/15 |
| Format fidelity (4) | 12/12 | 11/12 | 12/12 | 12/12 | 12/12 |
| Long context (4) | 12/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| Test | Qwen3.6-35B-A3B (hub, Q4) | Qwen3.6-35B-A3B (OpenRouter) | gpt-oss-120b (Together) | Haiku 5.5 | Sonnet 5.5 |
|---|---|---|---|---|---|
| reasoning_date_arithmetic | 3/3 | 3/3 | 3/3 | 2/3 | 3/3 |
| reasoning_conditional_probability | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 |
| reasoning_mixed_units | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| reasoning_billing_trap | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| reasoning_knights_knaves | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| format_json_escaping | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| format_constrained_writing | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| format_csv_rfc4180 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| format_number_rounding | 3/3 | 2/3 | 3/3 | 3/3 | 3/3 |
| longctx_multi_needle | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| longctx_distractor_needles | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| longctx_scattered_facts | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| longctx_contradiction_override | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
Passed out of attempts, newest first. * marks a night that did not run the full suite, or had attempts classed as Not run, which are left out of the score. A * beside a score means some attempts were left out of it, so we do not compare it with the others.
| Ran (UTC) | Qwen3.6-35B-A3B (hub, Q4) | Qwen3.6-35B-A3B (OpenRouter) | gpt-oss-120b (Together) | Haiku 5.5 | Sonnet 5.5 |
|---|---|---|---|---|---|
| November 10, 2026 | 39/39 | 38/39 | 39/39 | 38/39 | 38/39 |
| November 9, 2026 | 38/39 | 39/39 | 39/39 | 39/39 | 38/39 |
| November 8, 2026 | 39/39 | 38/39 | 39/39 | 39/39 | 39/39 |
| November 7, 2026 | 38/39 | 39/39 | 39/39 | 39/39 | 38/39 |
| November 6, 2026 | 38/39 | 38/39 | 39/39 | 39/39 | 38/39 |
| November 5, 2026 | 38/39 | 39/39 | 39/39 | 39/39 | 39/39 |
| November 4, 2026 | 39/39 | 39/39 | 39/39 | 39/39 | 37/39 |
| November 3, 2026 * | 20/21 * | 37/39 | 39/39 | 39/39 | 36/39 |
| November 2, 2026 * | 35/36 * | 39/39 | 39/39 | 38/39 | 38/39 |
A separate experiment: Qwen3.6-35B-A3B on the Mac mini with its reasoning turned on and off, 16 tests, 3 attempts each. Run November 2, 2026 (UTC).
| Setting | Passed | Median time | Ran out of tokens |
|---|---|---|---|
| Thinking on | 47/48 | 16.4 s | 1 |
| Thinking off | 35/48 | 0.2 s | 0 |
Turning thinking off lowered the pass rate from 98% to 73%; the median answer took 0.2 s instead of 16.4 s.
13 short tests, 3 attempts each: a few points either way can be chance. "$0 API fees" leaves out the price of the machine and the electricity it draws while it answers. The local model is a Q4 quantized copy; the cloud copy of the same model id may not be. Tool use and open-ended writing are not tested here.
Tests from our Deep Suite that code can grade (reasoning, format and long-context recall), 3 attempts per test for each model every night from 04:00 Mac mini time (01:10 before 5 November 2026), every model with the same output budget and its host's default sampling, under a $5 nightly cap on cloud spend. A published night counts if it ran the full suite, or is marked partial if it stopped early (for example at the cap) or some attempts were classed as Not run. A night with no completed calls is not published; the figures are rebuilt the next morning.
What the same model does on the real inputs of our own jobs: we are testing Qwen on a Mac mini.