We are testing Qwen on a Mac mini

We run Qwen3.6-35B-A3B (Q4_K_XL, served by llama-server) on the Mac mini that already runs our benchmarks. The question is not how much it saves. It is what the model can actually do, first on short tests and then on the real inputs of jobs our system runs today. Every Qwen run happens on the Mac mini, costs no API fees, and changes nothing in production. We measure thinking on and thinking off separately. The same model also runs our nightly local-vs-cloud test.

Results so far, last updated November 9, 2026. One run per setting unless a row says otherwise.

So far

• Calling a tool, on 4 easy one-call tests run 5 times each: 20/20 with thinking off and 19/20 with thinking on. Harder, multi-step tool use is not tested yet.
• Summarising 12 real batches of alerts, ranked blind by one judge: both Qwen settings had a worse mean rank than both cloud models (3.25 and 3.33, against 1.08 and 2.33). Some of its errors called warnings normal.
• Triaging our review queue (197 real items in 17 batches): with thinking on, Qwen agreed with the verdicts Sonnet gave and the job applied on 168/197 (85%). Sonnet is the reference, not the truth, but our hand check mostly sided with it, so the job stays with Sonnet.
• Pulling facts out of 26 real emails (18 model deprecations, 8 look-alikes), graded on exact wording against an answer key: 25/26 emails perfect with thinking on, 24/26 with it off, and 25/26 for one Claude Opus 5.5 control run. Equal scores there hide a difference, explained in section 5. One night, not a trend.

1. Calling a tool

The tool tests from our Deep Suite: 4 short tests where the model must call the right tool with the right arguments, 5 runs each. These are test prompts, not inputs from a real job.

• With thinking on, there was one miss; it ran out of output tokens while still thinking.
• These are easy, one-call tests. Tool use over several steps, or inside a real job, is not measured yet.
• The frontier row is an approximate figure from a smaller grid (3 runs per test), so the rows are not a like-for-like race.

2. Summarising alerts

12 real batches of our own Telegram alerts, summarised by each model and ranked blind by one judge (Claude Opus). Each setting ran once per batch.

• Qwen with thinking off had the lowest median time of the four (2.0 s), and Qwen with thinking on the highest (29.8 s). Turning thinking on did not improve Qwen's mean rank (3.25 off, 3.33 on).
• Its errors were the kind that matter in alerts: it called warning heartbeats "normal" and "stable", and it said a session's "context was nearly full" when nothing said so.
• One judge, from the same model family as Claude Haiku, so it may favour Haiku. 12 batches is a small sample.

3. Triaging a review queue

Every logged batch of our daily article and email review (17 batches, 197 real items, 2026-10-18 to 2026-11-03), replayed on Qwen. We compared its verdicts with the ones Claude Sonnet gave and the job applied. Qwen's verdicts were never applied.

• With thinking off, 2 of 17 batches came back in a form the job refuses (one looped on its tools, one dropped an item). Its 117/191 (61%) still counts the items it returned in those batches; we have not scored the usable batches on their own.
• We checked the 29 thinking-on disagreements by hand against the job's written rules: Sonnet was right in about 20, Qwen in 1, and about 8 were arguable.
• Sonnet's verdicts are the reference, not the truth. Thinking on is slow here: a median of 98 s a batch.
• Verdict: this job stays with Sonnet.

4. What a labelling test taught us about our labels

Our chatbot labels each question with a type before it answers. We replayed the 100 most recent logged questions (2026-10-27 to 2026-11-02) through the live labelling prompt on Qwen, and compared each label with the one Claude Haiku 5.5 gave at the time. Our hourly shadow job had already put the same 100 logged questions to paid models.

• This is not a score for Qwen. Claude Haiku 4.5 and gpt-oss-120b also matched the live label only 36–39% of the time. Haiku 5.5, asked again, matched its own earlier label 82% of the time.
• The questions repeat: all 568 logged questions hold only 30 different ones, and 407 of them are "what is the highest rated model for X?". The 100 sampled rows hold 24 different questions; of the 24 asked more than once, live Haiku gave 7 more than one label.
• Qwen returned a valid label in 100% of its calls, and its two thinking-off runs gave the same label 87% of the time.
• What we take from it: the label set and the traffic need fixing before any model can be fairly scored on this job.

5. Pulling facts out of deprecation emails

26 real emails from our inbox: 18 model deprecations covering 48 model entries, plus 8 look-alike emails that are not deprecations. Each model had to tell them apart and pull out the model ids, shutdown dates and replacements. Grading is exact: an id or a replacement must be written the way the email writes it. This is one night (2026-11-04), not a trend, and Claude Opus 5.5 ran once as a control.

• Thinking on and Opus have the same counts, but those counts hide a difference. All three failed the same email: a notice retiring 8 models that names two possible replacements. Both Qwen settings said no replacement was named, which is a real miss. Opus gave both names joined by "or", which is right in meaning, but the email never writes that string, so exact grading failed it.
• Thinking off's other miss is one id: it wrote "k2 series" where the email says "K2 series". Exact grading counts that as a missing id and a wrong extra id, and it loses that entry's date and replacement too, although both were right in the reply. Ignoring letter case would accept the id.
• All three returned valid JSON on every email. Qwen's times are wall time on the Mac mini: thinking on took a median of 22.2 s an email, against 1.4 s with it off.
• An AI agent wrote the answer key from the email bodies, and Codex reviewed the key and the grader over 7 rounds. All three models ran a 2-email smoke test before the key's final edits, which came from that review. The key was then frozen before the scored run. The agent is Claude, the same family as the Opus control.
• Opus was a one-off control: $2.25 at list price, which is not our bill, so it is not added to any billed figure. Qwen paid no API fees.

6. A second local model: Qwen3.8 27B

On November 9, 2026 we ran Qwen3.8 27B (dense, UD-Q4_K_XL, licence Apache-2.0) on the same 64 GB Mac mini, against Qwen3.6-35B-A3B, the mixture of experts model in sections 1 to 5. One run per setting. We are keeping both: Qwen3.6-35B-A3B stays the default and the model in the nightly test.

Summarising alerts again, on 8 small real batches of 1 to 3 messages: six settings ranked blind by one judge (Claude Opus). This is the only test that ran both Qwen models on the same day.

• Qwen3.8 is much slower: 14.5–15 tokens a second against 50–67 for Qwen3.6. Its short tests took a median of 12.9 s, and 93.1 s at the 90th percentile. The two memory figures come from different work, so they are not a like-for-like comparison.
• With thinking off, Qwen3.8 got all 26 emails perfect and 48 of 48 replacements right, the only perfect run in our record so far. With thinking on it got 25 emails and 40 replacements.
• The email grader has changed since section 5's run. From November 7 it ignores letter case, so section 5's thinking-off run would now score 25, not 24. On November 9, 2026 the answer key began accepting "kimi-k2" for "K2 series". Under both changes, every Qwen3.6 night from November 4 to 9 scores 25 of 26 in both modes, and Qwen3.8's scores do not change.
• The tool row compares Qwen3.8 with Qwen3.6's run of November 8, 2026, the same 4 tests × 5 through the same harness. Section 1's run of November 3, 2026 also used 4 tests × 5, but we do not claim it used the same harness. Thinking on and off had different token budgets, and every thinking-on miss ran out of tokens while thinking.
• Haiku ranked first, and was best in 7 of 8 batches. Both Qwen3.8 settings ranked above both Qwen3.6 settings, and both ranked above Groq. 32 of 32 local summaries matched the required output format, which checks the shape of a summary, not whether it is right. The judge flagged false reassurance twice in Qwen3.6 with thinking on, and nowhere else.
• What this does not show: each Qwen3.8 figure, and each alert ranking, is one run per setting. These ranks cannot be compared with section 2's (another day, 12 other batches, other models), and the judge is from Haiku's own family. Qwen3.6's short-test, tool and email figures come from its earlier nights, not a rerun on November 9, 2026. No API fees is not free: the Mac mini and its electricity are left out.

The same model on short graded tests every night, next to paid cloud models: local model vs the cloud.