Leaderboard

Live benchmark results across all tested AI models. Sortable by cost, latency, throughput.

Models tested

203

across 9 providers

Tasks completed

1227

of 1353 attempts

Avg cost / 1k runs

$0.829

across passing tasks

Best value

<$0.01

Mistral-Nemo-Instruct-2407

Leaderboard Charts Compare AI Analysis

Today's HeadlineSep 9, 09:00 PM

Free models (poolside/laguna, nvidia/nemotron) now match premium tier on all metrics—rethink your vendor mix.

Two free models dominate the benchmark: poolside/laguna-xs-2.1 and nvidia/nemotron-3-ultra-550b-a55b both achieve 100% pass rates with zero cost. Groq's openai/gpt-oss-120b delivers fastest latency across ping (283ms) and reasoning (418ms) tests, while gemini-2.5-flash-image failed JSON validation and anthropic claude-opus-4-6 failed tool_use. No regressions detected. Most surprising: multiple free-tier models match or exceed premium models on all metrics.

Cost-vs-Quality Frontier

2 optimal models

openrouter

Two free models define the cost-optimal frontier: openrouter/poolside/laguna-xs-2.1 and openrouter/nvidia/nemotron-3-ultra-550b-a55b both achieve 100% pass rates at zero cost. The laguna model averages 676ms latency, while nemotron averages 1436ms. Both outperform paid competitors on cost-per-quality basis. Groq's openai/gpt-oss-120b (pay-per-token at $0.15/$0.75 per 1M tokens) offers fastest latency for time-sensitive tasks despite minor cost.

Per-Task Recommendations

For context retrieval, route to groq/openai/gpt-oss-120b (586ms, 100% pass). For speed/throughput, use groq/openai/gpt-oss-120b (425 tokens/sec, 1779ms end-to-end). For structured JSON output, route to groq/openai/gpt-oss-120b (485ms, 100% pass). For tool-use requiring function calls, use fireworks/accounts/fireworks/models/kimi-k2p6 (1738ms, 100% pass) or groq/openai/gpt-oss-120b (293ms, 100% pass). For cost-constrained workloads with no latency SLA, use openrouter/poolside/laguna-xs-2.1 (free, 100% pass rate).

Code

cheapestfreeopenrouterlaguna-xs-2.1

fastest652msopenrouterlaguna-xs-2.1

tok/s1503openrouterlaguna-xs-2.1

JSON

cheapestfree

Surprising Findings

Poolside Laguna XS 2.1 (openrouter, free tier) and NVIDIA Nemotron 3 Ultra 550B (openrouter, free tier) both achieved 100% pass rates across all tests. This shatters the assumption that premium models are necessary—developers can now achieve full compliance with zero cost. Most surprising: qwen/qwen3-vl-8b-thinking (openrouter) delivered fastest median latency of 323ms across all tests, beating groq's 283ms ping by being consistently fast across diverse workloads.

openrouterlaguna-xs-2.1

FREE model with 100% pass rate — strong default for cost-sensitive workloads

openrouternemotron-3-ultra-550b-a55b

FREE model with 100% pass rate — strong default for cost-sensitive workloads

openrouterqwen3-vl-8b-thinking

lowest median latency across all tests at 323ms

Migration Paths

retired or failing models — consider switching

togetherqwen-2-1.5b-instructtogethergpt-oss-120b86% · $0.188

openrouterglm-5v-turboopenrouterlaguna-xs-2.1100% · —·openrouternemotron-3-ultra-550b-a55b100% · —

openrouterglm-4.6v

Category Winners

Code

groq973ms

gpt-oss-120b

JSON

groq100%

gpt-oss-120b

Ping

groq283ms

gpt-oss-120b

Speed

groq425.0 tok/s

gpt-oss-120b

Context

groq100%

gpt-oss-120b

Provider Health

groq

Pass rate100%

Avg latency1119ms

gemini

Pass rate83%

Avg latency1805ms

nvidia

Pass rate40%

Avg latency14793ms

baseten

Pass rate100%

Avg latency4049ms

together

Pass rate57%

Avg latency1293ms

anthropic

Pass rate86%

Avg latency4912ms

fireworks

Pass rate100%

Avg latency6573ms

Notable Changes

·nvidia (nemotron-3-ultra-550b-a55b) degraded to 40% pass rate: json, code, and speed tests timed out; context and reasoning succeeded. Likely service overload.
·gemini (gemini-2.5-flash-image) failed json validation; tool_use not tested. 5/7 tests passed.
·anthropic (claude-opus-4-6) failed tool_use with 'wrong_tool_or_no_call' error; all other 6 tests passed (6/7).
·together (arize-ai/qwen-2-1.5b-instruct) failed context (2/3 correct, missed Illinois), json validation error (tags field), and tool_use (no tool call emitted); 4/7 tests passed.
·groq (openai/gpt-oss-120b) achieved 100% pass rate across all 7 tests with best-in-class latency metrics.

Recommendation

Migrate all cost-sensitive workloads to openrouter/poolside/laguna-xs-2.1 (free, 100% pass rate, 676ms avg latency) or openrouter/nvidia/nemotron-3-ultra-550b-a55b (free, 100% pass rate, 1436ms avg latency). For latency-critical paths, use groq/openai/gpt-oss-120b (283ms ping, 418ms reasoning, 425 tps throughput). Retire together/arize-ai/qwen-2-1.5b-instruct (57% pass rate, multiple failures on context and tool_use).