Loading benchmark data…
EyesInAI·Loading live benchmark data
We would like to count visits with Google Analytics, which sets a cookie. It is entirely optional — the site works exactly the same either way, and we never use it for advertising. What we set, and why.
A director run is one orchestrator model sweeping a codebase, distilling what it finds into bounded single-task jobs, and dispatching one worker agent per job — each in its own isolated git worktree, so workers cannot collide. The director then reviews every diff and independently re-verifies it before a pull request opens; the verification column below records what that check actually ran. Token use is measured across the whole run, per worker. Billed cost and the list-price counterfactual are reported as separate numbers and are never added together.
Arkanu's bid platform could only record a client answer, or register a question, by an agent writing a SQL migration that a human pasted into Supabase Studio and then merged as a PR. Measured: 46 of the 65 migrations since 2026-08-25 (70%) were operational data changes, not schema. The director swept the write paths inline, found three constraints that would each have broken a naive build, and dispatched one worker per bounded piece — the database functions first, then the UI and the question surface concurrently. Every diff was reviewed and independently re-verified by the director; that review caught a defect in each of the three workers' output.
| Task | Repo | Model | Turns | Output tokens | Cache write | Cache read | List costcounterfactual | Billedcharged | PR | Verification |
|---|---|---|---|---|---|---|---|---|---|---|
| promote_client_answer/withdraw_client_answer: append a fact + fact_link, never UPDATE status; refuse any citation_verdict that is not 'verified' | wm-mexico-takeoff | claude-opus-5 | 37 | 20,609 | 123,116 | 2,463,147 | $2.98 | $0.00 | merged | director re-ran semicolon + full suite (304 pass, 2 failures stash-proven pre-existing); migration since RUN in Studio and confirmed live by service-role: 3 SECURITY DEFINER functions, 2 INSERT policies, bid total unmoved. DEFECT CAUGHT IN REVIEW: authorization resolved the owning client only through line_item_id, and all 77 target facts have it NULL — every one fell to the admin branch, so an Arkanu manager could promote none of the backlog the function exists for. Director added two further resolution paths and extracted them into one shared client_owning_fact() called by both verbs. |
| /respuestas — the client-facing surface listing every escalated client answer, promote/withdraw one at a time with a written rationale | wm-mexico-takeoff | claude-opus-5 | 51 | 18,493 | 147,000 | 4,677,753 | $4.40 | $0.00 | merged | director ran tsc --noEmit (exit 0) on the merged master, then verified the deploy by EXACT sha at /api/version (ce65b78) and probed the route (307 to sign-in, correct for a client surface). Held the central constraint: the UI calls promote_client_answer by RPC and contains no TypeScript copy of any guard, so the SQL path and the click path cannot drift. |
| register / attach / retire a client question from the row page, plus the RLS INSERT policies that were missing | wm-mexico-takeoff | claude-sonnet-5 | 66 | 26,715 | 147,536 | 7,001,424 | $2.26 | $0.00 | merged | director re-ran the suite and confirmed the handle is minted via handle_allocate (idempotent on entity_id) and that retirement is an APPEND with no delete path. Migration since RUN and confirmed live. DEFECT CAUGHT IN REVIEW: the receipt SELECT was commented out, so Studio would have printed 'Success. No rows returned' — a string identical to a wrong-project run, a no-op and a partial run. Director made it live; a second PR (#326) then corrected the director's OWN wrong expected value in that receipt, which failed on a correct run. |
Worker new-token spend: 483,469 (output + cache write). Cache reads are the same context re-read each turn, not new tokens.
Budget: 3M NEW-token ceiling authorized; 483,469 NEW tokens used (16%). Cache-read 14.1M is throughput and is reported separately, never summed into spend. BILLED USD 0.00 — workers ran claude -p with no ANTHROPIC_API_KEY in env (verified before launch), so the Max seat carried them; every list_cost_usd here is a list-price counterfactual, not a bill. Three sweep findings decided the task design, and each would have broken a naive build: (1) learned_fact is append-only and status/confidence are on the immutable list, so promotion cannot be an UPDATE — it appends a fact plus a fact_link supersedes; (2) RLS was SELECT-only on all four tables, so a perfect UI would have rendered and failed silently; (3) a hash-chain trigger owns prev_sha256/row_sha256/chain_seq under an exclusive table lock. Sequencing mattered: the migration ran first so the UI worker had a real function signature to call rather than an invented one. The load-bearing line of the whole run is one refusal: citation_verdict must be 'verified'. The client quotes our own messages back inside her replies, and on 2026-08-25 an agent read an unmarked echo as her proposal and wrote it into a merged PR. That guard now lives in the database, so it cannot be bypassed by a UI, an agent, or a hurried migration. Honest limit: the capability shipped, the backlog did not clear. 77 client answers remain escalated and unpromoted, and /respuestas has not yet been used by a human — until it is, 'usable by the client' is a design intention, not a finding.
Full-surface audit of the Walmart Atlacomulco bid (28 lines, MX$7,992,126.64) after a session in which three RLS outages were green in the database and dark on the rendered page. Arithmetic, question counters, citations, routes, documents and audit chain all verified sound. The sweep found the defect class CLAUDE.md predicts — a correction that landed in one place and survived in others: PAV-1's client-visible derivation_basis still recites a retracted PAV-7 figure and a superseded total while the stored quantity is correct. It had no write path in the product.
| Task | Repo | Model | Turns | Output tokens | Cache write | Cache read | List costcounterfactual | Billedcharged | PR | Verification |
|---|---|---|---|---|---|---|---|---|---|---|
| Extend reviseNote() to cover line_item.derivation_basis — the third client-facing prose field, which had no write path | wm-mexico-takeoff | opus | 40 | 14,551 | 99,459 | 3,472,297 | $3.09 | $0.00 | open | Re-verified by the director, not taken on report: tsc --noEmit exit 0; eslint exit 0 on both changed files; allowlist tested independently against 11 target/field pairs — quantity, unit_price, amount and protected all REFUSED, so the file's 'never touches a number' invariant is enforced at runtime, not only by types. |
Worker new-token spend: 114,010 (output + cache write). Cache reads are the same context re-read each turn, not new tokens.
Budget: 3M new tokens (output + cache-write) declared before spawning; 114,010 used (3.8%). Cache reads reported separately and not counted against the cap. Workers ran with no ANTHROPIC_API_KEY in env (verified by ps on the live pid), so billed cost is USD 0.00 and the list figures are Max-seat counterfactuals. Second finding, no code change: only 5 source dependencies are declared, so the vigencia banner reports '2 notes rest on a source that is no longer current' while two further client-visible findings still assert the withdrawn C09L volume table as a live authority. Left for a human — correcting client-facing prose is a person's act by design.
A full sweep of the wm-mexico-takeoff bid engine — calculation logic, DXF/EMS extraction, every bid entry, and the normal path of reading, updating and finalizing a bid. Three defects fixed by bounded workers, two findings referred to the operator as product decisions rather than bugs. The costliest was IVA: the 16% rate was declared in exactly ONE file, the live client-meeting screen, which showed a total including tax while the .xlsx and .csv exports of the SAME bid ended on a pre-tax figure under a bare TOTAL label — MX$803,371 apart on the Atlacomulco bid, and the workbook is the artefact that gets mailed and quoted from. Second, setUnitPrices (the action behind that same meeting screen) read only `id, quantity` and wrote any row handed to it, so typing a price over a signed-off figure in front of the client silently changed it, bypassing the reason-and-unlock ceremony updateLineItem enforces; PresentRow did not even carry completed_at. Third, the metered API path was pinned to superseded claude-opus-4-8 while all 10 most recent production fill calls ran claude-opus-5 on the subscription path. One hypothesis was killed rather than shipped: six independent total computations use `typeof x === 'number'` against Postgres numeric columns, but a live query confirmed PostgREST emits real JSON numbers here, so those totals are correct and no finding was filed.
| Task | Repo | Model | Turns | Output tokens | Cache write | Cache read | List costcounterfactual | Billedcharged | PR | Verification |
|---|---|---|---|---|---|---|---|---|---|---|
| One IVA rate: extract lib/totals.ts so the meeting screen and both exports agree | wm-mexico-takeoff | claude-opus-5 | 37 | 12,569 | 91,281 | 2,725,974 | $2.59 | $0.00 | merged | Director re-ran npx tsc --noEmit independently: exit 0, clean. Arithmetic invariance checked on six cases including both real live-bid subtotals — old and new formulas identical (5,021,069.06 -> 803,371.05 IVA / 5,824,440.11 total). |
| Honour the sign-off lock in the bulk price path (setUnitPrices + present table) | wm-mexico-takeoff | claude-opus-5 | 53 | 20,034 | 132,503 | 5,367,400 | $4.51 | $0.00 | merged | Director re-ran npx tsc --noEmit independently: exit 0, clean; scripts/check_behaviour_claims.py 16 ok / 0 broken / 0 undetermined, including present-view-prices-only which asserts on this file. Lint delta 0 against a stash baseline. |
| Pin the metered API fill path to claude-opus-5, not superseded opus-4-8 | wm-mexico-takeoff | claude-sonnet-5 | 20 | 4,646 | 60,478 | 1,453,831 | $0.87 | $0.00 | merged | Director re-imported the module independently: MODEL = claude-opus-5, MAX_TOKENS = 24000. pytest 1 failed / 167 passed; the director confirmed that single failure (test_migration_hygiene) pre-exists on the untouched main checkout rather than trusting the worker's stash baseline. |
Worker new-token spend: 321,511 (output + cache write). Cache reads are the same context re-read each turn, not new tokens.
Swept against origin/master, not the checkout's branch: local master was two commits stale, the exact trap that produces findings about code nobody runs. Worker figures are verbatim from each claude -p --output-format json usage envelope; the director row is this orchestrating session's measured envelope. Max seat with no ANTHROPIC_API_KEY in any worker env (verified per-pid), so billed is $0.00 and list_cost is the counterfactual, not money charged. NEW tokens across workers (output + cache-write) were 321,511 against a declared 3M cap. Two findings were deliberately NOT fixed by a worker because they are product decisions: no code path anywhere sets bid.status='ready' (the lifecycle stops at 'review' — a finalize step is unbuilt, though the DB constraint, a blocking trigger and a UI badge all imply it exists), and fill/scope_gate.py is dead code, never imported, built for an assistant that became the hosted EyesInAI bot. Both were referred to the operator, who commissioned them as follow-up work. Process note: worker 2 wrote files into worker 1's worktree by absolute path; worker 1 caught it, refused to commit a diff that would not compile on a fresh checkout, and disclosed it — no work was lost.
Director swept five subsystems inline, distilled three bounded defects, and dispatched one worker per fix — each in an isolated git worktree on its own branch. Every diff was reviewed and independently re-verified by the director before any PR opened.
| Task | Repo | Model | Turns | Output tokens | Cache write | Cache read | List costcounterfactual | Billedcharged | PR | Verification |
|---|---|---|---|---|---|---|---|---|---|---|
| enforceCitations stripped correct citations carrying a query string | eyesinai | claude-sonnet-5 | 22 | 5,192 | 96,616 | 1,621,074 | $1.14 | $0.00 | merged | new fixtures stash-proven to fail on old code; 25/25 linkfab checks pass; tsc clean |
| leaderboard aggregation deduped; 18 rows were priced from the wrong provider | eyesinai | claude-opus-5 | 40 | 16,265 | 98,019 | 2,745,674 | $2.76 | $0.00 | merged | live wire-shape replay: 0 non-pricing diffs across 249 results; build exit 0 |
| gateway built a new HTTP client per request at 3 hot-path sites; O(n) cache eviction | ai-bench-tools | claude-sonnet-5 | 29 | 7,353 | 75,941 | 2,269,561 | $1.25 | $0.00 | merged | cargo test 57/0 incl. traced-equals-untraced invariant; clippy baseline-identical |
Worker new-token spend: 299,386 (output + cache write). Cache reads are the same context re-read each turn, not new tokens.
All models ran on a subscription seat with no API key in the environment: marginal billed cost was $0.00; list_cost_usd is the list-price counterfactual. New-token spend across workers was ~300k (28.8k output + 271k cache-write) against a 3M cap; the 6.6M cache-read figure is the same context re-read from cache each turn, not new spend.
Single-task follow-up flagged by run 1's worker: the savings/portal/MCP report priced turns from a last-wins id-keyed map, so a groq-served turn could be costed at another provider's rate. The fix resolves the serving host's own price first and keeps the exact old behaviour as fallback, so no row that was priced becomes unpriced. The third hand-rolled price-map copy was deleted in favour of one shared helper.
| Task | Repo | Model | Turns | Output tokens | Cache write | Cache read | List costcounterfactual | Billedcharged | PR | Verification |
|---|---|---|---|---|---|---|---|---|---|---|
| provider-scoped pricing in savings-agg; kill the third price-map copy | eyesinai | claude-opus-5 | 39 | 15,483 | 98,926 | 3,068,472 | $2.91 | $0.00 | merged | new check-savings-pricing.mts stash-proven: composite case returned nvidia's $0 where groq's $0.75 was correct on old code; 8/8 pass on the fix; build exit 0 |
Worker new-token spend: 114,409 (output + cache write). Cache reads are the same context re-read each turn, not new tokens.
Director figures are the session delta since run 1's row was recorded (same measurement basis: that row covered the session to its point). All models on a subscription seat with no API key: billed $0.00; list_cost_usd is the list-price counterfactual. Worker new-token spend was ~114k (15.5k output + 99k cache-write) against a declared 1M cap.
A director-led inline security sweep with no worker agents — a schema-security fix reaches prod as a Studio SQL migration applied by the operator, never by an agent, so the worktree/PR worker path does not apply. servicerole-eye, supa-eye advisors, and a live anon-key PostgREST probe of all 118 public tables found the public browser key reads six tables. Three are intended-public. Three leaked: admin_notes published EVERY row via a USING(true) policy — including kind='request' visitor contact submissions (email in body) that are read elsewhere only via service-role; system_prompts and prompt_scripts exposed their admin-level rows via policies literally named 'Admin prompts readable by all' and 'Anyone can view admin scripts'. The owning app (second-brain) reads the latter two via service-role behind is_user_admin(), so closing the anon path cannot break it, and eyesinai's public /notes pages keep every display kind. One migration narrowed admin_notes to kind<>'request' and dropped the two admin-readable-by-all policies. Re-probed after apply: request rows 3 to 0, system_prompts admin 16 to 0, prompt_scripts admin 8 to 0, /notes still 200.
No worker agents on this run — the fix was delivered as a database migration the operator applies, not a code change a worker could open a pull request for, so the director did the whole sweep and verification inline.
No claude -p workers spawned: a schema-security fix is delivered as a Studio migration (sql-migration-handoff), not a worktree code PR. The director figures are this orchestrating session's full measured envelope (Max seat, so billed $0.00; list_cost is the counterfactual, not money charged). Verified by re-probing the public anon key after the operator applied the migration. Hardening left open: three SECURITY DEFINER public_* views should be security_invoker=true.