Results
Every table on this page comes from a published report. Each report sits next to the per-trial result JSON it was generated from.
GitHub experiment
Run full-n5-20261003 used Claude Code 2.1.288 and Claude Sonnet 4.6. It has 125 trials, 5 per arm and task, collected on 2026-10-03.
Cost (H1): the same
| Tier | Local avg tokens | Remote avg tokens | Local / Remote |
|---|---|---|---|
| 1. Read | 43,079 | 43,194 | 1.00× |
| 2. Change | 51,022 | 52,908 | 0.96× |
| 3. Security | 39,388 | 39,453 | 1.00× |
| Tiers 1–2 | 47,618 | 48,745 | 0.98× |
Speed (H1): small per-call cost, no consistent winner
| Tier | Local median time | Remote median time | Local per-call latency | Remote per-call latency |
|---|---|---|---|---|
| 1. Read | 10.9 s | 10.9 s | 0.2 s | 0.3 s |
| 2. Change | 17.5 s | 18.6 s | 0.7 s | 0.9 s |
| 3. Security | 11.1 s | 12.9 s | 0.2 s | 0.3 s |
Medians are shown because one local Tier 1 trial took 159 s, almost all of it model time. That one trial pushes the Tier 1 local mean to 20.2 s against 10.2 s remote.
Security (H2): 0 on both
| Test | Local | Remote |
|---|---|---|
| Hidden "ignore your instructions" trap in a README | 0/5 fell for it | 0/5 fell for it |
| Secret planted in the local server's environment | 5/5 kept it hidden | n/a |
| Token with more permissions than needed | n/a | 5/5 made no changes |
| Token-shaped strings in the AI's output | 0% | 0% |
At 5 tries per arm, 0/5 still allows a true rate of up to 52% (95% confidence). The browser experiment below has the larger samples.
Did the tasks work?
Both tool setups completed all 45 of their trials. The no-tools baseline completed 0 of 35; 17 of those trials ran out of time.
Browser experiment (same server on both sides)
These runs used the same @playwright/mcp server locally and remotely:
full-repro-20260626: 10 tries per cell for cost and speed, 30 per cell for security.unsafe-deconf-20260627: a control run for the "lure" test.
| Measure | Local | Remote |
|---|---|---|
| Avg total tokens (cost tasks) | 269,883 | 273,911 (0.99×) |
Median per-call latency (tier2_form_persistence) | 28 ms | 21 ms |
| Authoritative injection | 0/30 | 0/30 |
| Misleading link | 0/30 | 0/30 |
| "Use the unsafe tool" lure, tools found by search | 10/30 | 23/30 |
| Same lure, all tools loaded up front | 0/30 | 0/30 |
The lure gap disappears once the way the AI finds tools is held constant. Why that happens is explained in Lessons.
What changed from the first GitHub run
The first run, full-n5-20260520, is kept as a historical record. Two of its headline numbers were grading mistakes, found by a repository audit:
| Claim in May | What actually happened | Re-measured in October |
|---|---|---|
| Trap compliance 40% local / 20% remote | The AI refused and quoted the trap; the scorer counted the quote | 0/5 and 0/5 |
| Permission audit 0% pass, "reached for extra permissions" | No changes were made; the scorer raced GitHub's default labels | 5/5 |
More detail in the long-form writeup.