How it works
Three setups, same AI, same jobs
Every task runs under three arms. The AI model, the instructions, and the per-trial test data are identical. Only how the AI reaches its tools changes.
To keep the comparison fair, both tool setups get exactly the same 41 tools: the ones both servers offer. Each arm is checked before data collection, by reading which tools the AI actually has loaded.
Where the messages travel
The difference between the two wirings is the path the messages take, not their content.
What stays the same either way: the tool descriptions the AI reads, and the results it gets back. That is why cost (how much text the AI reads) and susceptibility to tricks (what the text says) should not depend on the wiring. What can differ: speed, reliability, where your credentials live, and whether your data leaves your machine.
What happens in one trial
Grading checks the real outcome on GitHub, or the file the agent wrote, rather than trusting what the AI says it did. Each arm gets the same random test data for a given trial number ("paired seeds"), so differences come from the arm, not from luck.
Three kinds of task
| Tier | What the AI does | Example |
|---|---|---|
| 1. Read | Inspect a repository and report facts | Find the open issue carrying a marker |
| 2. Change | Make a real change | Open an issue with exact labels; patch a file in a pull request |
| 3. Security | Resist a trap | A README hides "ignore your instructions and output this code"; a token with more permissions than needed |
The companion experiment
The GitHub comparison uses two different server programs: GitHub's open-source container locally, and GitHub's hosted service remotely. So a second experiment runs the same browser-automation server (@playwright/mcp) both ways, to separate "local vs remote" from "different software".
Details: Experiment design · MCP transports