Tooling study · visual version
CLI skill or MCP server: choose by primitive fit, not familiarity.
The same agent, prompt, model, and task can behave very differently depending on whether it reaches through a CLI skill or a typed MCP server.
repo_inventory and file_patch_pr all passed, every one by stepping outside the intended gh-only surface.
The Experiment In One Screen
Every trial changes one thing: the tool surface. The task, seed, prompt, model, and wall-clock budget stay fixed. The harness then scores both the end state and whether the agent stayed inside the surface it was supposed to use.
gh or playwright-cli.
How isolation works in this run: each arm gets a positive --tools list; file tools are denied under ~ and the repo; the GitHub skill arm's Bash runs inside Claude Code's OS sandbox (keychain and $HOME blocked); each trial gets an empty GH_CONFIG_DIR, and inherited GH_/GITHUB_ variables are removed. GitHub Tier 1 uses a read-only agent token, Tier 2 a separate read-write one (no admin); both are owned by the same GitHub user. The first run had looser isolation; see First run (May 2026).
Finding 1: When Primitives Match, MCP Costs Less
In the clean comparisons where both arms passed and stayed valid, MCP used fewer tokens on every task in both experiments: skill/MCP was 1.5x-2.1x everywhere except Playwright form at 2.67x. Turn counts are within about 3 on the GitHub tasks but not on all Playwright tasks: form took 18.8 skill vs 13.6 MCP turns, products 17.4 vs 21.4. The form gap comes from per-field fill calls versus one batched browser_fill_form.
Widths are normalized to the largest clean-token task, Playwright checkout at 266,073 skill tokens. Ratios use total tokens, averaged over trials that passed and stayed valid. GitHub repo_inventory and file_patch_pr have no clean skill trials, so they have no bars.
Finding 2: Success Alone Tells The Wrong Story
On two GitHub tasks the skill arm got the right answer in every trial by using shell behavior outside the intended gh surface. The classifier is what makes that visible.
| GitHub skill task | Raw success | Valid surface |
|---|---|---|
repo_inventory |
5/5 | 0/5 |
issue_triage |
5/5 | 5/5 |
file_patch_pr |
5/5 | 0/5 |
Success-only view: skill looks fine. Validity-aware view: on repo_inventory all five trials decoded with gh api … | base64 -d; on file_patch_pr all five used base64 -d on read, plus shell variable assignment and $(…) command substitution to build the PUT body. issue_triage is now a valid comparison: both arms passed 5/5 in surface, and skill used 1.67x MCP's tokens.
Finding 3: MCP Was Clean On Every Task
MCP passed and stayed in surface 5/5 on all 13 tasks, 6 Playwright and 7 GitHub. That includes the two cases the first run couldn't settle: GitHub issue_triage, now run with a read-only agent token that can read issues, and Playwright tier2_recovery.
| Case | MCP | Skill | Skill/MCP tokens |
|---|---|---|---|
GitHub issue_triage |
5/5 | 5/5 | 1.67x |
Playwright tier2_recovery |
5/5 | 5/5 | 2.06x |
| GitHub workflow status | 5/5 | 5/5 | 1.52x |
Counts are trials that passed and stayed in surface, out of 5.
Finding 4: Prompting Can Shift An Affordance Gap, Not Erase It
A directed prompt told the skill arm exactly how to stay in surface for file patching: use Write files and pass them to gh api --input. More trials stayed in surface, but none of those finished, and no trial both passed and stayed valid.
| Metric | Original | Directed |
|---|---|---|
| Pass success | 5/5 | 2/5 |
| Valid surface | 0/5 | 3/5 |
| Pass and valid | 0/5 | 0/5 |
| Avg tokens, passing | 153,022 | 401,861 |
Both passing directed trials escaped: one with $(…) in gh pr create, one with base64 -d. They averaged 2.63x the original's tokens. The three valid trials timed out at 240 s after 10–13 turns; their transcripts end in extended thinking with no further tool calls.
MCP file patch PR
- Get file contents
- Create branch
- Create or update file
- Create pull request
Skill file patch PR
- Read file through raw REST, decoded with
base64 -d - Get main HEAD SHA
- Create branch through raw REST
- Build the PUT body with shell variables and
$(…) - Write file through raw REST
- Create pull request
How To Choose
The practical rule: pick the surface whose primitives already match the workflow. If neither does, expect shell escapes on the CLI side, and measure both before standardizing.
Pick or build a CLI skill when
gh issue create, gh pr diff. The agent does not need shell glue.
Pick or build MCP when
First Run (May 2026)
The first full run, n5, is kept as history. Its numbers differ from the ones above, and the reasons are worth keeping next to them.
What n5 showed, and why it differs
- Setup: Claude Code 2.1.142–2.1.143 with looser isolation.
--allowed-toolsdid not restrict tools, and the runner's env scrub didn't apply. The results have been re-classified with the current classifier. issue_triagewas invalid: the agent token was mis-scoped and lacked Issues read, and the skill arm's passes used escalated credentials.- Figures: Playwright Tier 1 1.64x and Tier 2 1.86x skill/MCP (1.52x on completed trials); Playwright clean tasks 1.29x–1.42x, form 2.51x; MCP
tier2_recovery2/5; GitHub clean tasks 1.13x–1.38x; directed prompt 1/5 pass and valid. - Why the ratios rose in n5-v2 (an observation, not a proven cause): MCP got cheaper by more than skill did (e.g.
pr_diff_answerMCP 55,534 → 38,840 tokens, skill 73,146 → 66,564). At the same time the fixed first-turn context per arm shrank: MCP ~12.3k → ~8.3k (Playwright) and ~8.8k (GitHub), skill ~16.6k → ~13.0k and ~15.1k, baseline ~12.1k → ~9.6k. The skill arm keeps a larger fixed context, the loaded skill text. - Two things changed together: the tool restriction (
--tools, from ~22–24 built-in tools to 2–5) and the Claude Code upgrade from 2.1.143 to 2.1.288. This run can't separate them. The multiple depends on the harness setup; the direction (MCP cheaper) held in both runs.
What This Version Leaves Out
This page is the short version. The trial JSON, Bash classifier, runner flags, and quick start commands live in the repo under harness/src/ and experiments/.
Keep three caveats attached to the visuals: N=5 (N=2 for baseline, which passed 0/26) is directional rather than benchmark-grade; the validity classifier is a witness, not a sandbox; MCP tool catalogs change between server versions; verify names against the version you're running.
Continue with the Playwright visual brief or the GitHub visual brief.