Experiment 1 · Browser automation

Playwright: MCP cuts token load — when the protocol shape fits.

Same browser, same fixtures, same model. The skill arm drives Chromium through playwright-cli; the MCP arm drives it through typed browser tools.

Run n5-v2 · N=5 (baseline N=2) · 6 tasks · 3 arms · 240 s wall budget · Claude Code 2.1.288, claude-sonnet-4-6, @playwright/cli 0.1.13, @playwright/mcp 0.0.75 · exact run data in experiments/playwright/runs/n5-v2

20/20

Tier 1 success for both skill and MCP.

1.92x

Tier 1 token ratio: skill / MCP.

10/10

Tier 2 success for both skill and MCP, recovery included.

1.84x

Tier 2 token ratio: skill / MCP.

Abstract browser automation diagram with CLI and MCP paths reaching the same browser

Both paths drive the same browser; only the tool surface differs.

First run (May 2026): this page now shows the n5-v2 run. The first run, n5, used looser isolation; see the first-run note for its numbers and why they differ.

The Task Suite

The fixture server renders per-trial state over HTTP, so the agent has to use the browser. The same seed produces the same page for every arm.

Task	What the agent must do	Skill	MCP
`tier1_login`	Sign in and save a dashboard screenshot.	5/5	5/5
`tier1_scrape`	Extract a seeded table into JSON.	5/5	5/5
`tier1_form`	Submit a form and capture the per-trial token.	5/5	5/5
`tier1_products`	Visit five product pages and write title/price JSON.	5/5	5/5
`tier2_checkout`	Find a marked product, add to cart, complete checkout.	5/5	5/5
`tier2_recovery`	Trigger an error, read the code, resubmit correctly.	5/5	5/5

The Cost Picture

Both browser surfaces completed every task cleanly, and MCP was cheaper on every one, 1.53x–2.07x. The form task is the outlier at 2.67x: the skill arm fills one field per fill call, while MCP fills the whole form with one batched browser_fill_form.

skill tokens mcp tokens

2.07x

scrape

1.54x

form

2.67x

products

1.53x

checkout

1.68x

recovery

2.06x

Widths are normalized to checkout skill tokens, the largest clean task at 266,073 total tokens. Averages are over trials that passed and stayed valid.

Why The Form Task Costs More

The data doesn't support a general mechanism, and the skill arm did not snapshot after every action. On tier1_form it made 25 fill calls but only 8 snapshot calls across five trials; MCP made 10 browser_snapshot calls. The form gap comes from per-field fill calls versus one batched browser_fill_form.

Skill form fill

playwright-cli fill for one field
Repeat for each remaining field
25 fill calls over five trials
Only 8 snapshot calls over five trials

MCP form fill

browser_fill_form with every field
One batched call per trial
10 browser_snapshot calls over five trials

Recovery Now Converges

Lower cost and convergence went together in this run. On tier2_recovery, the first run's weak spot for MCP, both arms passed 5/5 and stayed in surface.

Conservative read: MCP solved the recovery flow in all five trials, averaging 115,117 tokens and 16.2 turns against the skill arm's 236,852 tokens and 20.6 turns (2.06x). The first-run result for this task is in the first-run note.

Tier	Skill success	MCP success	Skill avg tokens	MCP avg tokens	Skill/MCP
Tier 1	100%	100%	179,932	93,546	1.92x
Tier 2	100%	100%	251,463	136,538	1.84x

Tier averages as the harness report computes them. Every skill and MCP Playwright trial passed and stayed in surface.

Takeaway

For browser automation, MCP was cheaper on every Playwright task, with the same 5/5 success on all six, recovery included. The multiple depends on the harness setup, 1.53x–2.07x here with form at 2.67x; the direction (MCP cheaper) held in both runs. Success and cost still have to be read together.

Read the GitHub visual brief or the cross-experiment visual guide.