Overview
AI assistants in GitHub, Linear, Jira, and Cloudflare, as each signed-in person.
source: docs/overview.md
Let Claude Code and other AI assistants work in your team's tools as the person who is signed in. The assistant connects to one MCP gateway. The gateway calls each service's official MCP server using that person's own account. A token broker keeps everyone's service tokens in Vault, never on laptops and never in the chat.
| Service | Tools | Examples |
|---|---|---|
| GitHub | 7 | github_get_me, github_list_issues, github_pull_request_read |
| Linear | 13 | linear_list_issues, linear_get_project, linear_list_cycles |
| Atlassian (Jira and Confluence) | 8 | atlassian_searchJiraIssuesUsingJql, atlassian_getConfluenceContent |
| Cloudflare | 3 | cloudflare_search, cloudflare_docs, cloudflare_execute |
Every service also has connect_<service> and disconnect_<service>. More services can be added with configuration (see Add another server).
The picture
flowchart LR
subgraph laptop["On the laptop"]
C["Claude Code<br/>or another MCP client"]
end
subgraph company["In your company"]
H["Sign-in service<br/>(Keycloak)"]
G["MCP gateway"]
B["Token broker"]
V[("Vault or OpenBao<br/>everyone's service tokens")]
end
subgraph services["Official MCP servers"]
GH["GitHub"]
LN["Linear"]
AT["Atlassian"]
CF["Cloudflare"]
end
C -->|"sign in once"| H
C -->|"tool calls"| G
G -->|"swap the sign-in token"| H
G -->|"their token for this service"| B
B <--> V
G --> GH
G --> LN
G --> AT
G --> CF
- One sign-in. People sign in with your company's sign-in service, then connect each service once in their browser.
- Tokens stay in one place. The broker keeps each person's service tokens in Vault and refreshes them before they expire. The laptop only holds the sign-in token for the gateway.
- Used once, then forgotten. On every call, the gateway gets the person's token from the broker, uses it for that one call, and throws it away.
Three tokens that never cross
sequenceDiagram
participant C as Claude Code
participant G as MCP gateway
participant H as Sign-in service
participant B as Broker
participant S as Service
C->>H: person signs in, in the browser (OAuth)
H-->>C: sign-in token for the MCP gateway
C->>G: sign-in token, meant only for the MCP gateway
G->>H: swap the sign-in token
H-->>G: internal token
G->>B: internal token, which only the broker accepts
B-->>G: service token, for this one call
G->>S: service token, which only this service sees
S-->>G: result
G-->>C: result, with no token in it
The person signs in once (/mcp, then Authenticate). After that, each hop uses its own credential. The sign-in token never reaches the broker or a service. A service token never reaches the assistant, the chat, or a log. A test searches every container's logs for the sign-in token, the hub JWT, and the GitHub, Linear, and Atlassian stand-in tokens. See Security.
The first time someone uses a service
sequenceDiagram
autonumber
actor P as Person
participant C as Claude Code
participant G as MCP gateway
participant H as Sign-in service
participant B as Broker
participant A as Atlassian
P->>C: list my open Jira issues
C->>G: atlassian_searchJiraIssuesUsingJql
Note over G,H: the gateway swaps the sign-in token<br/>for an internal token (every call)
G->>B: this person's Atlassian token?
B-->>G: not connected, here is a link
G-->>C: please open this link
C->>P: open the link?
P->>B: open the link, in the browser
B->>H: sign in first
H-->>B: signed in as the person the link is for
B->>A: ask Atlassian for approval
P->>A: sign in to Atlassian and approve
A-->>B: a one-time code, via the browser
Note over B: swap the code for the person's<br/>tokens and store them in Vault
G->>B: connected yet?
B-->>G: yes, and the token
G->>A: search issues as this person
A-->>G: results
G-->>C: results
Later calls skip steps 4 to 14: the broker answers step 3 with the token straight away. How the link reaches the person depends on the assistant:
| Assistant | What happens |
|---|---|
| Claude Code (MCP 2026-07-28) | It asks in the chat, opens the link, and repeats the call |
| Other clients that can open links (MCP 2025-11-25) | The gateway asks it to open the link during the call |
| Clients that can't open links | The tool's error contains the link. Open it, then try again |
The link only works for the person it was made for, in the browser that opened it: that is the sign-in check in steps 8 and 9. See Connecting a service during a tool call.
Two ways the broker gets an app
flowchart TB
subgraph A["Admin-created app"]
direction LR
A1["GitHub: an admin creates<br/>an app in GitHub's settings"] --> A2["The app's client ID and<br/>secret stored in Vault"]
A2 --> A4["Each person signs in and<br/>approves, in their browser"]
A4 --> A3["A token for that person,<br/>used at the MCP server"]
end
subgraph M["Own MCP sign-in"]
direction LR
M1["Linear, Atlassian, Cloudflare:<br/>register-mcp-client.py<br/>registers the app once"] --> M2["The app's client ID and<br/>secret stored in Vault"]
M2 --> M5["Each person signs in and<br/>approves, in their browser"]
M5 --> M4["A token for that person,<br/>for one MCP server only"]
end
A ~~~ M
Either way, the client ID and secret only identify the broker as an OAuth app. Only a person signing in and approving produces a token, and it acts as that person. The broker uses no system-to-system (client credentials) grant.
Linear, Atlassian, and Cloudflare run their own sign-in for their MCP servers. The broker is registered with each of them once, and names the MCP server when the person connects (the resource parameter, RFC 8707). The token then only works at that server: Linear's only at its read-only server. Registering again would disconnect everyone. See Servers with their own sign-in.
What keeps each service read-only
Each service has several layers. If one fails, the others still hold.
| Service | Tool list | At the service | Scopes the token gets |
|---|---|---|---|
| GitHub | 7 read tools | X-MCP-Readonly header |
Read-only App permissions |
| Linear | 13 read tools | tokens only work at its read-only MCP server | read |
| Atlassian | 8 read and search tools | none | read and search scopes only |
| Cloudflare | search, docs, execute |
none | The only layer: 12 read scopes plus offline_access |
cloudflare_execute can call any Cloudflare API. What stops it writing is that the token only carries read scopes. The disconnect_<service> tools change only the person's own connection in the broker, never data at the service.
What happens to a connection
stateDiagram-v2
direction LR
[*] --> NotConnected
NotConnected --> Connected: approve in the browser
Connected --> Connected: broker refreshes it before it expires
Connected --> NeedsReconnect: the service revoked it, or a refresh failed
NeedsReconnect --> Connected: connect again
Connected --> NotConnected: disconnect_service
- Refresh. When many calls need a new token at once, only one refresh happens, even across several broker replicas (through Redis). A compare-and-swap write in Vault is the final safeguard.
- Disconnect.
disconnect_<service>asks the service to cancel the tokens first, then deletes the broker's copy. If the service is down, the broker keeps retrying and the service can't be used until the cancel succeeds. - Needs reconnect. The next tool call asks the person to connect again.
Token lifecycle walks through every path step by step.
Built in
| Consent bound to the person | A connect link works once, for 5 minutes, only for the person it was made for, in the browser that opened it |
| Fails closed, visibly | A Vault, Redis, or broker outage is a "try again shortly" error, never "please connect" |
| Saved tool lists | Tools are listed from the moment the gateway starts. Changes in a service's live list are logged, never adopted, so one person's details never reach another |
| Audited, without tokens | Every decision is a JSON log line with IDs and outcomes only |
| Kept fresh ahead of time | A background sweeper refreshes tokens before they're needed and retries cancels that are still pending |
| Alert on mass breakage | When many connections to one service break within a minute, the broker logs a broker.stale.mass alert: the service may have revoked the app |
| Custodian, not issuer | The broker has exactly seven routes and never issues tokens of its own |
Where to go next
| You want to | Read |
|---|---|
| Try it on your laptop | Quickstart |
| Understand and configure the gateway | MCP gateway |
| Connect your own MCP server instead | Connect your own MCP server |
| Call the broker directly | Broker API |
| Deploy, configure, or troubleshoot | Deploy and operate |
| Check the security controls and limits | Security |
| Learn how refresh and concurrency work | Design and Token lifecycle |
Status
Software 1.1.0, marked unreleased in the changelog. Tested manually end to end (September 2026) with Keycloak as the sign-in service, Claude Code 2.1.283 as the assistant, and the real GitHub, Linear, Atlassian, and Cloudflare MCP servers. The automated suites use stand-ins for all four.
These docs describe MCP 2026-07-28 (the MCP specification version) and were reviewed against the code on 2026-09-30. Security lists which controls are built in, which are partial, and which your deployment must supply. This project makes no blanket claim of MCP conformance.
Words used in these docs
| Word | Meaning |
|---|---|
| MCP | Model Context Protocol: how AI assistants talk to tool servers |
| MCP client | The AI assistant side, such as Claude Code |
| Hub | Your company's sign-in service (identity provider), such as Keycloak |
| Vendor, service | An outside service that people connect, such as GitHub |
| Connection (grant) | A person's stored permission to use one service |
| Custody | The protected storage (Vault or OpenBao) that holds tokens and app secrets |
| Scope | A named permission, such as "read issues" |
| Refresh | Swapping an expiring token for a new one without asking the person again |
| STALE | A connection that no longer works, so the person must reconnect |
| Single-flight | When many requests need a refresh at once, only one does it and the rest share the result |
| Saved tool list (snapshot) | The checked-in copy of a service's tools that the gateway lists from startup |
| Stand-in | A test double of a service's MCP server, used by the local test stack |
Quickstart
Try it on your laptop: sign in, connect services, and call tools from Claude Code.
source: docs/quickstart.md
Try the whole thing on your own machine: sign in, connect accounts, and call GitHub, Linear, Atlassian, and Cloudflare tools from Claude Code. Everything runs in Docker. You don't need any of those accounts for the first part: stand-ins play all four services.
You need Docker Compose (Docker Desktop, or Colima with at least 4 GB of memory: Keycloak alone takes about 1.5 GB), Python 3.14, and a copy of this repository. Run every command from the repository's top folder.
The stack publishes host ports 6390, 8180, 8210, 8211, 8300, 8310, 8320, 8330, 8500, and 8600 (the multi-replica profile adds 8400 to 8402), and its containers have fixed names (vtb-*), so run one copy per Docker host.
This setup is for trying things out. It uses plain HTTP, test passwords, and storage that is wiped when the containers stop. For a real deployment, see Deploy and operate.
Install
make install # creates .venv with python3.14 and installs the hash-pinned dev lock
Without make, or if python3.14 is not on your PATH (uv can provide one):
uv venv --seed --python 3.14 .venv # or: python3.14 -m venv .venv
.venv/bin/pip install --require-hashes -r requirements-dev.lock
.venv/bin/pip install --no-deps -e .
Run the MCP gateway
1. Start everything
docker compose -f tests/stack/docker-compose.yml --profile gateway up -d --build --wait
.venv/bin/pytest tests/integration -m "gateway and not external" -q
This starts:
| Service | Address | What it is |
|---|---|---|
| Keycloak | http://localhost:8180 |
The sign-in service. Test users: alice / alice and bob / bob |
| MCP gateway | http://localhost:8500/mcp |
What your MCP client connects to |
| Token broker | http://localhost:8600 |
Keeps each user's GitHub, Linear, Atlassian, and Cloudflare tokens |
| Stand-ins | http://localhost:8330 |
Pretend to be GitHub's, Linear's, Atlassian's, and Cloudflare's MCP servers |
flowchart LR
C["Demo client or<br/>Claude Code"] --> G["MCP gateway<br/>:8500"]
C -.->|"sign in"| K["Keycloak<br/>:8180"]
G --> K
G --> B["Broker<br/>:8600"]
B <--> V[("OpenBao<br/>:8211")]
G --> M["Stand-ins<br/>:8330"]
B --> MV["Test sign-in<br/>for the stand-ins<br/>:8310"]
X["Second broker :8300<br/>with a sign-in stub"] -.- MV
The gateway uses the broker on port 8600, which trusts Keycloak. The second broker on port 8300 uses a simple sign-in stub instead. It's for looking under the hood and the broker's own tests.
The tests should all pass. They sign in, connect accounts, call tools, and disconnect, the same way you are about to.
2. Try the demo client
.venv/bin/python tools/mcp-demo-client.py github_get_me
.venv/bin/python tools/mcp-demo-client.py linear_list_issues
.venv/bin/python tools/mcp-demo-client.py disconnect_github
- A browser tab opens on Keycloak. Sign in as
alice. - The first time you use a service, the gateway asks you to connect it, and a second tab opens. Finish there. With the stand-ins, the tab approves by itself.
- The client prints the list of tools and the result.
disconnect_githubcancels the GitHub tokens and deletes the broker's copy. The next GitHub tool asks you to connect again.
Tool names start with the service: github_…, linear_…, atlassian_…, or cloudflare_…. The demo client signs in as the pre-registered app mcp-demo-cli and listens for the sign-in result on port 33418.
3. Try Claude Code
Add the gateway to Claude Code:
claude mcp add --transport http vtb-gateway http://localhost:8500/mcp --callback-port 33419
Then, in a new Claude Code session:
- Run
/mcp, choosevtb-gateway, then Authenticate. - Keycloak opens. Sign in as
aliceand click Yes on the "Grant Access" screen. (Claude Code registers itself with Keycloak the first time.) - Ask Claude to use the gateway, for example: "Using vtb-gateway, list my Linear issues, then tell me my GitHub login."
- If a service isn't connected yet, Claude Code asks to open a link. Accept, finish in the browser, and the answer comes back. Each service asks once.
- To disconnect, ask for it, for example: "Disconnect my Linear account from vtb-gateway."
disconnect_linearis marked as a destructive tool, so MCP clients can ask you before they run it.
If sign-in later fails with an error that names an old address, run claude mcp remove vtb-gateway, add it again, and sign in once.
4. Switch to the real services
- GitHub: create a GitHub App at https://github.com/settings/apps/new:
- Callback URLs:
http://localhost:8300/v1/callback/githubandhttp://localhost:8600/v1/callback/github - Expire user authorization tokens: on
- Webhook: off
- Repository permissions, all read-only: Contents, Issues, Pull requests, Metadata
Then generate a client secret, and install the App on the repositories you want to use.
2. Linear, Atlassian, and Cloudflare: these register the broker with a script instead of a developer console (why).
- For Atlassian, first ask an org admin to allow http://localhost:*/** in Atlassian Administration, under Rovo → Rovo MCP server → Domain settings.
- Then register once per service. The script writes the ID and secret into tests/stack/.env:
```sh
.venv/bin/python tools/register-mcp-client.py linear \
--redirect-base http://localhost:8600 --env-file tests/stack/.env
.venv/bin/python tools/register-mcp-client.py atlassian \
--redirect-base http://localhost:8600 --env-file tests/stack/.env
.venv/bin/python tools/register-mcp-client.py cloudflare \
--redirect-base http://localhost:8600 --env-file tests/stack/.env
```
Don't run it again later: a new registration disconnects every account connected with the old one. Linear's and Cloudflare's secrets expire after about 90 days, and the script prints the date.
- Put the GitHub App's ID and secret in
tests/stack/.envtoo (git ignores this file):
sh
GITHUB_CLIENT_ID=...
GITHUB_CLIENT_SECRET=...
Run the up command from step 1 again, so the broker gets them. You can set up any subset of the services.
4. Point the gateway at the real services. Give people two minutes to finish connecting, and give slow calls 15 seconds:
sh
GATEWAY_UPSTREAMS= GATEWAY_CONSENT_WAIT_S=120 GATEWAY_HTTP_TIMEOUT_S=15 \
docker compose -f tests/stack/docker-compose.yml --profile gateway up -d --no-deps --wait mcp-gateway
To serve only some services, add for example GATEWAY_ENABLED_UPSTREAMS=github,atlassian.
5. Use the demo client or Claude Code as before. github_get_me returns your own GitHub account, linear_list_teams your Linear teams, and atlassian_getAccessibleAtlassianResources your Atlassian sites. connect_cloudflare connects Cloudflare. To check them all automatically, run .venv/bin/pytest tests/integration/test_external_mcp_servers.py -m external.
To go back to the stand-ins, run the up command for mcp-gateway again without those variables.
Look under the hood: the broker on its own
The gateway gets tokens from the broker. You can talk to the broker directly, the way a gateway does. Step 1 also started a second broker on port 8300, with a simple sign-in stub instead of Keycloak. This section uses that one. If you skipped step 1, start it with:
docker compose -f tests/stack/docker-compose.yml up -d --build --wait
curl -fsS http://localhost:8300/healthz
Expected: {"ok":true,"custody":"ok"}.
The script below pretends to be the gateway, so it receives a token. It prints only status codes and scope names. A real MCP client must never see that token.
.venv/bin/python - <<'PY'
import requests
broker = "http://localhost:8300"
hub = "http://localhost:8320"
subject = "wf-quickstart"
mint = requests.post(f"{hub}/_test/token", json={"sub": subject}, timeout=10)
mint.raise_for_status()
headers = {"Authorization": "Bearer " + mint.json()["access_token"]}
body = {"vendor": "mockhub", "required_scopes": ["issues:read"]}
# Make the walkthrough repeatable for this dedicated test subject.
reset = requests.delete(f"{broker}/v1/grants/mockhub/{subject}",
headers=headers, timeout=15)
assert reset.status_code in (200, 404), reset.status_code
challenge = requests.post(f"{broker}/v1/tokens/resolve", json=body,
headers=headers, timeout=15)
assert challenge.status_code == 404, challenge.status_code
print("Resolve:", challenge.status_code, challenge.json()["title"])
# Session preserves the consent cookie across hub and vendor redirects.
# The mock hub signs in automatically; a real hub requires browser login.
with requests.Session() as browser:
connected = browser.get(challenge.json()["authorize_uri"], timeout=15)
assert connected.status_code == 200 and "Connected" in connected.text
print("Consent: connected")
resolved = requests.post(f"{broker}/v1/tokens/resolve", json=body,
headers=headers, timeout=15)
resolved.raise_for_status()
print("Resolve:", resolved.status_code, "scopes:", resolved.json()["granted_scopes"])
deleted = requests.delete(f"{broker}/v1/grants/mockhub/{subject}",
headers=headers, timeout=15)
deleted.raise_for_status()
print("Disconnect:", deleted.json())
again = requests.post(f"{broker}/v1/tokens/resolve", json=body,
headers=headers, timeout=15)
assert again.status_code == 404
print("Resolve after disconnect:", again.status_code, again.json()["title"])
PY
What you should see:
404 needs-consent: not connected yet.- The account connects.
200: the broker hands over a token.{"revoked": true}: disconnected.404 needs-consentagain.
The test service widens scopes when it refreshes, so step 3 may list both issues:read and issues:write (see scope policy). For a guided tour with a browser, including attack attempts, see Smoke tests.
Run the automated checks
The Makefile wraps these commands (make help lists every target):
| Command | What it runs |
|---|---|
make check |
Lint, format check, mypy, unit tests with coverage, generated-docs check. No Docker |
make test-all |
make test, then starts the stack and runs the broker and gateway suites |
make test-multi |
Restarts the stack in the multi-replica profile (fresh Redis, so no lease to wait out) and runs its suite |
make stack-down |
Stops every profile |
The same, by hand:
.venv/bin/pytest tests/unit -q
.venv/bin/pytest tests/integration -m "not external and not multi and not gateway" -q
To run the same checks with Redis doing the coordination:
COORD_BACKEND=redis docker compose -f tests/stack/docker-compose.yml \
up -d --no-deps --wait broker
.venv/bin/pytest tests/integration -m "not external and not multi and not gateway" -q
To test two brokers sharing the work, first stop the single broker. Then wait until its sweep lease has run out (up to 120 seconds with default settings). Don't flush a Redis that others share.
docker compose -f tests/stack/docker-compose.yml stop broker
docker compose -f tests/stack/docker-compose.yml --profile multi \
up -d --build --wait broker-a broker-b broker-lb
# Inspect the remaining lease; wait until the previous lease has expired.
docker compose -f tests/stack/docker-compose.yml exec -T redis \
redis-cli PTTL vtb:sweep-lease
BROKER_URL=http://localhost:8400 BROKER_CONTAINERS=vtb-broker-a,vtb-broker-b \
.venv/bin/pytest tests/integration/test_multi_replica.py -q
The two brokers use a 10-second sweep lease. If PTTL shows more than 10000 ms, the old broker's lease is still there, so wait. When the test finishes, it restarts the single broker. Stop the two brokers before you run the single-broker tests again, so their background sweeps don't interfere.
Stop everything
docker compose -f tests/stack/docker-compose.yml --profile multi --profile gateway down
This wipes the test storage. Next time, run the setup again and reconnect accounts.
MCP Gateway
How the gateway works, what your sign-in service must do, and how to configure it.
source: docs/mcp-gateway.md
Reviewed 2026-09-30 · MCP 2026-07-28 · FastMCP 4.0.10 and Keycloak 26.7.4 in the automated suites; manually tested with Claude Code 2.1.283.
The MCP gateway lets AI assistants such as Claude Code use tools from several services, each as the signed-in person. It serves GitHub, Linear, Atlassian (Jira and Confluence), and Cloudflare.
- To the assistant, it looks like one ordinary MCP server, protected by your company's sign-in service (the "hub").
- Behind it, it is an MCP client of each service's own MCP server.
- On every tool call, it gets that person's token for that service from the token broker, uses it once, and throws it away.
The gateway is a separate service (src/mcp_gateway/). The broker has exactly seven routes and issues no tokens of its own.
Supported servers
| Service | Its MCP server | Tools | Read-only by |
|---|---|---|---|
| GitHub | https://api.githubcopilot.com/mcp/ |
7: github_get_me, github_search_repositories, github_get_file_contents, github_list_issues, github_issue_read, github_list_pull_requests, github_pull_request_read |
X-MCP-Readonly: true header (plus lockdown mode and a fixed toolset list), plus the tool list |
| Linear | https://mcp.linear.app/mcp/readonly |
13: linear_list_issues, linear_get_issue, linear_list_comments, linear_list_projects, linear_get_project, linear_list_cycles, linear_list_teams, linear_get_team, linear_list_issue_statuses, linear_list_users, linear_get_user, linear_list_documents, linear_get_document |
Tokens are bound to Linear's read-only MCP server (its full MCP server refuses them), the read scope, plus the tool list |
| Atlassian | https://mcp.atlassian.com/v2/mcp |
8: atlassian_getJiraIssue, atlassian_searchJiraIssuesUsingJql, atlassian_getConfluenceContent, atlassian_searchConfluence, atlassian_executeRead, atlassian_discover, atlassian_atlassianUserInfo, atlassian_getAccessibleAtlassianResources |
Read and search scopes only, plus the tool list |
| Cloudflare | https://mcp.cloudflare.com/mcp |
3: cloudflare_search, cloudflare_docs, cloudflare_execute |
Scopes only. execute runs code against the whole Cloudflare API. What stops it writing is that the token only has 12 read scopes, plus offline_access |
For GitHub, only X-MCP-Readonly blocks writes. Lockdown mode (X-MCP-Lockdown) is a defence against prompt injection: in public repositories, it only returns content from people with push access. X-MCP-Toolsets limits which tool groups the server offers.
Each service also has two account tools:
| Tool | What it does |
|---|---|
connect_<service> (connect_github, connect_linear, connect_atlassian, connect_cloudflare) |
Connects the person's account, opening a browser if needed. See Connecting a service during a tool call |
disconnect_<service> (disconnect_github, disconnect_linear, disconnect_atlassian, disconnect_cloudflare) |
Cancels the person's connection. See Disconnecting a service |
Each person connects each service separately: connecting GitHub doesn't connect Linear.
Linear, Atlassian, and Cloudflare run their own sign-in for their MCP servers, so the broker is registered with each of them once. See Servers with their own sign-in.
What happens on a tool call
flowchart LR
C["AI assistant<br/>(Claude Code, demo client)"] -->|"sign-in token for the gateway"| G["MCP gateway"]
C -.->|"sign in"| H["Sign-in service (Keycloak)"]
G -->|"swap for an internal token"| H
G -->|"internal token, ask for the service's token"| B["Token broker"]
B --- V["Vault (custody)"]
G -->|"GitHub token"| M1["GitHub MCP server"]
G -->|"Linear token"| M2["Linear MCP server"]
G -->|"Atlassian token"| M3["Atlassian MCP server"]
G -->|"Cloudflare token"| M4["Cloudflare MCP server"]
The assistant only talks to the gateway and the sign-in service. The service tokens stay in the broker's vault, and each one goes only to its own MCP server.
sequenceDiagram
autonumber
participant C as AI assistant
participant G as MCP gateway
participant H as Sign-in service
participant B as Broker
participant M as Service MCP server
C->>G: call linear_list_issues (with sign-in token)
Note over G: check the token: signature, issuer,<br/>meant for this gateway, has the mcp-gateway scope
G->>H: swap it for an internal token
H-->>G: internal token (same person)
G->>B: resolve: this person's Linear token
alt Linear is connected
B-->>G: 200 and the Linear token
else not connected yet
B-->>G: 404 needs-consent and a link
G-->>C: please open this link
Note over C: the person connects Linear in the browser
C->>G: accepted
loop every 2 s, up to CONSENT_WAIT_S
G->>B: GET /v1/grants - is Linear ACTIVE yet?
end
G->>B: resolve again
B-->>G: 200 and the Linear token
end
G->>M: new session, call list_issues as this person
M-->>G: result
G-->>C: result (no tokens in it)
Notice that the gateway resolves a second time only after the connection shows up in the grant list. Then it runs the tool as usual.
Four rules keep this safe:
- The sign-in token only goes to the sign-in service. The gateway swaps it and never forwards it. The broker only sees the internal token.
- Each service only ever gets its own token. GitHub never sees a Linear token, and Linear never sees a GitHub token.
- Service tokens are used once. Each call opens a new session with the service and closes it afterwards. The token is never cached, logged, or sent to the assistant.
- While waiting, the gateway checks the connection list, not the token. After the person agrees to connect, the gateway polls
GET /v1/grantsevery 2 seconds, for up toCONSENT_WAIT_S. Asking for the token instead would create a new connection link every time.
Identity handoff: what the hub must do
The gateway swaps the person's sign-in token for an internal token (a "hub JWT") that the broker accepts. The hub (your sign-in service) does the swap, so it must support the following. The right-hand column shows how the test setup does it in Keycloak (tests/stack/keycloak/mcp-realm.json).
| The hub must | Why | In the Keycloak test realm |
|---|---|---|
| Let the gateway's app swap tokens (token exchange) | So the sign-in token is never forwarded | standard.token.exchange.enabled on mcp-gateway |
Sign tokens with an algorithm the broker accepts (its HUB_ALGORITHMS, default PS256 and ES256) |
The broker never accepts RS256 or shared-secret (HMAC) signatures | defaultSignatureAlgorithm: PS256 |
Give the swapped token exactly one mcp://tier/* audience and an mcp_contract claim |
That's what the broker checks | hub-tier client scope |
| Make sign-in tokens name the gateway as their audience | The gateway refuses tokens meant for anything else | mcp-gateway client scope |
| Use the same user IDs for sign-in, the swap, and the broker's own sign-in page | The broker refuses a connection link opened by a different person (403) | All three apps are in one realm |
Two notes on other sign-in services:
- Keycloak ignores the "resource" parameter that MCP clients send to say which server a token is for. So the gateway tells clients to ask for the
mcp-gatewayscope instead, and that scope sets the audience. - Okta and Entra sign with RS256, and Entra uses its own "on-behalf-of" flow instead of standard token exchange. With those, you'd need a small internal service to issue the internal token.
The test realm lets MCP clients register themselves, which is how stock clients like Claude Code sign up. It only allows this for apps that return to localhost or 127.0.0.1, and only after a consent screen. Such apps may use the mcp-gateway and offline_access scopes plus the realm's default scopes (basic). That's fine on a laptop, not in production. The pre-registered demo client is mcp-demo-cli.
Standards: token exchange is RFC 8693. The resource parameter is RFC 8707. The audience a token is meant for is its aud claim.
Tools
Every service tool is named <service>_<tool>, for example github_get_me or linear_list_issues. The gateway strips the prefix before calling the service. Tools that change things, such as GitHub's create_issue or Linear's save_issue, are never listed.
All of these are listed from the moment the gateway starts. Their descriptions come from a saved copy of each service's tool list (src/mcp_gateway/snapshots/<service>.json). The live tool list is read only when someone runs connect_<service>, once per service for each gateway process. Ordinary tool calls never read it. The gateway then writes a gateway.catalog log line for that service (fields upstream, listed, added, changed, missing):
changed: the live list describes the tool differently from the saved copy. The gateway keeps listing the saved copy, because the list is shared by every user and a service may personalize what it returns (Cloudflare puts the signed-in user's email and account ID in a description). Refresh the saved copy when you see this.added: allowed but missing from the saved copy. The gateway registers it and tells clients the list changed.missing: the live list doesn't include it. It stays listed, and calling it returns the service's error.
flowchart TD
S["Gateway starts"] --> L["List every allowed tool<br/>from its saved copy"]
L --> K["Someone runs connect_service"]
K --> R["Read the live list<br/>once per service and process"]
R --> CH["changed: logged,<br/>saved copy stays listed"]
R --> AD["added: registered and<br/>announced with list_changed"]
R --> MI["missing: logged,<br/>stays listed"]
Notice that a tool with a saved copy always keeps it. Only a tool with no saved copy takes its description from the live list.
Why keep saved copies? MCP 2026-07-28 only lets servers announce a changed tool list over a separate subscription channel, which FastMCP 4.0.10 doesn't support. Without the saved copies, Claude Code would only see the connect_… and disconnect_… tools until it reconnected.
Protocol details:
- From 2026-07-28,
notifications/tools/list_changedmay only be sent on thesubscriptions/listenstream. The gateway still sends it when it adds a tool, which only older clients act on. - GitHub's saved tool list carries
x-mcp-headerannotations. On 2026-07-28, MCP clients mirror those arguments intoMcp-Param-*request headers, but only for tools they have listed. So list tools before calling them.
To update a saved copy from the live service:
UPSTREAM_TOKEN=$(gh auth token) .venv/bin/python tools/refresh-tool-snapshot.py github
UPSTREAM_TOKEN=<Linear API key> .venv/bin/python tools/refresh-tool-snapshot.py linear
UPSTREAM_TOKEN=<token from the broker> .venv/bin/python tools/refresh-tool-snapshot.py atlassian
UPSTREAM_TOKEN=<token from the broker> .venv/bin/python tools/refresh-tool-snapshot.py cloudflare
For Atlassian and Cloudflare, the token has to come from the broker: connect the account first, then resolve it at the broker (see Broker API).
The script:
- always reads the bundled
src/mcp_gateway/upstreams.json(notGATEWAY_UPSTREAMS), and uses exactly the gateway's URL, headers, auth scheme, and allowlist for that service; - writes the file named in that entry's
snapshot, undersrc/mcp_gateway/snapshots/; - fails, and writes nothing, if an allowlisted tool is missing from the live list.
Tests fail if any saved copy and its tool list ever disagree, if a saved tool isn't marked read-only (Cloudflare is the one exception: its scopes are what keep it read-only), or if a saved copy contains an email address or a 32-character hex ID.
Check a new saved copy before you commit it. It was fetched with someone's token, and a service may write that person's details into it. Cloudflare's execute description names the person's email and account ID. The committed copy replaces that line with "your Cloudflare account".
Treat what services return (issue text, file contents, comments) as untrusted input for the assistant. The gateway passes it through unchanged and never acts on it.
Connecting a service during a tool call
If the person hasn't connected a service yet, any of its tools, or its connect_… tool, asks the assistant to open the broker's connection link for that service. What that looks like depends on the assistant:
| Assistant | What happens |
|---|---|
| Speaks MCP 2026-07-28 and can open links (Claude Code) | The call comes back asking for input. The assistant asks the person, opens the link, and repeats the call with their answer. The gateway runs the tool again from the start on that second try. |
| Speaks MCP 2025-11-25 and can open links | The gateway asks the assistant to open the link during the call, and waits for the answer. |
| Can't open links | The tool returns an error that contains the link. Open it yourself, then try again. |
Protocol details: on 2026-07-28 the call returns an InputRequiredResult carrying a URL-mode elicitation request. On 2025-11-25 the gateway sends elicitation/create with mode url during the call. Both carry a fresh elicitationId. "Can open links" means the client advertised the url elicitation capability.
Saying yes only means the person chose to open the link. The gateway then waits up to CONSENT_WAIT_S for the connection to appear, and resolves once more. Saying no, or cancelling, ends the call with an error straight away.
Disconnecting a service
disconnect_<service> cancels the person's connection to that service. It needs no browser and no input.
sequenceDiagram
autonumber
participant C as AI assistant
participant G as MCP gateway
participant H as Sign-in service
participant B as Broker
participant V as Service sign-in
C->>G: call disconnect_github
G->>H: swap the sign-in token for an internal token
H-->>G: internal token
G->>B: DELETE /v1/grants/github/sub
B->>V: revoke the tokens first
V-->>B: revoked
Note over B: delete the broker's copy
B-->>G: 200
G-->>C: GitHub is disconnected
Notice that the service is asked to cancel the tokens before the broker deletes its copy. The gateway never calls the service's MCP server here.
How it works:
- The gateway calls the broker's self-service
DELETE /v1/grants/{vendor}/{sub}with the internal token. It takessubfrom that token, so a person can only disconnect themselves. - The tool is marked destructive (
destructiveHint: true,readOnlyHint: false,idempotentHint: true). Clients such as Claude Code may ask the person before running it. - Each outcome is explained to the person and logged as
gateway.disconnectwithoutcome:
outcome |
Broker answer | What the person is told |
|---|---|---|
revoked |
200 | Disconnected: the service was asked to cancel the tokens, and the broker deleted its copy. Run connect_<service> to connect again |
unsupported |
200 with vendor_revocation: unsupported |
Disconnected here, but the service can't cancel tokens remotely. Also remove the app's access in the service's account settings |
not-connected |
404 no-grant |
The service was not connected |
pending |
502 revoke-pending |
The service could not be reached to cancel the tokens. The broker keeps retrying, and until it succeeds the service can't be used here |
While a disconnect is pending, the broker parks the connection as REVOKE_PENDING and its sweeper retries the cancel. Tool calls for that service fail with revoke-pending (see Errors).
Errors
Messages name the service involved, for example "Connect Linear first".
| What went wrong | What the assistant gets | Log line |
|---|---|---|
| Not connected, and the client can't open links | Error with the link: open it, then retry | gateway.consent outcome manual |
| Not connected, and the client can open links | A request to open the link | gateway.consent outcome elicit, with era (the protocol revision) |
| The person said no or cancelled | Error: the service was not connected | gateway.consent outcome decline or cancel |
| Said yes, but didn't finish in time | Error: finish in the browser, then try again | gateway.consent outcome timeout |
| Said yes, and the connection appeared | The tool runs | gateway.consent outcome connected |
| Connection appeared, but the second resolve gave no token | Error: the connection is not usable, retry | none |
| Broker 5xx or unreachable while waiting for the person to connect | Error: try again shortly. Never "finish connecting in the browser" | gateway.call outcome unavailable |
| Broker, sign-in service, or the service down (or 5xx), or too slow | Error: try again shortly. Never "connect" | gateway.call outcome unavailable (broker or sign-in service) or upstream-error (the service, with a detail, such as a timeout) |
| The sign-in service refused the swap, or the broker answered 401 | Error: a gateway configuration problem | gateway.call outcome error |
A disconnect is still in progress (409 revoke-pending) |
Error: the service is not usable right now (revoke-pending), try again later. No link |
gateway.call outcome deny |
Any other broker 4xx on resolve, such as 404 unknown-vendor or a 404 or 409 without a link |
Error: the service is not usable right now (the problem title), try again later | gateway.call outcome deny, reason the problem title |
| The service refused the token (its error mentions 401) | Error: reconnect the service and try again | gateway.call outcome upstream-error, reason rejected |
| The service's tool ran but reported an error | The tool's own error result | gateway.call outcome ok with is_error: true |
| The call worked | The result | gateway.call outcome ok with is_error: false |
disconnect_<service>: broker 5xx or unreachable |
Error: try again shortly | gateway.call outcome unavailable |
disconnect_<service>: broker 401, 403, or another 4xx |
Error: a gateway configuration problem | gateway.call outcome error |
| The assistant's sign-in token was refused | HTTP 401, with WWW-Authenticate: Bearer scope="mcp-gateway", resource_metadata=… telling it how to sign in |
gateway.auth outcome deny, with reason expired, audience, issuer, scope, algorithm, signature, malformed, or missing |
Log lines are one JSON object each, with event and ts. They contain IDs and outcomes, never tokens:
gateway.call,gateway.consent,gateway.disconnect:upstream(the service),tool, the person'ssub,outcome, and areasonwhere one applies.gateway.catalog:upstream,listed,added,changed,missing.gateway.auth(refused sign-in tokens):subandclient(the app's client ID, fromazp) when the token could be read,algfor reasonalgorithm, andexpired_s_agofor reasonexpired. None of these formalformedormissing.detailonupstream-erroris the service's error text, cut to 200 characters, with the token replaced by<token>if it ever appears.
A test (tests/integration/test_mcp_gateway.py::test_no_token_material_in_any_container_log) searches every container's logs for the assistant's sign-in token, the internal token (hub JWT), and the GitHub, Linear, and Atlassian stand-in tokens.
Configuration
Required settings: GATEWAY_PUBLIC_URL, HUB_ISSUER, HUB_JWKS_URI, HUB_TOKEN_ENDPOINT, GATEWAY_CLIENT_ID, GATEWAY_CLIENT_SECRET, BROKER_URL. If any are missing, the gateway won't start, and it lists every missing name.
| Setting | Default | What it does |
|---|---|---|
GATEWAY_PUBLIC_URL |
— | The gateway's public address. Clients connect to {GATEWAY_PUBLIC_URL}/mcp, and sign-in tokens must name that address as their audience |
HUB_ISSUER |
— | The sign-in service's public address, as it appears in tokens and as clients see it |
HUB_JWKS_URI, HUB_TOKEN_ENDPOINT |
— | Where to get the sign-in service's signing keys and where to swap tokens. These may be internal addresses |
GATEWAY_CLIENT_ID, GATEWAY_CLIENT_SECRET |
— | The gateway's own app at the sign-in service, used for the swap |
BROKER_URL |
— | The broker's internal address |
GATEWAY_UPSTREAMS |
the bundled upstreams.json |
Path to a different upstreams file (see below) |
GATEWAY_ENABLED_UPSTREAMS |
all | Comma-separated service names to serve, for example github. An unknown name stops startup |
HUB_ALGORITHM |
PS256 |
Signature type sign-in tokens must use. One of the broker's allowed algorithms: PS256, PS384, PS512, ES256, ES384, ES512, EdDSA. Anything else stops startup |
GATEWAY_SCOPE |
mcp-gateway |
Scope sign-in tokens must include. Clients are told to ask for it |
HUB_EXCHANGE_SCOPE |
hub-tier |
Scope requested in the swap. It must produce what the broker expects |
MIN_TTL_S |
120 |
Minimum seconds of life left on each service token. Must be 0 or more |
CONSENT_WAIT_S |
120 |
How long to wait for someone to finish connecting after they say yes. Must be 0 or more |
HTTP_TIMEOUT_S |
15 |
Time limit for each call to the sign-in service, the broker, and a service. Keep it well above the slowest tool (some Linear list calls take several seconds) |
The upstreams file
src/mcp_gateway/upstreams.json lists the services. Each entry:
| Field | Meaning |
|---|---|
name |
Short name matching ^[a-z][a-z0-9]{0,19}$ (1–20 lowercase letters or digits, starting with a letter). It becomes the tool prefix, connect_<name>, and disconnect_<name> |
display_name |
Used in messages, for example "Connect your Linear account" |
vendor |
The broker's vendor ID for this service's tokens (see the vendor registry) |
url |
The service's MCP server. Use its read-only address where it has one |
tools |
The allowlist: the service's own tool names |
headers |
Optional. Extra headers on every call, such as GitHub's X-MCP-Readonly. Authorization is not allowed here |
auth_scheme |
Optional. Bearer (default), or Sentry-Bearer for Sentry's MCP server |
protocol |
Optional. How the gateway, as a client, picks the MCP revision. legacy (default): the classic handshake, for servers that stop at MCP 2025-11-25, like GitHub and Linear. auto: try revision 2026-07-28 first, and fall back to the classic handshake for servers that don't show they support it. 2026-07-28: use that revision without trying. All four bundled services use legacy |
snapshot |
Optional. The saved tool list. A bare file name (no /) always means the bundled src/mcp_gateway/snapshots/ folder. A value with a / in a custom upstreams file (GATEWAY_UPSTREAMS) is a path relative to that file; in the bundled file it is looked up under snapshots/ too. Without it, the service's tools only appear after someone runs its connect_<name> tool |
The gateway checks the file at startup and refuses to start if:
- the file can't be read, isn't JSON, or an entry lacks
name,display_name,vendor,url, ortools, or a named snapshot file can't be read; - a
namebreaks the rule above, or two entries share a name; toolsis empty;auth_schemeisn'tBearerorSentry-Bearer;protocolisn'tlegacy,auto, or2026-07-28;headerssetsAuthorization(in any letter case);GATEWAY_ENABLED_UPSTREAMSnames a service that isn't in the file;- no services are left to serve.
The gateway is built from Dockerfile.gateway. Its dependencies come from its own checked lock file, requirements-gateway.lock, so FastMCP never ends up in the broker's image.
Add another server
Services whose MCP server accepts an ordinary token from that service's own OAuth app work like GitHub. Sentry, GitLab, Azure DevOps, Slack, and PagerDuty are in this group.
- Register an OAuth app with the service. Use callback
https://<broker>/v1/callback/<vendor>and read-only scopes. - Add a broker registry entry (
registry.example.json) with the service's OAuth endpoints,scope_ceiling, andenabled_env. Store the app's client ID and secret in custody atvendor-clients/<vendor>. See Deploy and operate. - Add an upstreams entry with the MCP server's URL, the read-only tools you want, and any read-only headers or URL.
- Save its tool list:
UPSTREAM_TOKEN=… .venv/bin/python tools/refresh-tool-snapshot.py <name>. - Test it: add the service to the stand-in (
tests/stack/mock-mcp) for CI, and totests/integration/test_external_mcp_servers.pyfor a real-account check.
Services that run their own MCP sign-in (Linear, Atlassian, Cloudflare, Notion, and many others) need two more steps; see the next section.
Servers with their own sign-in
Linear's, Atlassian's, and Cloudflare's MCP servers run their own OAuth sign-in, and the token it issues is only valid at that one MCP server. (Atlassian's and Cloudflare's accept no other kind of token. Linear's read-only server also accepts a personal API key, which the snapshot script can use.) So the broker is an OAuth client of that sign-in service, just like it is of GitHub's:
- It registers itself once, with dynamic client registration. There is no developer console to create an app in.
- It names the MCP server on every request with the
resourceparameter: when the person connects, when it swaps the code for a token, and on every refresh. The registry entry'sresourcefield holds that address. - Except where refresh refuses it. Linear rejects
resourceon refresh, and keeps the refreshed token bound to the same server anyway. Its registry entry setsresource_on_refresh: false.
sequenceDiagram
autonumber
participant P as Person's browser
participant B as Broker
participant A as Service sign-in
participant G as MCP gateway
participant M as Service MCP server
B->>P: redirect to authorize, resource = MCP server URL
P->>A: sign in and approve
A-->>B: callback with a code
B->>A: swap the code, resource = MCP server URL
A-->>B: token for that MCP server only
Note over B: later, before the token runs out
B->>A: refresh (resource too, except Linear)
A-->>B: new token pair
B-->>G: resolve answers with the token
G->>M: tool call with the token
Note over M: any other server refuses this token
resource goes on the authorize, code-swap, and refresh requests (Linear: only the first two). The token that comes back works only at that MCP server.
Standards: dynamic client registration is RFC 7591. The resource parameter is RFC 8707.
To add one:
- Add a registry entry with
auth_metadata_url(the sign-in service's metadata),resource(the MCP server's URL),scope_ceiling(only read scopes), andenabled_env. Addresource_on_refresh: falseif the service refusesresourceon refresh. - Register the broker, once:
```sh
# writes
# or straight into custody (needs VAULT_ADDR and an admin VAULT_TOKEN) .venv/bin/python tools/register-mcp-client.py atlassian \ --redirect-base https://broker.example.com --vault ```
What the script does:
- It reads the vendor's entry from
registry.example.jsonby default, or from--registryorREGISTRY_PATH. The entry must haveauth_metadata_url. - It registers as
vtb-mcp-gateway, with the callback<redirect-base>/v1/callback/<vendor>, the registry'sscope_ceilingas its scopes, and the registry'stoken_endpoint_auth_method(defaultclient_secret_post). - It refuses a service whose metadata has no registration endpoint, and a registration that comes back without a client secret (a public client): the broker needs a confidential client.
- It never prints the secret. It warns if the registration sets an expiry date for the secret.
- Continue with steps 3–5 above: the upstreams entry, its saved tool list, and tests.
Register only once. Each registration creates a new client, and every connection made with the old one stops working. The script refuses when credentials already exist, unless you pass --force. With --env-file, it checks whether <VENDOR>_CLIENT_ID is already set in that file. With --vault, it checks for vendor-clients/<vendor> in custody.
What each service needs:
| Linear | Atlassian | Cloudflare | |
|---|---|---|---|
| Before registering | Nothing | An org admin allows the broker's callback address: in Atlassian Administration, under Rovo → Rovo MCP server → Domain settings | Nothing |
| MCP server the token is bound to | https://mcp.linear.app/mcp/readonly |
https://mcp.atlassian.com/v2/mcp |
https://mcp.cloudflare.com/mcp |
| Scopes the broker asks for | read |
read:me, read:account, offline_access, and read and search for Jira and Confluence (…:agent-interface) |
12 read scopes plus offline_access: user:read, account:read, workers-scripts.read, workers-routes.read, workers-observability.read, workers-tail.read, workers-ci.read, workers-kv-storage.read, workers-r2.read, workers-r2-bucket-item.read, logs.read, account-logs.read. Asked without a scope, Cloudflare granted 194 read scopes |
| Tokens | 24 hours, refresh token rotates, reusing an old one fails. resource is left off refresh |
8 hours, refresh token rotates | 1 hour, refresh token rotates |
| Client secret | Expires after about 90 days. The test registration expires on 2026-12-27. Register again, update custody, and everyone reconnects | The script warns if the registration sets an expiry date | Expires after about 90 days. The test registration expires on 2026-12-26. There's no way to renew it: register again, update custody, and everyone reconnects |
Cloudflare's separate observability and builds MCP servers each have their own sign-in and would need their own registration. The main server covers the same read APIs through execute.
The test stack stands in for Atlassian's and Cloudflare's sign-in: mock-vendor plays it, and refuses a code exchange or refresh that doesn't name the right server. The stand-ins at /atlassian/mcp and /cloudflare/mcp refuse tokens issued for any other server. Linear's stand-in uses an ordinary test app (vendor mockhub-jwt), and its resource_on_refresh: false rule is covered by unit tests (tests/unit/test_vendor_client.py).
Figma's MCP server also runs its own sign-in, but only lets clients from its approved list register. It refused the broker's registration.
Run it locally
The Quickstart walks through it step by step. In short:
docker compose -f tests/stack/docker-compose.yml --profile gateway up -d --build --wait
.venv/bin/pytest tests/integration -m "gateway and not external" -q
This starts:
- Keycloak at
localhost:8180; - a broker that trusts it at
localhost:8600, with its own OpenBao; - the gateway at
localhost:8500; - stand-ins for GitHub, Linear, Atlassian, and Cloudflare at
localhost:8330(/github/mcp,/linear/mcp,/atlassian/mcp,/cloudflare/mcp), which only accept live test tokens.
By default the gateway uses the stand-ins (tests/stack/gateway/stand-in-upstreams.json). To use the real services, set GATEWAY_UPSTREAMS= (empty). The test users are alice and bob, and each password is the same as the username.
The stack sets shorter limits so the tests run quickly:
| Gateway setting | Stack default | Override with |
|---|---|---|
CONSENT_WAIT_S |
15 |
GATEWAY_CONSENT_WAIT_S (use 120 when a person connects in the browser) |
HTTP_TIMEOUT_S |
5 |
GATEWAY_HTTP_TIMEOUT_S (use 15 against the real services) |
Known limitations
- No subscription channel. The saved tool lists avoid needing one. But if you allow a tool that isn't in a saved copy, it only appears after someone runs
connect_<service>, and clients on MCP 2026-07-28 only see it after they reconnect. - Many dependencies.
requirements-gateway.lockpins 80 packages with checksums, including FastMCP 4.0.10 and the broker's own dependencies. Review updates to that file like any other supply-chain change. - FastMCP reads
FASTMCP_*settings from a.envfile in the working directory (override the file name withFASTMCP_ENV_FILE). The gateway's own settings come only from the environment. The image's working directory holds no.env(.dockerignoreexcludes it), so this matters only when you run the gateway from a checkout. - One swap and one service session per call. This is simple and keeps nothing between calls, but it adds some delay. The gateway doesn't reuse connections.
- Read-only. Write tools are blocked by the tool lists, and by each service's read-only header, URL, or scope. For Cloudflare, only the scopes do this, because
executecan call any Cloudflare API.disconnect_<service>changes only the person's own connection in the broker, never data at the service. - Linear's and Cloudflare's client secrets expire after about 90 days. The test registrations expire on 2026-12-27 and 2026-12-26. Register again, update custody, and everyone reconnects. See Servers with their own sign-in. Put the date in your calendar.
- Saved tool lists win. If a service changes a tool, the gateway keeps listing the saved copy until you refresh it. Cloudflare's
executeis always reported aschangedwhen someone runsconnect_cloudflare, because its live description is personalized. - Linear account IDs aren't recorded. Linear's API only speaks GraphQL, so the broker logs Linear connections with
vendor_user_idunknown. This only affects audit joins. - Signing keys are cached for an hour. New keys from the sign-in service are picked up on first use. But a key the sign-in service removes (for example after a leak) stays trusted by the gateway until the cache runs out. Restart the gateway when you revoke a signing key. (The broker's window is shorter: it never caches single keys, and it re-fetches the whole key set at least every 5 minutes (
JWKS_CACHE_S), so a removed key stops working there within 5 minutes.) - What we saw with Claude Code 2.1.283. After the sign-in service moved to a new address, Claude Code kept using the old token endpoint. After a failed token refresh, its next tool call arrived with no usable sign-in (logged as
gateway.auth). In both cases, removing and re-adding the server, then signing in once, fixed it. It also registered itself even when a pre-registered app ID was set.
Connect Your Own MCP Server
Use the token broker from an MCP server or gateway you build yourself.
source: docs/mcp-integration.md
Reviewed 2026-09-30 · MCP 2026-07-28.
Use this page to connect your own MCP server or gateway to the broker.
- Just want GitHub, Linear, Atlassian, or Cloudflare? You do not need this page. The shipped MCP gateway already connects their MCP servers to the broker and handles all the MCP work described here.
- The broker itself does not speak MCP. It offers an internal REST API (API reference). Your gateway sits between MCP clients and that API.
Three authorization boundaries
Three separate credentials are in play. Keep them apart.
| Boundary | Credential and validator | Owner |
|---|---|---|
| MCP client → MCP server | Access token issued for that MCP resource; checked by the MCP server | Your MCP client and authorization server; the shipped gateway checks it |
| Gateway → broker | Internal hub JWT, checked again by the broker | Your identity handoff and this broker |
| Gateway → vendor API | Vendor access token obtained through broker consent | Gateway, broker, vendor |
The hub is your company's sign-in service (identity provider). The hub JWT is the internal token it issues, which the broker accepts.
Do not forward the MCP client's token to the broker. Get an internal credential that matches the broker's configured issuer, audience, and contract instead. The shipped gateway does this with a token exchange at the hub (RFC 8693).
These parts of the hub JWT are this project's own conventions. They are not MCP protocol versions or standard MCP claim names:
- the
mcp_contractclaim - the
mcp://tier/*audiences - the algorithm policy
Your MCP server must handle MCP authorization itself. The broker does not provide these MCP endpoints. The shipped gateway does. You need:
- protected-resource metadata
- authorization-server discovery
- resource indicators
- checking the token's intended audience
- the right HTTP authorization challenges
See MCP authorization requirements.
Calling the broker
For each tool call that needs a vendor:
- Check and authorize the MCP tool operation in your server.
- Map the operation to an approved vendor and the vendor scopes it needs.
- Have the trusted gateway call resolve with the internal hub JWT.
- On success, use the token only for the approved vendor API call. Keep it out of tool results, model context, browser storage, and logs.
- Return business data to the MCP client. Never send the hub JWT to the vendor.
The broker gives a token to any caller that passes its internal authentication. So you must isolate the network and authenticate the gateway workload when you deploy. The REST response cannot enforce this for you.
Connecting a vendor during a tool call
The shipped gateway already does this. See its consent behavior. The guidance below applies to any adapter.
A missing vendor connection is a different problem from the client's permission to call the MCP server. Handle it separately.
How to ask the user to connect (MCP 2026-07-28):
- If the client advertises
elicitation.url, put a URL-modeelicitation/createinside anInputRequiredResult. - Set its URL to the broker's
authorize_uri. - The MCP bearer token stays the same.
- The client asks the user for permission before it opens the URL.
- Use the SDK for the negotiated protocol revision to build the full envelope.
See URL-mode elicitation.
The nested request looks like this. This is an example: the URL comes from resolve, and elicitationId is a fresh random ID for each request. MCP 2025-11-25 requires it in URL mode, and the shipped gateway always sends one. Clients on 2025-11-25 get the same request directly as elicitation/create during the call.
{
"method": "elicitation/create",
"params": {
"mode": "url",
"message": "Connect your vendor account to continue this operation.",
"url": "https://broker.example.com/v1/authorize/acme?txn=opaque-handle",
"elicitationId": "a-fresh-random-id"
}
}
What the adapter should do with each broker result:
| Broker outcome | Adapter action |
|---|---|
| 200 | Call the approved vendor with the returned token |
400 invalid-request |
Fix the resolve body your adapter sends. Do not ask the user to connect |
400 sub-mismatch |
Drop or fix the sub field. The broker uses the hub JWT's subject |
401 invalid-hub-token |
Fix internal authentication. Do not assume the MCP client's own token expired |
403 scope-exceeds-ceiling |
The tool asks for scopes the vendor's registry entry does not allow. Fix the tool mapping or the registry. Do not loop through consent |
404 needs-consent with authorize_uri |
Ask for a vendor connection through the supported browser flow |
404 unknown-vendor (no authorize_uri) |
Configuration error: the vendor is not registered or not enabled. Never show a connect prompt |
409 needs-reconsent-scope |
Explain the extra vendor permissions and use the supplied connection URL |
409 revoke-pending |
Say that a disconnect is pending. Do not restart consent automatically |
| 503 | Retry a limited number of times with backoff. Do not treat an outage as missing consent |
Tell the two 404s apart by authorize_uri, not by status: only needs-consent carries a link. Full list: Errors and caller actions.
flowchart LR
R["Resolve answer"] --> OK["200: use the token<br/>for this one call"]
R --> NC["404 with authorize_uri:<br/>ask the user to connect"]
R --> UV["404 unknown-vendor:<br/>configuration error"]
R --> RS["409 needs-reconsent-scope:<br/>ask to reconnect with the link"]
R --> RP["409 revoke-pending:<br/>say a disconnect is in progress"]
R --> UA["401: handoff configuration error"]
R --> UN["503: retry later, never connect"]
Only two answers lead to a connect prompt, and both carry a link. Every other failure is either a configuration problem or an outage.
After the browser flow:
- Confirm the connection before you resolve again, with the same user and required scopes.
- The user accepting the prompt does not prove the connection worked.
- Each failed resolve creates a new connection link. So while you wait, poll the grant list (
GET /v1/grants), not resolve. Wait for the vendor to showstateACTIVE. The grant list reports a grant that is mid-refresh asACTIVEtoo. - Treat a 5xx or unreachable broker while polling as an outage (retry later), never as "not connected yet".
- A suggested policy: at most one consent retry per operation, then return an error the user can act on.
- Stop if the user declines or cancels.
- If the client does not support URL elicitation, return a clear tool error. Point to a manual connection route that your application supports.
- Never ask for vendor tokens in chat or form fields.
Offering disconnect. An adapter can let the user disconnect a vendor with the broker's self-service DELETE /v1/grants/{vendor}/{sub} and the hub JWT. Take sub from that hub JWT: the broker refuses any other subject (403 forbidden). Handle each answer:
| Broker answer | Tell the user |
|---|---|
200 {"revoked": true} |
Disconnected: the vendor cancelled the token, and the broker deleted its copy |
200 with "vendor_revocation": "unsupported" |
Disconnected here only. The vendor can't cancel tokens remotely, so the user should also remove the app's access in the vendor's settings |
404 no-grant |
Nothing was connected |
502 revoke-pending |
The vendor could not be reached. The broker keeps retrying, and the vendor can't be used until it succeeds |
5xx (such as 503 vault-unavailable or coordination-unavailable) |
Retry later |
The shipped gateway does this as disconnect_<service> (Disconnecting a service).
What the shipped gateway's tests cover. Use them as a model for your own. tests/integration/test_mcp_gateway.py drives the gateway with a real MCP client against Keycloak and the stand-in servers. It covers:
- consent in both protocol revisions (2025-11-25 and 2026-07-28), and a connected user never being prompted;
- declining (
test_declining_leaves_github_disconnected) and a link opened by someone else (test_link_opened_by_someone_else_never_connects); - a client without URL elicitation (
test_client_without_url_elicitation_is_given_the_link); - a revoked connection prompting again, and
disconnect_githubrevoking at the vendor first; - broker and vendor outages being retryable and never asking to connect;
- parallel calls never burning a rotating refresh-token family;
- each vendor getting only its own token, and tokens working only at the server they were issued for;
- tokens not issued for the gateway being refused;
- no token material in any container log.
Unit tests (tests/unit/test_gateway.py) add consent timeouts, polling the grant list instead of resolve, and a broker outage while waiting.
The shipped gateway never sends required_scopes, so it never asks for extra scopes and never sees needs-reconsent-scope. If your adapter asks for scopes per tool, test that case yourself.
List all tools from the start. Do not count on notifications/tools/list_changed to reveal tools after a connection. From 2026-07-28, servers may only send it on subscriptions/listen. The gateway's pinned tool list shows this approach.
Existing gateway compatibility
Existing gateway plugins can keep calling the internal broker API as they are: its statuses, fields, and problem titles are frozen.
Some existing gateway plugins turn broker 404/409 consent results into custom 401 authorization_required and 401 step-up responses. That is an adapter convention, not a tested MCP flow.
If you are keeping an existing deployment, see the legacy mapping.
Protocol revision and enterprise authorization
Use the revision your MCP implementation negotiated for capabilities and message envelopes. The 2026-07-28 release:
- adds per-request protocol capabilities, Multi Round-Trip Requests, and routing headers
- deprecates Dynamic Client Registration in favor of Client ID Metadata Documents
Your integration layer handles these, not the broker. See the release notes.
Enterprise-Managed Authorization is an optional extension. It involves the client, the enterprise identity provider, and the resource's authorization server.
- This broker does not implement ID-JAG exchange.
- The registry field
ema_statusonly tracks how ready a vendor is to migrate. Changing it does not turn off consent or remove tokens. - If you consider it as a replacement, check that it also gives your tools the vendor access they need.
Broker API
The broker's internal REST API: requests, answers, and what to do on each error.
source: docs/api.md
Use this page if you are writing your own gateway that calls the broker. It is the main reference for the broker's internal REST API.
- This is not the MCP protocol. MCP clients never talk to the broker directly.
- The shipped MCP gateway serves GitHub's, Linear's, Atlassian's, and Cloudflare's MCP servers and already calls this API for you. Use it if those are the servers you need.
- Connect your own MCP server explains how an MCP server or gateway sits in front of the broker.
Authentication and routes
The broker has exactly seven routes. Most of them need a hub JWT.
- The hub is your company's sign-in service (identity provider). A hub JWT is the signed token it issues for a user.
- Send
Authorization: Bearer <hub-jwt>on resolve, list, delete, and admin requests. - The broker trusts the JWT subject (
sub) as the user's identity. - The consent routes do not use a bearer token. They use short-lived transaction state and a cookie that ties the flow to one browser.
/healthzneeds no authentication.
| Method | Route | Purpose |
|---|---|---|
| POST | /v1/tokens/resolve |
Get a vendor token for the caller |
| GET | /v1/authorize/{vendor}?txn=… |
Start browser consent |
| GET | /v1/callback/{vendor} |
Finish one consent step; _hub is the callback for the hub sign-in step |
| GET | /v1/grants |
List the caller's connections, without any tokens |
| DELETE | /v1/grants/{vendor}/{sub} |
Disconnect the caller from a vendor |
| GET | /v1/admin/vendors/{vendor} |
Read one registry entry; the JWT's groups array must contain ADMIN_GROUP |
| GET | /healthz |
Report whether this replica's custody token is healthy |
flowchart LR
subgraph callers["Gateway, with a hub JWT"]
R["POST /v1/tokens/resolve"]
L["GET /v1/grants"]
D["DELETE /v1/grants/{vendor}/{sub}"]
end
subgraph browser["Browser, no token"]
A["GET /v1/authorize/{vendor}"]
C["GET /v1/callback/{vendor}<br/>and /v1/callback/_hub"]
end
subgraph admin["Admin, with ADMIN_GROUP"]
V["GET /v1/admin/vendors/{vendor}"]
end
H["GET /healthz<br/>load balancer, no auth"]
R -.->|authorize_uri| A
A --> C
Three kinds of caller use the broker. Only the resolve answer links them: its authorize_uri is what the user's browser opens to start consent.
The broker has no endpoint that issues tokens. It also has no JWKS, OpenAPI, Swagger, or ReDoc endpoint. tests/unit/test_routes.py checks this.
Hub JWT checks
On every request that carries a hub JWT, the broker checks:
- The signature, with a key from
HUB_JWKS_URIand an algorithm inHUB_ALGORITHMS. - The issuer (
HUB_ISSUER) and the audience (HUB_TIER_AUDIENCE). The token must carry exactly onemcp://tier/*audience. - The required claims
exp,iat,sub, andjti. Time checks allow 30 seconds of clock skew. - The
mcp_contractclaim, which must equalHUB_CONTRACT_VERSION.
A failed check returns 401 invalid-hub-token. If the broker cannot fetch the hub's signing keys, it returns 503 hub-unavailable.
Resolve a token
Call resolve to get a live vendor token for the signed-in user. The broker may refresh the token during the call.
POST /v1/tokens/resolve
Authorization: Bearer <internal-hub-jwt>
Content-Type: application/json
{"vendor":"mockhub","min_ttl_s":30,"required_scopes":["issues:read"]}
| Field | Default | Meaning |
|---|---|---|
vendor |
Required | ID of an enabled vendor in the registry |
sub |
JWT subject | Optional. If you send a non-empty value that differs from the JWT subject, the broker rejects the request |
min_ttl_s |
120 | Non-negative integer. How long the token should still be valid, in seconds. Capped at REFRESH_BUFFER_S |
required_scopes |
[] |
Array of scope strings. The vendor's scope policy decides how these are used |
On success the broker returns HTTP 200 with:
access_token— the vendor token. Keep it inside the trusted gateway.expires_at— expiry time in Unix seconds. It may have a fractional part.granted_scopes— the scopes the broker has recorded for this connection.
Lifetime is a target, not a guarantee.
- A vendor can issue a new token that lives less than
min_ttl_s. The broker still returns it. - Check
expires_atbefore you use the token. - If you ask for more than the refresh buffer, the broker caps the request. The
broker.resolveaudit event then includesmin_ttl_clamped_from. - When the broker serves a token that another request or replica has just refreshed, and that token has less than
min_ttl_sleft, the audit event includesshort_ttl. This is separate from capping.
Scope policy
Each vendor in the registry has a scope ceiling (scope_ceiling): the most scopes the broker will ever ask for.
When the ceiling is not empty:
- It caps both requested and recorded scopes.
- A request for any scope outside the ceiling fails with 403
scope-exceeds-ceiling. - If a caller sends no
required_scopes, first-time consent asks for the whole ceiling, so callers need not send scopes. - When the broker asks for more scopes, it asks for the scopes already held plus the new required ones, within the ceiling.
When the ceiling is empty, the vendor decides the scopes (GitHub works this way: the app's permissions are set at GitHub):
- The broker ignores
required_scopesfor that vendor. It neither requests nor enforces them, and never answers 403 or 409 for scopes. - Filtering the stored
granted_scopeslist does not reduce what the token can actually do.
flowchart TD
Q["required_scopes in the resolve request"] --> E{"Ceiling empty?"}
E -- yes --> V["Ignore required_scopes<br/>vendor decides the scopes"]
E -- no --> S{"All required scopes<br/>inside the ceiling?"}
S -- no --> F["403 scope-exceeds-ceiling"]
S -- yes --> N{"No required_scopes?"}
N -- yes --> W["Consent asks for the whole ceiling"]
N -- no --> R["Consent asks for required<br/>plus held, within the ceiling"]
The ceiling is a hard boundary: nothing a caller sends can go past it. An empty ceiling hands the whole decision to the vendor.
In both cases:
- If a vendor grants scopes beyond a non-empty ceiling, the broker drops them from the stored list. It records them in the
scope_widenedfield of thebroker.refreshorbroker.consent.completeaudit event. With an empty ceiling there is nothing to compare, so the field never appears. - Your gateway or server must still check what each operation is allowed to do.
Errors and caller actions
Errors come back as application/problem+json. Branch on the HTTP status and the title.
- Each error has
type,title, anddetail, plus any extra fields in the table below. - The
titleis stable. Do not parsedetail(it is for people) or thetypeprefix (it is configurable). - Request validation errors from the web framework may use FastAPI's 422 response shape instead.
| Status / title | Additional fields | Action |
|---|---|---|
400 invalid-request |
— | Fix the malformed resolve input |
400 sub-mismatch |
— | Use the signed-in user's subject |
400 invalid-transaction |
— | Get a fresh connection URL |
401 invalid-hub-token |
— | Fix the internal token's issuer, signature, expiry, audience, or contract |
403 scope-exceeds-ceiling |
required, ceiling |
Review the vendor's policy. Do not loop through consent |
403 forbidden |
— | Check the self-service subject or the admin group |
404 unknown-vendor |
— | Check the registry and whether the vendor is enabled |
404 needs-consent |
authorize_uri |
No usable connection. Connect or reconnect |
404 no-grant |
— | Nothing to delete. The connection is already gone |
409 needs-reconsent-scope |
authorize_uri, missing_scopes |
Ask the user for more vendor permissions |
409 revoke-pending |
— | Resolve cannot use a connection that is being revoked |
502 revoke-pending |
— | Delete could not revoke at the vendor. The sweeper will retry |
503 vendor-unavailable |
— | Retry with backoff. Check whether the vendor is up or refreshes are contending |
503 vault-unavailable |
— | Restore custody (token storage). Do not ask users to reconnect |
503 coordination-unavailable |
— | Restore Redis for the operations that need coordination |
503 hub-unavailable |
— | Restore access to the hub's JWKS or OIDC discovery, or the hub itself |
The consent callbacks are browser pages, so they mostly return short HTML pages:
- 400 — bad state, issuer, or browser
- 403 — wrong user
- 502 — the token exchange failed
- 503 — a dependency is down
One exception: if the hub sign-in callback cannot start the vendor step, it returns problem JSON 503 (vendor-unavailable, vault-unavailable, or coordination-unavailable). Consent internals explain how the broker handles state.
Browser consent
Consent is how a user connects their vendor account. It happens in the user's browser in two steps.
- The user opens the
authorize_urifrom resolve. - The broker checks who the user is through the hub.
- The broker then starts vendor consent for that same user.
Time limits:
- Authorize link: the user must open it within five minutes (fixed), and it starts only one flow. If
TXN_TTL_Sis shorter than five minutes, the link expires afterTXN_TTL_S. - Each consent step: the state for the hub step and for the vendor step lasts
TXN_TTL_Sseconds (default 600, ten minutes) from when that step starts. - Binding cookie: it lives
TXN_TTL_Sseconds.
The authorize route answers with its own problem JSON when it cannot start:
| Status / title | When |
|---|---|
400 invalid-transaction |
The link is unknown, expired, already used, or for another vendor |
404 unknown-vendor |
The vendor is not in the registry or not enabled |
503 hub-unavailable |
The broker cannot reach the hub's OIDC discovery document |
503 coordination-unavailable |
The broker cannot read or store consent state (Redis is down) |
Rules:
- The same browser must finish both steps. Keep cookies across redirects.
- The binding cookie is HttpOnly, SameSite=Lax, and scoped to
Path=/v1/callback. It gets the Secure flag only whenBROKER_PUBLIC_URLuses HTTPS. - Use the exact registered callback URLs.
- If the transaction expired, was cancelled, or was already used, start again with a new resolve or connection attempt.
List and disconnect
List. GET /v1/grants returns {"grants":[…]}.
- Each entry has
vendor,state,granted_scopes,vendor_user_id, andcreated_at. - The internal
REFRESHINGstate shows asACTIVE. - Vendors switched off through
enabled_envare left out, even if a stored connection exists. - Tokens are never included.
Disconnect. DELETE /v1/grants/{vendor}/{sub} removes a connection. The MCP gateway's disconnect_<service> tools call this route with the user's hub JWT.
- The
subin the path must match the JWT subject, or the broker answers 403forbidden. URL-encode it when you build the path. - The broker revokes at the vendor first, then deletes its own copy.
- It holds the connection's refresh lock while it does this. If it cannot get the lock within
LOCK_TIMEOUT_S, it answers 503vendor-unavailable("grant is being refreshed; retry"). - If the stored token pair changes after the vendor revoke, the broker revokes the newer pair too. It tries up to three times, then answers 503
vendor-unavailable. - If the vendor is down, the broker marks the connection
REVOKE_PENDINGand answers 502revoke-pending. The sweeper retries the revoke. - Success returns
{"revoked":true}. - If the vendor does not support revocation, success also includes
"vendor_revocation":"unsupported". The broker deletes its own copy, but the vendor may still honour the permissions. Disconnect at the vendor too if you need to.
Admin
GET /v1/admin/vendors/{vendor} returns that vendor's registry entry.
- The caller's
groupsclaim must be a list that containsADMIN_GROUP. Otherwise the answer is 403forbidden, audited asbroker.admin.deny. - Any key whose name contains
secretis removed from the answer. - Vendors switched off through
enabled_envare still returned. Only a vendor missing from the registry gives 404unknown-vendor.
Health
GET /healthz normally returns {"ok":true,"custody":"ok"}.
- If custody is down, it can still return HTTP 200 with
"custody":"unreachable". This keeps replicas serving valid cached entries. - If the custody credentials are rejected or nearly expired, it returns 503.
- The broker caches health responses for ten seconds.
See the runbook for what to do next.
Legacy gateway mapping
Some existing gateway plugins translate broker results as shown below. It is an adapter convention, not part of this repository. tests/integration/test_wire_compat.py covers the broker side.
| Broker result | Legacy gateway behavior |
|---|---|
| 200 | Replace upstream Authorization with the vendor access token |
404 with authorize_uri |
Custom 401 authorization_required challenge |
409 with authorize_uri |
Custom 401 step-up challenge |
| Plain 409 | 403 carrying the problem title |
| 5xx | Retriable 503 vendor_unavailable |
For a new MCP adapter, follow the integration guide instead. Change the client-facing protocol if you need to, but keep this internal API as it is.
Deploy and Operate
Set up storage, configure the broker, and fix common problems.
source: docs/operations.md
This page shows how to set up and run the broker in a private deployment.
- To try it on your own machine, use the quickstart.
- For how callers use the broker, see the API reference.
- Before rollout, read the security limits.
- If you are deploying the MCP gateway, you need the broker too. The gateway section below covers what changes.
Production topology
flowchart LR
C["MCP clients<br/>Claude Code and others"] --> G["MCP gateway"]
G --> LB["Load balancer"]
LB --> B1["Broker replica"]
LB --> B2["Broker replica"]
B1 & B2 --> R[("Redis<br/>locks, consent state")]
B1 & B2 --> K[("OpenBao / Vault<br/>vendor-tokens<br/>vendor-clients")]
B1 & B2 --> H["Hub<br/>JWKS, OIDC sign-in"]
G --> H
B1 & B2 --> V["Vendors<br/>OAuth endpoints"]
G --> M["Vendor MCP servers"]
The gateway is the only caller of the token routes. Browsers call the consent routes, and admins call /v1/admin. Every replica shares one Redis and one secrets store. No session affinity is needed. The gateway talks to the vendors' MCP servers, and the broker talks to the vendors' OAuth endpoints. A single replica can run without Redis (COORD_BACKEND=memory).
Set up token storage (OpenBao / Vault, KV v2)
The broker keeps vendor tokens in a secrets store (OpenBao or Vault, KV v2 engine). You create two storage paths ("mounts"), each with its own access rules:
| Mount | Holds | Broker access | Admin access |
|---|---|---|---|
vendor-tokens |
Each user's vendor tokens ("grant entries") | Read and write | — |
vendor-clients |
Each vendor's client credentials (client ID and secret or key) | Read only | Write only |
The supplied broker policy does not limit what storage administrators can read. Use your own deployment policies to restrict human read access.
bao secrets enable -path=vendor-tokens kv-v2
bao secrets enable -path=vendor-clients kv-v2
bao write vendor-tokens/config max_versions=2 # retain up to two credential versions
bao policy write broker deploy/openbao-policy.hcl
bao token create -policy=broker -period=24h -orphan -display-name=broker
The broker's storage token
- Give the broker the scoped token from the last command, in
VAULT_TOKENorVAULT_TOKEN_FILE. Never give it the root token. - Use a periodic token, as shown above. The broker renews its own token at half the remaining TTL. A periodic token can be renewed forever.
- A token with a hard maximum TTL (an explicit max TTL such as
-explicit-max-ttl=768h, or the auth mount's or system's max TTL) can only be renewed up to that maximum./healthzstarts failing 10 minutes before it expires. You must then replace the token and restart the broker. - The token also needs the built-in
defaultpolicy. It grantslookup-selfandrenew-self. The policy is attached unless you pass-no-default-policy.
Storage auditing
Turn on an audit device in the storage backend. Check that read events reach your audit sink. The broker's own audit events do not replace storage auditing.
Startup checks
Before it serves requests, the broker checks three things: its storage token, the hub's signing keys (JWKS), and the hub's OIDC discovery document. The hub is your company's sign-in service (identity provider).
- If the storage backend or hub is unreachable, the broker retries for
STARTUP_TIMEOUT_S. - Startup stops at once, with a message that names what to fix, when:
- the storage token is rejected,
- the hub JWKS is unusable (for example, it has no keys),
- the hub discovery document is malformed or names an issuer other than
HUB_ISSUER, - a vendor ID in the registry does not match the allowed pattern (IDs become storage path segments).
Vendor client credentials
Write each vendor's confidential client credential on the admin path:
bao kv put vendor-clients/github client_id=… client_secret=…
bao kv put vendor-clients/acmehub client_id=… private_key=@key.pem alg=RS256
A vendor with enabled_env in the registry (for example GitHub: GITHUB_CLIENT_ID, Atlassian: ATLASSIAN_CLIENT_ID) is only active when that variable is set on the broker.
Linear, Atlassian, and Cloudflare (their MCP servers' own sign-in) have no developer console: tools/register-mcp-client.py <vendor> --redirect-base <broker URL> --vault registers the broker once and writes vendor-clients/<vendor> itself. Register again only when you must: it disconnects everyone. Linear and Cloudflare secrets expire (see Client secret expiring). See Servers with their own sign-in.
The credential must match the vendor's token_endpoint_auth_method in the registry:
token_endpoint_auth_method |
Needs |
|---|---|
client_secret_post or client_secret_basic |
client_secret |
private_key_jwt |
private_key (PEM; PKCS#8 or PKCS#1). Optional: alg (default RS256) and kid |
If storage goes down
The broker fails closed. Resolve returns 503 vault-unavailable. The only grace is each replica's in-memory cache (CACHE_TTL_S, default 60s).
Configuration reference
The broker reads its settings from environment variables.
Required. If any are missing, startup stops and lists every missing name:
HUB_ISSUER, HUB_JWKS_URI, BROKER_PUBLIC_URL, VAULT_ADDR,
VAULT_TOKEN or VAULT_TOKEN_FILE, REGISTRY_PATH, HUB_LOGIN_CLIENT_ID.
If both VAULT_TOKEN_FILE and VAULT_TOKEN are set, the file wins. If the file cannot be read, startup stops.
Optional, no default value shown in the table:
HUB_DISCOVERY_URL: where to fetch the hub's OIDC discovery document. Use it when the broker reaches the hub by an internal URL. Defaults toHUB_ISSUER/.well-known/openid-configuration. The document must still nameHUB_ISSUER.HUB_LOGIN_HINT:sub(the default) sends the user's subject as the OIDClogin_hint. Setnonefor a real IdP such as Keycloak, which expects a username.HUB_LOGIN_CLIENT_SECRET: leave it out for a public client. The broker always uses PKCE.
Optional, with defaults:
| Variable | Default | Notes |
|---|---|---|
HUB_TIER_AUDIENCE |
mcp://tier/internal |
The hub token must have exactly one mcp://tier/* audience |
HUB_ALGORITHMS |
PS256,ES256 |
Allowed: PS256/384/512, ES256/384/512, EdDSA. RS256, HMAC, and none fail startup |
HUB_CONTRACT_VERSION |
1.0 |
Required value of the mcp_contract claim |
ADMIN_GROUP |
mcp-platform-admin |
Callers of /v1/admin/* must have this in their groups array claim |
PROBLEM_URN_PREFIX |
urn:vendor-token-broker |
Changes only the type prefix. title slugs never change |
COORD_BACKEND |
memory |
Use redis for more than 1 replica |
REDIS_URL |
redis://localhost:6379/0 |
Used by the redis backend |
REFRESH_BUFFER_S |
300 |
A resolve refreshes the token if it expires within this many seconds |
PROACTIVE_REFRESH_S |
900 |
Upper edge of the window where the sweeper refreshes tokens ahead of time |
TXN_TTL_S |
600 |
Lifetime of consent state: each consent step's state and the browser binding cookie. An authorize link must still be opened within 5 minutes, or within TXN_TTL_S if that is shorter |
CACHE_TTL_S |
60 |
Per-replica cache lifetime, and so the most grace during a storage outage. The broker does not cap it. Keep it ≤ 60 to keep the documented bound |
SWEEP_INTERVAL_S |
60 |
0 turns off the sweeper. Every other integer setting must be ≥ 1 (CACHE_TTL_S may be 0: no cache) |
SWEEP_MAX_ENTRIES |
500 |
Entries checked per sweep pass. The next pass starts where this one stopped |
LOCK_TIMEOUT_S / LOCK_TTL_MS |
10 / 20000 |
How long a waiter waits for the lock / how long a redis lock lives. With COORD_BACKEND=redis, startup rejects LOCK_TTL_MS below (VENDOR_TIMEOUT_S + 2 × VAULT_TIMEOUT_S) × 1000 |
REFRESHING_TTL_S |
30 |
After this, another replica may take over an abandoned refresh marker |
MASS_STALE_THRESHOLD / MASS_STALE_WINDOW_S |
3 / 60 |
How many entries at one vendor must go stale within the window to raise the uninstall alert (broker.stale.mass) |
VAULT_TIMEOUT_S |
3 |
Time limit on storage calls, so an outage is detected quickly and fails closed |
VENDOR_TIMEOUT_S |
10 |
Time limit on each vendor HTTP call (metadata, token, userinfo, revocation) |
JWKS_TIMEOUT_S |
5 |
Time limit on fetching the hub JWKS, the hub OIDC discovery document, and the hub token-endpoint call during consent |
STARTUP_TIMEOUT_S |
30 |
How long startup waits for an unreachable storage backend, hub JWKS, or hub OIDC discovery |
Hub login client (consent)
When a user connects a vendor ("consent"), they must sign in at the hub in the same browser that opened the authorize link (consent). For this, register the broker at the hub as an OIDC client:
- Redirect URI:
{BROKER_PUBLIC_URL}/v1/callback/_hub(exact match). - Grant: authorization code, with PKCE (S256) required. Scope
openid. - ID tokens: signed with an algorithm in
HUB_ALGORITHMS, by keys the hub publishes atHUB_JWKS_URI. - Client auth: confidential (
HUB_LOGIN_CLIENT_SECRET, sent asclient_secret_post) or public (no secret).
The broker finds the hub's endpoints in its OIDC discovery document, at HUB_DISCOVERY_URL if set, otherwise at {HUB_ISSUER}/.well-known/openid-configuration. At startup it checks that this document names HUB_ISSUER.
Serve the broker over https. The consent cookie that ties the flow to one browser is then marked Secure.
Vendor registry review workflow
The registry (REGISTRY_PATH) lists the allowed vendors and each vendor's policy. Treat every change as a reviewed change:
- Edit a copy of
registry.example.json. - Check it against
schemas/vendor-registry.schema.jsonwithjsonschema. CI checksregistry.example.jsonon every push (tests/unit/test_registry.py). Do the same for your copy locally. - Get the same sign-off for a
scope_ceilingchange as for a token-contract change. The ceiling is a security boundary: a caller can never go past it at runtime. - Prefer
auth_metadata_url(RFC 8414). Hardcode endpoints only when the vendor documents no authorization server metadata. GitHub is the one case: it serves metadata that its documentation does not yet describe (see design §2.1). - Set
resourceonly for a vendor whose tokens are bound to one MCP server (Linear, Atlassian, Cloudflare). The broker sends it on authorize, code exchange, and every refresh (RFC 8707), unlessresource_on_refreshis false (Linear, which refuses it on refresh). Changing either later can break existing connections at their next refresh. revocation.typeisrfc7009orgithub_grant. For GitHub Enterprise Server, setrevocation.grant_url(defaulthttps://api.github.com/applications/{client_id}/grant).- Never put credentials in the registry. They live only in
vendor-clients/*.
Gateway contract
Gateways that call the broker depend on a frozen contract. That includes the shipped MCP gateway (src/mcp_gateway) and any other gateway plugin. Do not change it:
- Resolve statuses 200, 404, 409, and 5xx.
- Response fields
access_token,authorize_uri, andmissing_scopes. - Problem
titleslugs. Only the URN prefix intypeis configurable (PROBLEM_URN_PREFIX). - Audit event names (below).
For the fields and titles in detail, see the API reference and the legacy gateway mapping. For the protocol boundaries and how a gateway asks the user to connect during a tool call, see Connect your own MCP server.
Frozen audit event names: broker.resolve, broker.consent.start,
broker.consent.complete, broker.consent.fail, broker.refresh,
broker.stale, broker.stale.mass, broker.revoke, broker.admin.deny,
broker.sweep.error, broker.custody.renew_failed. These names never change.
Runbook
Each entry says what you see and what to do.
Which 503 is it?
flowchart TD
T{"503 problem title"} --> VA["vault-unavailable"]
T --> CO["coordination-unavailable"]
T --> HU["hub-unavailable"]
T --> VE["vendor-unavailable"]
VA --> VA2["Check OpenBao / Vault at VAULT_ADDR<br/>and /healthz custody field"]
CO --> CO2["Check Redis at REDIS_URL"]
HU --> HU2["Check the hub JWKS<br/>and OIDC discovery"]
VE --> VE2["Check the vendor, its client credential,<br/>and refresh lock contention"]
Read the problem title, not only the status. Each title points at one dependency:
vault-unavailable: see 503vault-unavailableand/healthzbody says unreachable.coordination-unavailable: see 503coordination-unavailable.hub-unavailable: the broker cannot fetch the hub's signing keys (HUB_JWKS_URI) or its discovery document. Check the hub and the network path to it. Calls are limited byJWKS_TIMEOUT_S.vendor-unavailable: the vendor is down or slow (VENDOR_TIMEOUT_S), its client credential is missing fromvendor-clients/<vendor>, or refreshes of one connection are contending (a waiter timed out afterLOCK_TIMEOUT_S). Retrying with backoff is safe. See also A replica died mid-refresh.
/healthz returns 503
/healthz returns {"ok", "custody"}. It returns 503 only when this replica's own storage token has a problem:
token-rejected: the token is revoked, expired, or wrong.token-expiring: under 10 minutes left.
Do this: replace the token and restart the replica.
/healthz body says "custody": "unreachable"
The storage backend is down. /healthz still returns 200 on purpose. Every replica shares the outage. Draining or restarting them would throw away the cache hits that keep serving meanwhile.
Do this: restore the storage backend. Alert on the body, not only the status code. The health result is cached for 10 seconds.
broker.custody.renew_failed
The broker could not renew its storage token. It keeps retrying with backoff. ttl_remaining_s says how long is left.
Do this: if the backend is up, check the token's policies and max TTL.
broker.consent.fail with reason: login_sub_mismatch
This event has security_event: true. Someone opened an authorize link issued to another user. sub is the user the link was for. login_sub is who signed in.
Do this: look at the pattern.
- One event can be a user opening a colleague's link by mistake.
- Repeats from one
login_sub, or links sent to many users, suggest a linking attack.
broker.consent.fail with reason: browser_mismatch
Someone finished a consent step in a browser that did not start it, for example through a forwarded callback or vendor URL. The broker did not redeem the code.
broker.stale.mass
Many entries at one vendor went stale at once. The likely cause is an org-level app uninstall or credential rotation at that vendor. The broker does not send pages itself.
Do this:
- Set up monitoring to route this event to on-call.
- Investigate the cause at the vendor before you notify users in bulk.
- Each user recovers by connecting again (re-consent) themselves.
503 vault-unavailable
The storage backend is down. The broker fails closed by design.
Do this: restore the backend. Entries and generations stay intact (the storage write check, CAS, protects them).
503 coordination-unavailable (redis backend)
Redis is down. Operations that need coordination fail: refresh, minting consent links, the authorize and callback routes, and DELETE. Every resolve that needs no refresh keeps serving, whether it comes from the cache or from a storage read.
Do this: restore Redis. Consent records lost with Redis need a fresh browser flow. Stored credentials stay safe in storage.
A replica died mid-refresh
Do nothing. The broker recovers on its own:
- The lock TTL expires.
- Another replica takes over the abandoned
REFRESHINGmarker. - In the worst case, a rotating refresh-token family burns. The entry goes STALE and the user re-consents.
CAS stops a losing writer from overwriting a newer version. It cannot recover credentials already used up at the vendor.
Sweeper errors or missed proactive refreshes
The sweeper refreshes tokens before they expire. With the redis backend, only the replica holding the leader lease runs it.
- A
broker.sweep.errorevent for one entry carries that entry'svendorandsub. A bad entry never stops the loop. Abroker.sweep.errorwithoutvendororsubmeans the whole pass failed. The loop tries again on the next tick. - Each pass lists the entries of every enabled vendor, then checks at most
SWEEP_MAX_ENTRIESof them. Vendors switched off throughenabled_envare skipped. The next pass starts where the previous one stopped. - A full cycle over all entries fits in the 10-minute proactive window (
PROACTIVE_REFRESH_S − REFRESH_BUFFER_S) whiletotal entries ≤ SWEEP_MAX_ENTRIES × 600 / SWEEP_INTERVAL_S(5,000 with the defaults). - With the redis backend each interval is jittered by ±20%. In the worst case (every interval 20% longer) the bound is
SWEEP_MAX_ENTRIES × 600 / (1.2 × SWEEP_INTERVAL_S), about 4,166 with the defaults. - Past that, some entries miss the proactive refresh. They are refreshed on their next resolve instead.
502 revoke-pending on disconnect
The service was down when someone disconnected. The broker marked the connection REVOKE_PENDING and kept its tokens, so it can revoke them later.
- Meanwhile, resolve answers 409
revoke-pendingfor that connection. - The sweeper retries the revoke on each pass. It takes the entry's lock, reads the entry again, and revokes at the service. It then deletes only the version it revoked.
- A repeated DELETE from the person also retries the revoke.
Do this: check that the broker can reach the service's revocation endpoint. The broker.revoke event with outcome: pending carries the error. The sweeper does not run when SWEEP_INTERVAL_S is 0.
Client secret expiring (Linear, Cloudflare)
tools/register-mcp-client.py registers the broker with Linear's and Cloudflare's MCP sign-in. Their client secrets expire after about 90 days. The test registrations expire on 2026-12-27 (Linear) and 2026-12-26 (Cloudflare). The script prints the expiry date when it registers.
Do this before that date:
- Register again:
tools/register-mcp-client.py <vendor> --redirect-base <broker URL> --vault --force. It writes the new credential tovendor-clients/<vendor>. - Tell people to reconnect. Every connection made with the old client stops working.
Slow requests while storage is frozen
Storage (hvac) and hub JWKS calls run in worker threads, never on the event loop. A frozen storage backend holds one thread per in-flight uncached request for up to VAULT_TIMEOUT_S. Cache hits and /healthz keep answering.
Storage layout and upgrades
Grant entries live at vendor-tokens/{vendor}/sub-b64.{base64url(sub)}. There is one key per subject, whatever characters the hub puts in sub (including URIs with /).
Entries may also sit at the unencoded key vendor-tokens/{vendor}/{sub}:
- The broker reads both keys.
- It moves an entry to the encoded key on its next refresh or re-consent. A delete removes both keys.
- You need no migration job.
Run one broker version across all replicas. A replica without encoded-key support answers needs-consent for moved entries.
What stays in storage:
- A current STALE entry holds no tokens (both are blanked). Older retained KV versions may still hold credentials.
REVOKE_PENDINGentries keep their tokens until the vendor revocation succeeds.
MCP gateway
The MCP gateway is a separate deployment (Dockerfile.gateway, requirements-gateway.lock). It has its own configuration, hub requirements, audit events, and limitations: see MCP gateway.
Two broker settings matter when you run it with a real IdP:
- If the broker reaches the hub by an internal address that differs from the public issuer (as in the Keycloak profile), set
HUB_DISCOVERY_URL. - Set
HUB_LOGIN_HINT=nonefor a real IdP.
Single-replica deployment
deploy/docker-compose.yml is the single-replica shape. Next to it, put .env (a copy of .env.example; every variable in it reaches the broker, including the vendor client IDs that enable registry vendors), your reviewed registry.json, and broker-vault-token. The image runs as UID 65534, and compose bind-mounts the token file with its host owner and mode: on a Linux host a 0600 file is unreadable, so make it readable by that user (chown 65534, or chmod 0444). The broker serves plain HTTP on 8300; terminate TLS in front of it and set BROKER_PUBLIC_URL to the https address.
Multi-replica deployment
Run 2 or more replicas only with COORD_BACKEND=redis. See deploy/docker-compose.multi.yml for the shape: two replicas sharing OpenBao and Redis. Put any L4/L7 load balancer in front; no session affinity is needed.
Timing rules:
- Keep
LOCK_TTL_MSabove the slowest vendor token round-trip plus storage overhead. REFRESHING_TTL_Sshould be larger thanLOCK_TTL_MS / 1000. Watch the units: seconds versus milliseconds.
Sweep leader handoff: a stopped sweep leader can keep its Redis lease for up to twice its sweep interval (120 seconds with defaults). A successor must then reach its next sweep attempt. Allow for that handoff when you change profiles or timing settings. The test guide explains how to keep standalone and multi-replica runs apart.
Security
Which protections are built in, which are partial, and what we tested.
source: docs/security.md
Reviewed 2026-09-30 · software 1.1.0 (unreleased) · MCP 2026-07-28.
This page lists which security controls the broker and gateway have, where the proof lives, and what is still missing.
- The broker stores each user's vendor tokens and runs the vendor sign-in (OAuth) flows. It is one part of a larger MCP deployment.
- The shipped MCP gateway handles the MCP boundary for GitHub's, Linear's, Atlassian's, and Cloudflare's MCP servers.
- Passing these tests does not mean a complete deployment meets the MCP standard. The current specification sets the protocol baseline.
Control ownership and evidence
Status words in the table:
- Implemented: present in this broker (or the gateway, where named).
- Partial: present, with limits the row names.
- External: another component must provide it.
- Planned: not available here.
Source and test paths are relative to the root of the source repository, chief-builder/vendor-token-broker. Its THREAT_MODEL.md lists the assets, trust boundaries, threats, mitigations, and residual risks, with the test behind each control.
| Control | Status / owner | Implementation and evidence |
|---|---|---|
| PKCE S256, nonce, subject/browser binding | Implemented / broker | consent.py, hub_login.py; tests/unit/test_consent_binding.py, test_hub_login.py, tests/integration/test_consent.py |
| Vendor callback issuer validation | Partial / broker | consent.py callback checks; tests/integration/test_consent.py; see the metadata limits below |
| Hub JWT revalidation | Implemented / broker | The hub is your company's sign-in service. hub.py; tests/unit/test_hub.py, tests/integration/test_security.py |
| Hub signature algorithms | Implemented / broker + gateway | The broker's HUB_ALGORITHMS and the gateway's HUB_ALGORITHM are checked against one allowlist: PS256/384/512, ES256/384/512, EdDSA. Anything else (RS256, HMAC, none) stops startup. src/token_broker/config.py, src/mcp_gateway/config.py; tests/unit/test_config.py, tests/unit/test_gateway_auth.py (test_config_fails_fast) |
| Scope ceiling and scope union | Implemented / broker policy | resolve.py, refresh.py; tests/unit/test_scope_math.py, tests/integration/test_refresh.py |
| Per-user custody, generation CAS, single-flight | Implemented / broker + custody + coordination | custody.py, refresh.py, coordination.py; storage, refresh, and multi-replica tests |
| Revocation | Implemented with exceptions / broker + vendor | Vendor first, then the broker's copy. For RFC 7009 vendors the broker revokes the access token and then the refresh token. Not every vendor revokes the access tokens with the refresh token (Linear's live on for up to 24 hours). A 4xx on the access token is ignored; the refresh token's answer decides. GitHub: grant deletion. grants.py, vendors.py, sweeper.py; tests/unit/test_storage_hygiene.py, tests/integration/test_grants.py |
| No access-token issuance endpoint | Implemented / broker | tests/unit/test_routes.py checks there are exactly seven routes. The broker may still sign client assertions |
| Sensitive values absent from tested logs | Implemented test coverage / broker | tests/integration/test_security.py checks sample hub/vendor tokens, the client secret, and the assertion key. It does not prove every possible provider error is clean |
| Protected-resource metadata and MCP discovery | Implemented / gateway | mcp_gateway/server.py (build_auth); tests/unit/test_gateway_auth.py, tests/integration/test_mcp_gateway.py; checked with Claude Code 2.1.283 |
| Resource-specific token audience | Partial / gateway + hub | The gateway requires aud = its resource URL, a pinned algorithm, issuer, and scope (tested). Keycloak ignores the RFC 8707 resource parameter, so the audience comes from the required mcp-gateway scope |
| MCP authorization challenges | Implemented / gateway | 401 WWW-Authenticate with resource_metadata and scope. Every refusal is audited as gateway.auth with its reason |
| MCP client registration | External / hub | The development realm allows dynamic registration from localhost only, with consent (tested). It has no pre-registered Claude Code client: Claude Code registers itself. Production policy belongs to the IdP |
| Broker registration with vendor MCP sign-in | Implemented / operator tool | tools/register-mcp-client.py: one RFC 7591 registration per vendor, with the broker's callback and the registry's scope ceiling. It refuses to register again unless forced, never prints the secret, and warns about an expiring secret. tests/integration/test_register_mcp_client.py |
| Resource indicators to vendor MCP servers | Implemented / broker | The registry resource goes on authorize, code exchange, and every refresh (RFC 8707), unless resource_on_refresh is false (Linear). tests/unit/test_resource_indicator.py, test_vendor_client.py; tests/integration/test_mcp_resource.py against a mock that refuses a missing or wrong resource; accepted by the real Atlassian and Cloudflare servers |
| Identity handoff without token passthrough | Implemented / gateway + hub | RFC 8693 token exchange. The broker rejects a raw MCP token (tests/integration/test_keycloak_hub.py) |
| Vendor consent via URL elicitation | Implemented / gateway | Both protocol eras and the fallback for clients without the capability; tests/unit/test_gateway.py, tests/integration/test_mcp_gateway.py; checked with Claude Code 2.1.283 |
| Token-free MCP results and logs | Implemented for the shipped gateway | The leak test (tests/integration/test_mcp_gateway.py, test_no_token_material_in_any_container_log) greps every container's logs for the MCP token, the hub JWT, and the GitHub, Linear, and Atlassian stand-in tokens. Upstream error details are logged with the token replaced by <token>. Custom gateways must run their own test. Resolve returns a token to its trusted caller on purpose |
| Read-only tool allowlist | Partial / gateway + broker policy | Per-service allowlist, plus each service's read-only control: X-MCP-Readonly for GitHub, the read scope and tokens bound to the read-only MCP server for Linear (its full MCP server refuses them), read and search scopes for Atlassian. Write tools stay hidden (tested against the stand-ins). The gateway also sends GitHub X-MCP-Lockdown. That is GitHub MCP's lockdown mode: it limits content from untrusted users in public repositories. It does not block writes. The disconnect_<service> tools change only the person's own broker connection, never data at the service. Cloudflare relies on scopes alone: cloudflare_execute runs code against the whole API, and only the scope ceiling (12 read scopes plus offline_access) stops writes |
| Shared tool list never taken from one user | Implemented / gateway | Every user sees the saved tool lists. Live schemas that differ are logged, never listed, because Cloudflare writes the signed-in user's email and account ID into a description. tests/unit/test_gateway.py (test_drifted_schema_is_reported_but_the_snapshot_stays_listed), and test_gateway_auth.py refuses saved lists that contain an email address or a 32-character hex ID |
| Outage never asks to connect | Implemented / gateway | Broker and hub outages (unreachable or 5xx), including while the gateway waits for the person to finish connecting, end the call with a retryable "retry shortly" error, never a request to connect. A rejected hub token is reported as a gateway configuration problem. tests/unit/test_gateway.py (test_broker_failures_never_ask_to_connect, test_broker_outage_while_waiting_is_retryable_not_a_timeout), tests/unit/test_gateway_clients.py |
| User-initiated disconnect | Implemented / gateway + broker | disconnect_<service> calls the broker's self-service DELETE /v1/grants/{vendor}/{sub} with the person's hub JWT. The broker revokes at the vendor first, then deletes its copy. tests/unit/test_gateway.py (test_disconnect_*), tests/unit/test_gateway_clients.py (test_disconnect_outcomes), tests/integration/test_mcp_gateway.py (test_disconnect_revokes_at_the_vendor_and_the_next_call_asks_again) |
| Per-service token isolation | Implemented / gateway | Each service only receives its own vendor's token, and connecting one service never connects another. tests/unit/test_gateway.py (test_each_upstream_gets_only_its_own_vendors_token, test_connections_are_per_upstream), tests/integration/test_mcp_gateway.py |
| Issuer-bound vendor client credentials | Partial / configuration control | Storage keys use the vendor ID. A change of issuer does not invalidate stored credentials on its own |
| Enterprise-Managed Authorization / ID-JAG | Planned / cooperating identity systems | ema_status tracks readiness. There is no exchange and no automated drain |
What each component can see
flowchart LR
C["MCP client<br/>sees: MCP token only"] -->|MCP token| G["MCP gateway<br/>sees: MCP token, hub JWT,<br/>one vendor token per call<br/>stores no tokens"]
G -->|hub JWT| B["Broker<br/>sees: hub JWT, vendor tokens,<br/>vendor client secrets"]
B <-->|scoped storage token| K[("OpenBao / Vault<br/>vendor-tokens, vendor-clients")]
G -->|"that vendor's token only"| V["Vendor MCP server<br/>sees: its own token only"]
B -->|"refresh, revoke"| V
Each boundary narrows what passes through. The client never sees a hub JWT or a vendor token. The gateway gets one vendor token for one call and keeps none. Only the broker and the secrets store hold stored credentials. Each vendor sees only its own token.
Protected-resource metadata, resource indicators, and checking the intended audience belong at the MCP boundary. MCP authorization
Known limitations
Issuer discovery and callback checks
The broker does not fully check who issued a vendor's sign-in response (the "issuer"). In detail:
- Vendor discovery checks the shape of the endpoints. It does not require an issuer, or compare it with a separately trusted expected issuer.
- If no issuer is stored, the vendor callback skips the issuer comparison.
- The registry schema has no way to state issuer or support fields for explicit endpoints.
- The hub callback rejects an issuer that does not match. It does not reject a missing issuer when the metadata says one should be sent.
- Vendor credentials are stored per vendor ID, with no issuer stamp. Nothing blocks their reuse after a metadata change.
What to do: restrict who can change the registry and metadata, and review provider changes. These are follow-up code items. Documentation changes do not fix them.
So standards alignment here is partial. Current MCP callback checks depend on issuer metadata that is authenticated and checked. Authorization response validation
Scope and lifetime
- The broker can cap the scope names it requests and records. It cannot narrow a token the vendor issued by editing metadata.
- An empty ceiling leaves permission policy to the vendor.
- The minimum token lifetime is a target. It has clamping and exceptions for short-lived tokens.
See scope and lifetime semantics.
Revocation, historical storage, and recovery
- If the vendor does not support revocation, disconnecting removes only the local connection.
- If storage fails after the broker redeems a code, revoking the new grant is best-effort.
max_versions=2keeps up to two credential versions. Blanking the current STALE entry does not erase every older kept version.- CAS (the storage write check) stops a losing refresh write from overwriting a newer stored version. It cannot undo a refresh the vendor has already used. A replica failure at that point can burn a rotating-token family and force the user to re-consent.
- Cache invalidation is best-effort. Its limit is
CACHE_TTL_S(default 60 seconds).
Hub signing keys
- The broker trusts a fetched hub key set for up to 5 minutes (
JWKS_CACHE_Sinhub.py). A key the hub removes keeps validating until then.tests/unit/test_hub.pychecks both sides of that window.
MCP gateway
- It depends on FastMCP 4.0.10.
requirements-gateway.lockpins 80 packages by hash, FastMCP included. - Its JWKS cache can keep trusting a removed hub signing key for up to an hour.
- It cannot send tool-list changes to 2026-07-28 clients (no
subscriptions/listen). The pinned tool list works around this. - The local realm's self-registration policy is for development only.
Details: gateway limitations.
Deployment controls
The development Compose stack does not supply these controls. You must add them:
- Private ingress, and workload authentication for the gateway.
- TLS.
- Controlled storage policies.
- Audit logging set up in the storage backend.
The broker can sign private_key_jwt client-authentication assertions. It does not issue access tokens. Treat vendor client signing keys as credential material.
broker.stale.mass carries page: true, but that does not contact an alert service. Route the event to monitoring if you need an on-call notification. Availability and latency values in the design are targets, not measured guarantees.
Verification record
Review baseline: application commit e6c7f65 (the gateway serving GitHub, Linear, Atlassian, and Cloudflare, Linear through its own MCP sign-in). Checks on 2026-09-28:
| Check | Result |
|---|---|
| Unit suite | 433 passed |
| Docker integration / memory | 64 passed |
| Gateway profile (Keycloak hub, broker-kc, gateway, stand-ins for all four services) | 47 passed, including disconnect through the broker with a two-token vendor revoke |
Real Linear through the gateway (test_external_mcp_servers.py) |
Passed, through Linear's own MCP sign-in |
| Linear, checked by hand against the real service | Consent at mcp.linear.app for read only. A forced broker refresh (no resource) moved the entry from generation 1 to 2 and rotated the refresh token. The token works at the read-only MCP server and is refused by the full one. After disconnect_linear, Linear refuses the last access token |
Real Atlassian and Cloudflare (at 41e7aab, before disconnect_<service>) |
2 passed, with the broker registered by tools/register-mcp-client.py |
Claude Code 2.1.283 against the gateway (at 41e7aab) |
One session used all four real services: the Atlassian account and open Jira issues, the Cloudflare Worker's settings, Linear issues, and the GitHub profile. Linear then used an admin-created OAuth app |
Hardening review on 2026-09-30 (Python 3.14.7, refreshed locks, main.py split into broker, resolve, consent, and grants): unit suite 507 passed with 95% line coverage, mypy clean, pip-audit reported no known vulnerabilities in any lock. The Docker suites (memory, redis, gateway, multi-replica) run in CI on every push to main and every pull request; see the CI run for that change. AUDIT.md at the repository root records the full review.
How the single-flight tests stay exact
The 20-parallel single-flight tests (single replica and through the load balancer) expect exactly one vendor refresh. Mock tokens normally last 60 seconds, inside REFRESH_BUFFER_S (300), so a request that starts after the first refresh finished would correctly refresh again. For the burst, the tests switch the mock vendor to 1-hour tokens: the consent's 60-second token forces the one refresh, and every later request is served the new token.
A multi-replica run right after a standalone Redis broker can fail if that broker still holds its 120-second sweep lease, longer than the test's 60-second deadline. The documented test isolation avoids this.
What these checks do not show
- Most checks use the mock vendor and the stand-ins.
- The real-GitHub and Claude Code checks show that this one configuration works together. They say nothing wider.
- None of this shows production load capacity or penetration-test results.
Manual verification explains the individual probes.
Enterprise migration
Enterprise-Managed Authorization (EMA) is optional. It needs compatible clients, IdPs, and resource authorization servers.
- Check downstream vendor access before you retire a broker integration.
- Keep readiness evidence per vendor, with a review date.
- Changing
ema_statusalone has no runtime effect.
Design
The broker's full design: roles, rules, states, and failure handling.
source: docs/design.md
Software: 1.1.0 (unreleased) · Reviewed: 2026-09-30
This page is the normative design of the broker. It says how the broker must behave inside. For a gentler start, read the overview, the API reference, or security.
Where the broker sits
The broker keeps each user's tokens for outside SaaS vendors (GitHub, Linear, Atlassian, Cloudflare, …). It never talks to MCP clients directly. It sits behind a trusted gateway:
- the MCP gateway in this repo, which lets Claude Code and other MCP clients use the official GitHub, Linear, Atlassian and Cloudflare MCP servers, or
- any other trusted egress gateway (the service that makes outbound calls to vendors).
The gateway asks the broker for a user's vendor token. It then calls the vendor with that token. The user's browser talks to the broker only to connect a vendor account (the consent steps).
Terms used on this page:
- Hub — your company's sign-in service (the workforce identity provider, or IdP). It is the only service that issues platform tokens.
- Hub JWT — the signed token the hub issues for a signed-in user.
- Vendor AS — the vendor's authorization server (where users grant access and tokens come from).
- Custody — the secret store that holds tokens (OpenBao or Vault KV v2).
- Entry — one stored token record for one user and one vendor.
sub— the user id (the JWT "subject").- CAS — compare-and-swap: a write that succeeds only if the stored version has not changed since it was read.
Section 14 describes the two deployment profiles (one replica, or many
with Redis). token-lifecycle.md draws every lifecycle path on this page
as a sequence diagram.
1. Role definition
The broker gets, stores, refreshes, and revokes OAuth tokens for each user. Third-party SaaS authorization servers (GitHub, Linear, Atlassian, …) issue those tokens. On each request, the broker returns the right token to the egress gateway tier.
The broker is an OAuth client and it stores credentials. It is not an authorization server and MUST NOT issue access tokens.
- It may sign
private_key_jwtassertions to sign in to vendors. - The hub stays the only token issuer for the platform.
- If a future need seems to require broker-issued tokens, treat it as a design error. Escalate it. Do not implement it.
The broker supports vendors that need a separate user OAuth grant. MCP Enterprise-Managed Authorization (EMA) is a possible future migration path. It fits only if it also gives the downstream vendor access the application needs. This repo does not implement it. See the integration guide.
2. Standards basis
Each row shows a standard and what the broker must do to meet it.
| Leg | Standard | What the broker must do |
|---|---|---|
| Getting tokens | RFC 6749 authorization code + OAuth 2.1 rules | Use PKCE (S256) on every flow. Match redirect URIs exactly. No implicit or password grants |
| Getting tokens | RFC 7636 PKCE | Create the verifier and keep it on the server, one per transaction |
| Discovery | RFC 8414 AS metadata | Find vendor endpoints from .well-known. Hardcoding is forbidden where metadata exists (the registry schema enforces auth_metadata_url xor endpoints) |
| Getting tokens | RFC 9207 issuer identification | When a recorded issuer exists, the callback checks iss against the issuer recorded when the transaction started. Use strict string comparison, no URI normalization. If the AS advertises support but sends no iss, reject the callback as a mix-up signal |
| Getting tokens | RFC 8707 resource indicators | For a vendor whose tokens are bound to one MCP server (registry resource), send resource on the vendor authorize request and on every token request: code exchange and refresh (refresh only if resource_on_refresh is not false, §2.1) |
| Client auth | RFC 7523 private_key_jwt |
Prefer it where the vendor supports it. Otherwise use client_secret_basic or client_secret_post (set per vendor in the registry, default client_secret_post) |
| Hardening | RFC 9700 OAuth 2.0 Security BCP | Used as the review basis. Known limitations prevent a blanket conformance claim |
| Lifecycle | RFC 6749 §6 refresh | Only one refresh at a time per entry (single-flight, §8) |
| Lifecycle | RFC 7009 revocation | When a user removes a grant, call the vendor revocation endpoint before deleting the stored entry: the access token, then the refresh token. A 4xx on the access token is ignored (a server may not support revoking it); the last token's answer decides (the access token when there is no refresh token) |
| Exchange (future) | MCP EMA: ID-JAG via RFC 8693 token exchange + RFC 7523 grant | Possible future migration only. No exchange is implemented here |
private_key_jwt assertions (client_auth.py): iss = sub = the
client id, aud = the vendor's token endpoint, a fresh jti, and a
300-second lifetime. Revocation requests also use the token endpoint as
the audience (RFC 7523 accepts any identifier of the AS). The signing key,
alg (default RS256) and optional kid come from
vendor-clients/{vendor} in custody.
2.1 Per-vendor deviations
Some vendors do not follow the standards above. Each deviation is set in
the reviewed registry and handled in vendors.py:
| Deviation | Vendor | What the broker does |
|---|---|---|
| Explicit endpoints instead of RFC 8414 metadata | GitHub | The registry lists endpoints explicitly. GitHub now serves metadata at https://github.com/.well-known/oauth-authorization-server/login/oauth (observed 2026-09-30), but its documentation does not describe it, so the registry keeps the documented endpoints. Switching to auth_metadata_url would also make the callback require iss (RFC 9207), which that metadata advertises |
| Revocation by grant deletion, not RFC 7009 | GitHub (revocation.type: github_grant) |
DELETE https://api.github.com/applications/{client_id}/grant (registry revocation.grant_url overrides it, e.g. for GitHub Enterprise Server) with HTTP Basic (client id and secret) and the access token in the JSON body. 204, 404 and 422 count as revoked. Any other status is a vendor outage. An entry with no access token (a blanked STALE entry) needs no call. This deletes the whole user-to-app grant, which is why re-consent never revokes an ACTIVE or STALE predecessor (§4.3) |
| Token errors in a 200 response | GitHub | Any error field is a failure, whatever the HTTP status |
bad_refresh_token instead of invalid_grant |
GitHub | Treated exactly like invalid_grant (the entry goes STALE) |
| Scopes set by the vendor, not the request | GitHub App (empty scope_ceiling) |
The broker requests no scopes and ignores required_scopes (§4.1) |
| Tokens that never expire | GitHub App with expiry turned off | Stored with a far-future expiry and never refreshed (§5) |
| Tokens bound to one MCP server | Linear, Atlassian, Cloudflare (registry resource) |
RFC 8707 resource on authorize, code exchange and refresh (not on refresh for Linear, next row) |
resource refused on refresh |
Linear (resource_on_refresh: false) |
Sent on authorize and code exchange only. Linear keeps the refreshed token bound to the same MCP server |
| Revoking the refresh token leaves the access token alive | Linear | Why every RFC 7009 revocation also revokes the access token (§2 Lifecycle) |
3. Architecture and trust boundaries
The hub JWT never goes past the gateway and broker, and vendor tokens never reach the MCP client.
flowchart TB
subgraph userplane["User plane"]
C["MCP client"]
UA["Browser<br/>(consent steps only)"]
end
subgraph platform["Platform — private tier"]
S["MCP server"]
G["Egress gateway"]
B["Vendor Token Broker"]
K["Custody<br/>OpenBao / Vault KV-v2"]
R["Redis<br/>(multi-replica profile)"]
end
H["Workforce IdP — the hub<br/>ONLY token issuer"]
V["Vendor AS + API<br/>(GitHub, Linear, …)"]
C -- "MCP resource access token" --> S
S -- "allowed vendor operation" --> G
G -- "resolve — hub JWT<br/>re-checked by the broker" --> B
UA -- "authorize / callback<br/>(private ingress only)" --> B
B -. "JWKS + OIDC discovery<br/>and login code redemption" .-> H
B -- "OAuth steps, per-vendor<br/>client auth" --> V
G -- "vendor token upstream,<br/>hub JWT REMOVED" --> V
B -- "grants (CAS) +<br/>client credentials" --> K
B -.-> R
classDef broker fill:#6366f133,stroke:#6366f1
classDef vendor fill:#eab30833,stroke:#eab308
classDef custody fill:#10b98133,stroke:#10b981
classDef coord fill:#ef444433,stroke:#ef4444
classDef neutral fill:#94a3b833,stroke:#94a3b8
class B broker
class V vendor
class K custody
class R coord
class C,UA,S,G,H neutral
This repo's MCP gateway plays both the MCP server and the egress gateway role. Any other gateway must do the same jobs.
Every deployment needs these two boundaries:
- The hub JWT stops at the gateway and broker. The gateway removes it before anything goes upstream to a vendor.
- Vendor tokens must never reach the MCP client.
The gateway must enforce both. The internal hub JWT is a separate credential contract from MCP resource authorization. See three authorization boundaries.
Who calls whom, and how much each side is trusted:
- Egress gateway → broker (
/v1/tokens/resolve,/v1/grants,DELETE /v1/grants/…). The request carries the end user's hub JWT. The broker checks it again on its own (hub.py), in this order: - the signing key is found in the hub JWKS by
kid(JWKS unreachable → 503hub-unavailable, any other failure → 401), - one
jwt.decodecall checks together: the algorithm is inHUB_ALGORITHMS(default PS256, ES256; only PS256/384/512, ES256/384/512 and EdDSA may be configured), the signature,iss=HUB_ISSUER,audcontainsHUB_TIER_AUDIENCE,exp, and thatexp,iat,sub,jtiare present (30 s clock leeway), mcp_contract=HUB_CONTRACT_VERSION(default1.0),- exactly one audience starts with
mcp://tier/.
Any failure is 401 invalid-hub-token. This is defense in depth. The
broker never simply trusts the gateway. Production deployments add
mutual TLS with workload identity between gateway and broker.
- User browser → broker (/v1/authorize, /v1/callback). Private
ingress only. The callback host is never on the public tier.
- Broker → vendor AS. Outbound TLS. The broker signs in with its own
confidential client credential for that vendor, stored at
vendor-clients/{vendor}. It prefers private_key_jwt over shared
secrets where the vendor supports it.
- Broker → custody backend. The broker's policy is the only one this
repo defines for vendor-tokens/*. Keeping other roles (such as
admins) from reading it is up to the deployment. Operators must turn on
custody-backend audit logging to record reads. The broker does not emit
its own audit event for every storage read.
4. API (the whole surface — 7 routes)
The broker has exactly 7 routes: the six in §4.1–4.5 below, plus GET /healthz (health check, see docs/operations.md). Error and field rules for all of them:
- Errors use RFC 9457 problem details. Browser-facing consent steps
(
/v1/callback/…) answer with short HTML pages instead (§4.3). - No token material ever appears in an error body or log line.
- The
titleslugs and response fields are a frozen wire contract (seedocs/operations.md).
4.1 POST /v1/tokens/resolve
The gateway calls this route to get a user's vendor token. See resolve a token for request fields, lifetime exceptions, and scope policy. The full list of errors and caller actions is in the error/action table.
- The JWT subject (
sub) decides whose token is returned. A bodysubthat differs is 400sub-mismatch. min_ttl_s(default 120) is capped atREFRESH_BUFFER_S. A larger value is lowered, and the audit line recordsmin_ttl_clamped_from. Without the cap, a caller could force a vendor refresh on every call.- Resolve reads the local cache or custody (§7).
- If needed, it refreshes the token inline, during the request.
- §§7–8 explain how concurrent requests are handled.
Scope math. ceiling is the vendor's registry scope_ceiling, and
required is the request's required_scopes.
| Case | What the broker does |
|---|---|
Empty ceiling |
Scopes are governed by the vendor (for example GitHub App permissions). required is ignored. Consent requests no scopes |
required ⊄ ceiling |
403 scope-exceeds-ceiling. A caller can never widen past the ceiling |
| No entry, or STALE | 404 needs-consent. The consent link asks for required ∩ ceiling, or the whole ceiling when required is empty |
| REVOKE_PENDING | 409 revoke-pending. The entry is never used |
ACTIVE, but required has scopes the entry lacks |
409 needs-reconsent-scope with missing_scopes and an authorize_uri |
| Otherwise | Serve or refresh (§7) |
Union re-consent. The 409 link asks for the union of held and required scopes, capped by the ceiling, in ceiling order:
reconsent_scopes = [s for s in ceiling if s in (granted_scopes ∪ required)]
So a step-up never drops a scope the user already granted. It also never asks for more than the ceiling. The new consent overwrites the entry with a fresh generation 1 (§4.3).
The absent, STALE and 409 cases mint a consent link. That needs the
coordination store. In the redis profile, a Redis outage turns them into
503 coordination-unavailable (§9).
4.2 GET /v1/authorize/{vendor}?txn=…
This route starts consent (§6). The user opens this link in a browser.
- The link (
txn) is single use. It is taken atomically, and works only within 5 minutes of being minted (AUTHORIZE_LINK_TTL_S= 300). An unknown, used, expired or other-vendor link is 400invalid-transaction. - The broker sets a binding cookie
vtb_consent_{vendor}: a random value, HttpOnly,SameSite=Lax,Path=/v1/callback,Max-Age=TXN_TTL_S, andSecurewhenBROKER_PUBLIC_URLis https. The server keeps only its SHA-256 hash and compares it in constant time. - The broker sends the browser to sign in at the hub. This uses OIDC authorization code with PKCE (S256) and a nonce.
login_hint= the link'ssubwhenHUB_LOGIN_HINT=sub(the default).HUB_LOGIN_HINT=nonesends no hint, for real IdPs that expect a username or email. The hint is only for convenience: the signed-insubis still checked.- The hub's endpoints come from its OIDC discovery document:
HUB_DISCOVERY_URLwhen set (for an internal URL), otherwise{HUB_ISSUER}/.well-known/openid-configuration. The document must nameHUB_ISSUERas its issuer. - The broker builds the vendor step only after that sign-in succeeds as
the same
sub(§4.3).
Errors: 404 unknown-vendor, 503 hub-unavailable (hub discovery
failed), 503 coordination-unavailable (Redis down while taking the link
or storing the login state).
Each browser step's state is an opaque handle to a record on the server.
The records live TXN_TTL_S (default 600 s = 10 min) and are single use.
state is never a JWT and the browser can never decode it.
| Record | Fields |
|---|---|
Consent link (txn) |
sub, vendor, scopes, created_at |
| Hub-login state | leg: "hub", txn_id, sub, vendor, scopes, nonce, pkce_verifier, binding (hash), created_at |
| Vendor state | leg: "vendor", txn_id, sub, vendor, pkce_verifier, nonce, issuer, iss_required, binding (same hash as the hub leg), scopes, created_at |
issueris the vendor AS issuer from its metadata (none for vendors with hardcoded endpoints).iss_requiredis true when the metadata advertisesauthorization_response_iss_parameter_supported.scopesare always within the registry ceiling (§4.1).
The vendor authorize redirect carries client_id, response_type=code,
redirect_uri ({BROKER_PUBLIC_URL}/v1/callback/{vendor}), scope,
state, code_challenge, code_challenge_method=S256, and resource
when the registry sets one (RFC 8707).
4.3 GET /v1/callback/{vendor}?code&state
The browser returns here after each sign-in step. There are two kinds of
callback. Both answer the browser with short HTML pages sent with
Cache-Control: no-store, Referrer-Policy: no-referrer and
X-Content-Type-Options: nosniff. The raw vendor or hub error value goes
to the audit line only, never into the page.
Hub sign-in return (/v1/callback/_hub). _ never appears in a
vendor id. The broker, in order:
- checks the login state: it exists, has not expired or been used, and is a hub-leg state (else 400 and a security event),
- requires the binding cookie (a different browser gets 400 and a
security event,
reason: browser_mismatch), - checks
issagainstHUB_ISSUERwhenissis present (else 400 and a security event), - uses up the login state atomically (losing that race counts as a replay: 400 and a security event),
- stops with 400 if the hub returned an
error, - redeems the hub code with the PKCE verifier and checks the ID token:
signature from the hub JWKS, issuer, audience = the broker's client id
(
HUB_LOGIN_CLIENT_ID), expiry, nonce (failure: 502 page), - requires the ID token's
subto equal the link'ssub(otherwise 403 and a security event,reason: login_sub_mismatch), - starts the vendor step: reads the vendor metadata and client credential, stores the vendor state, and redirects to the vendor (§4.2).
Step 8 is the one place a callback answers with problem JSON, because the
browser has not yet reached the vendor: 503 vendor-unavailable (metadata
unreachable or no client credential), vault-unavailable, or
coordination-unavailable.
Vendor return. The broker, in order:
- Checks
state: it exists, has not expired, has not been used, matches the vendor, and was issued for the vendor step (else 400 and a security event,reason: state_invalid_or_replayed). - Checks the RFC 9207
isswhen a recorded issuer exists: a presentissmust equal it exactly, and a missingissis rejected wheniss_requiredis set (400 and a security event,reason: iss_mismatch). This runs before the state is used up, so a tampered callback never burns the state the real callback needs. - Requires the binding cookie. A different browser gets 400 and a security event, and no code is redeemed.
- Uses up the state atomically before redeeming the code. So exactly one callback per state ever reaches the token endpoint. Losing that race counts as a replay.
- Stops with 400 "Authorization failed." if the vendor returned an
error. - Handles the user's earlier grant for this vendor (see below).
- Exchanges the code + PKCE verifier for tokens, with client auth and
resourcewhen set (failure: 502 "Token exchange failed."). - Looks up the vendor user id (best effort, below).
- Waits for the entry's refresh lock (up to
LOCK_TIMEOUT_S), then writes the entry: fresh generation 1, overwriting any entry (cas=None). Granted scopes are capped to the ceiling (§5). If the lock never frees, or Redis is unavailable, it writes anyway: the code is spent, and losing the new grant would be worse. - Drops the cached entry on every replica (§7) and shows "Connected — return to your client."
What happens to the user's earlier grant for this vendor (step 6):
REVOKE_PENDING: the broker revokes and removes it before it exchanges the code, and auditsbroker.revokewithpath: reconsent. If the vendor has no revocation endpoint, the old entry is removed locally. If the vendor cannot revoke it yet, consent stops with a 503 page ("Your previous connection is still being revoked. Try again later.") and the user starts again later.- ACTIVE or STALE: the broker overwrites it. It does not revoke it, because some vendors revoke per user and client (GitHub grant deletion, many RFC 7009 servers), which would also revoke the new grant.
Other rules:
- Vendor user id. Read from the registry's
vendor_user_endpoint, or the metadatauserinfo_endpoint, usingid,loginorsubfrom the answer. It is for audit joins only. Any failure, or no endpoint, records"unknown"and consent goes on. - Code spent, entry not stored. If the custody write fails after the
code was redeemed, the broker revokes the new grant at the vendor (best
effort) so it is not orphaned there. It audits
broker.consent.failwithreason: post_redeem_failureandnew_grant_revoked, and shows a 503 "Credential store unavailable." page. The user runs consent again. - Custody down while reading the earlier grant: 503 "Credential store unavailable." page.
- Redis down (redis profile) during steps 1–4 of either callback: the
page "Coordination store unavailable." with status 503. It carries no
problem
title.
A state replay or mismatch is a security alert, not only a 400.
4.4 DELETE /v1/grants/{vendor}/{sub}
A user removes their own grant. The sub must match the hub JWT (else
403 forbidden). An unknown vendor is 404 unknown-vendor. The
MCP gateway's disconnect_<service> tools call this
route with the person's hub JWT.
The broker first waits for the entry's refresh lock (up to
LOCK_TIMEOUT_S, default 10 s). Holding it means no refresh can rotate
the token pair between the vendor revoke and the custody delete.
- Redis down (redis profile): 503
coordination-unavailable. - Lock not free in time: 503
vendor-unavailable("grant is being refreshed; retry").
Under the lock, in this order:
- Read the entry. None: 404
no-grant. - Revoke at the vendor (RFC 7009, or the vendor's deviation, §2.1).
- Re-read the entry. If its KV version changed (a pair landed after a
lost lock), revoke the newer pair too. Up to 3 revoke rounds, then 503
vendor-unavailable("grant kept changing during revocation; retry"). - Delete the custody entry (all versions and metadata) and drop it from every replica's cache.
- Audit
broker.revokewithoutcomeandhub_jti. - Answer
{"revoked": true}.
What the vendor revoke sends (RFC 7009):
- the access token (
token_type_hint: access_token), then the refresh token (token_type_hint: refresh_token). Some vendors keep access tokens alive after their refresh token is revoked (§2.1), - a 4xx on the access token is ignored (a server may not support revoking it). The last token sent decides: the refresh token, or the access token for a non-expiring grant with no refresh token,
- any 5xx, on either call, fails the revoke.
GitHub uses grant deletion instead (§2.1). A STALE entry has no tokens (§5), so it needs no vendor call.
If something goes wrong:
- Vendor revoke fails (unreachable, any 5xx, or a 4xx on the last
token sent): the broker CAS-writes
the entry it read, whatever its state, as
REVOKE_PENDING. If the entry moved since the read, it re-reads once and parks the newer pair (the pair the sweeper must revoke). If the entry is gone: 404no-grant. If it keeps moving: 503vendor-unavailable. Once parked, the response is 502revoke-pendingand the audit line hasoutcome: "pending". Resolve cannot use the entry. The sweeper retries (§8.1). Re-consent revokes it first (§4.3). - Vendor has no revocation endpoint: the broker cannot revoke. It
deletes the entry locally and audits it with
outcome: "unsupported". The response adds"vendor_revocation": "unsupported". The grant on the vendor side lives until it expires on its own. - Custody down: 503
vault-unavailable.
4.5 GET /v1/grants · GET /v1/admin/vendors/{vendor}
GET /v1/grantslists the caller's own grants on enabled vendors:vendor,state,granted_scopes,vendor_user_id,created_at. AREFRESHINGentry is listed asACTIVE.GET /v1/admin/vendors/{vendor}lets a signed-in admin read one registry record, without keys that containsecret. The hub JWT'sgroupsclaim must containADMIN_GROUP.groupsmust be a JSON array. A string claim never matches.
There is no route to change the registry, on purpose. Registry changes
are reviewed git changes, checked against
schemas/vendor-registry.schema.json. For this broker that is the
stronger control, not a gap. Changing scope_ceiling needs sign-off at
the token-contract level.
5. Custody schema
Custody holds two kinds of record: one token entry per user per vendor, and one client credential per vendor.
vendor-tokens/{vendor}/sub-b64.{base64url(sub), unpadded}
{access_token, refresh_token, expires_at,
granted_scopes[], vendor_user_id,
state: ACTIVE|REFRESHING|STALE|REVOKE_PENDING,
refresh_generation, last_refresh_at, created_at
(+ refresh_owner, refresh_started_at on a
REFRESHING marker)}
vendor-clients/{vendor} {client_id, client_secret | private_key (+alg, kid)}
The code binds this to OpenBao/Vault KV v2 (custody.py). Any backend
can replace it if it meets this contract:
- versioned compare-and-swap (CAS) on write,
- fails closed and can tell the difference (an outage is not the same as "absent"),
- two mounts: the broker may read and write
vendor-tokensbut only readvendor-clients. This is onebrokerpolicy covering both mounts (deploy/openbao-policy.hcl, which the test stack applies). It grants nothing to humans. Restricting admin reads is up to the deployment's own policies, - encryption of stored data at rest. This is a deployment requirement: this repo does not configure it.
refresh_generation is a counter that only goes up. §8's race defense
relies on it. The KV-v2 version is the CAS handle.
Key encoding. The sub is stored as one KV key: sub-b64. plus the
unpadded base64url of the sub. So any subject (URIs with /, dots,
spaces) is exactly one key, never a nested folder the sweeper cannot list.
Entries at the raw path vendor-tokens/{vendor}/{sub} are read too, and
move to the encoded path on their next write:
- a CAS write against a raw-path entry creates the encoded entry only if
none exists (
cas=0, so a replica that migrated first wins), then removes the raw one, - a consent overwrite also removes the raw one,
- a delete removes both,
- the sweeper's listing decodes and de-duplicates keys. It skips raw
folder keys (subjects with
/at the raw path), which only a direct read can reach.
How the broker builds an entry from a vendor token response:
| Vendor response | What the broker does |
|---|---|
Neither expires_in nor a refresh_token |
Treats it as a non-expiring token. Stores it with a far-future expires_at (10 years). Never refreshes it |
Has a refresh token but no expires_in |
Assumes it lasts 8 hours |
No refresh_token on a refresh |
Keeps the previous refresh token (a vendor that does not rotate) |
scope in the answer |
Recorded as granted_scopes, capped to a non-empty ceiling. Without scope, the requested scopes are recorded |
scope wider than the ceiling |
The extra scopes are dropped from the entry and listed as scope_widened on broker.consent.complete or broker.refresh |
A refresh keeps the entry's created_at (the consent time) and
vendor_user_id.
An entry with no refresh token that reaches its refresh buffer goes STALE
without a vendor call. The user then consents again. The broker audits it
as broker.stale, but it does not count toward the mass-STALE page.
STALE entries hold no tokens. Going STALE blanks both
access_token and refresh_token. A STALE entry is only ever
re-consented over or deleted.
6. Consent dance (first-time)
The first time a user needs a vendor, they connect the account in a
browser. (Full sequence diagram: token-lifecycle.md §1. Route details:
§4.2–4.3.)
- Resolve returns 404 +
authorize_uri(txn). - The browser opens
/v1/authorize/{vendor}. The link is used up and the binding cookie is set. - The user signs in at the hub.
- The browser returns to
/v1/callback/_hub. It must be the same browser, and the signed-insubmust equal the link'ssub. - The broker sends the browser to the vendor AS, with
resourcewhen the vendor's tokens are bound to one MCP server. The user consents there. - The browser returns to
/v1/callback/{vendor}?code&state&iss. - The broker checks state, iss, and binding, uses up the state, and
redeems the code on the server (with
resourcewhen set). - The broker writes the entry as
ACTIVE gen=1under the entry's refresh lock, or without it if the lock never frees (§4.3 step 9). - The browser shows a "connected" page.
- The caller retries, and resolve returns 200.
What this guarantees:
- PKCE verifiers stay out of browser URLs. The broker sends them only in server-side token requests.
statevalues the browser sees are opaque handles to records the broker holds.- The registry ceiling caps scopes, whatever the tool asked for.
-
Consent is bound to both the user and the browser:
-
User. The hub sign-in proves the browser belongs to the user the link was issued for. The broker refuses a link that was forwarded to someone else or stolen from its user. This happens before the vendor is involved. Without it, an attacker could send their own link to a victim and get the victim's vendor account stored under the attacker's
sub. -
Browser. The binding cookie ties every step to the browser that opened the link. So the broker refuses a forwarded hub sign-in callback (login CSRF) or a vendor authorization URL completed elsewhere. The user check alone would not stop an attacker who forwards the callback of their own successful sign-in.
-
Here the broker is an OIDC relying party of the hub. It uses a hub ID token and still issues nothing (§1).
7. Steady-state resolve
After consent, each resolve follows the same short path. (Sequence
diagrams: token-lifecycle.md §2 cache hit, §3 lazy refresh.)
- Check the in-memory cache on this replica (
CACHE_TTL_S, default 60s,0= no cache). This is also the only grace during a custody outage (§9). - Otherwise read custody. The broker emits no audit event for this. The custody backend's own audit log records it.
- Serve the token if
expires_at - now ≥ max(min_ttl, REFRESH_BUFFER_S). Becausemin_ttlis capped atREFRESH_BUFFER_S(§4.1), this isREFRESH_BUFFER_S(default 300 s) in practice. - Otherwise refresh inline, one refresh at a time (single-flight, §8).
The cache is bounded. It holds at most 10,000 entries per replica.
Past that, the oldest insert is evicted. Expired entries are purged at
most once per CACHE_TTL_S, when an entry is added, so token material
for users who never come back does not stay in memory.
When a cached entry is dropped. The broker drops the entry on this replica and, in the redis profile, broadcasts the drop to the others (best effort) when it:
- completes a consent (§4.3), including the removal of a REVOKE_PENDING predecessor,
- marks an entry STALE,
- parks an entry REVOKE_PENDING,
- deletes an entry (DELETE or sweeper retry).
It also drops the local copy before every re-read under the refresh lock.
A successful refresh replaces the local copy. Other replicas keep serving
their cached copy (the previous token, which has not expired) until their
CACHE_TTL_S runs out.
8. Refresh state machine and race defense
Each entry moves between four states. NoGrant means no stored entry.
Only one refresh may run per entry at a time.
stateDiagram-v2
classDef live fill:#10b98122,stroke:#10b981
classDef transient fill:#6366f122,stroke:#6366f1
classDef dead fill:#ef444422,stroke:#ef4444
[*] --> NoGrant
NoGrant --> ACTIVE : consent, gen=1
ACTIVE --> ACTIVE : refresh ok (memory profile), gen+1<br/>or scope re-consent, gen=1
ACTIVE --> REFRESHING : refresh starts<br/>(redis profile)
REFRESHING --> ACTIVE : refresh ok, CAS gen+1<br/>or vendor down, restored
REFRESHING --> STALE : invalid_grant
ACTIVE --> STALE : invalid_grant (memory profile)<br/>or no refresh token
STALE --> ACTIVE : re-consent, fresh gen=1
ACTIVE --> REVOKE_PENDING : DELETE, vendor revoke fails
REFRESHING --> REVOKE_PENDING : DELETE, vendor revoke fails
REVOKE_PENDING --> ACTIVE : re-consent, old grant<br/>revoked first, gen=1
ACTIVE --> NoGrant : DELETE, revoked<br/>or unsupported
REFRESHING --> NoGrant : DELETE, revoked<br/>or unsupported
STALE --> NoGrant : DELETE, nothing to revoke
REVOKE_PENDING --> NoGrant : sweeper retry<br/>or repeated DELETE
class ACTIVE live
class REFRESHING transient
class STALE,REVOKE_PENDING dead
What to notice:
- If the vendor revoke fails, DELETE parks whatever state it read as REVOKE_PENDING (in practice ACTIVE or an abandoned REFRESHING marker). A STALE entry has no tokens, so DELETE removes it without a vendor call.
- Both lazy (resolve) and proactive (sweeper) refreshes can end in STALE.
- Re-consent overwrites any state with a fresh
gen=1.
(Message-level sequences for every transition: token-lifecycle.md
§§3–5, 7–8.)
Some vendors (GitHub-class) rotate the refresh token on every refresh. If two refreshes race, the vendor burns the whole token family. The broker has three layers of defense:
- Single-flight lock per
{vendor, sub}. - In-process in the memory profile. Redis
SET NX PXin the redis profile. - Held for the whole refresh critical section: re-read, marker write, vendor round-trip, outcome CAS write.
- Resolves wait up to
LOCK_TIMEOUT_S(default 10 s). In the memory profile they park on the lock. In the redis profile they retry every 200 ms ± 25%, re-reading the entry each time, and serve the winner's token as soon as the generation moves forward. - The sweeper never waits: if the lock is held, it skips the entry.
- Persisted
REFRESHING(redis profile only), with owner and start time. - Other replicas can tell an in-flight refresh from a stale one.
- The next lock holder takes over a marker left behind longer than
REFRESHING_TTL_S(default 30 s). - Callers never see this state.
- Generation CAS — the correctness backstop in every profile.
- A CAS failure means another writer won.
- The loser throws away its result, re-reads, and never writes the older pair.
- So a lost lock can never corrupt custody.
What a resolve does once it holds the lock. It drops its cached copy and re-reads the entry:
| Re-read finds | Result |
|---|---|
| No entry, or STALE | 404 needs-consent |
| REVOKE_PENDING (parked while it waited) | 409 revoke-pending. Never refreshed back to ACTIVE |
| A fresh REFRESHING marker (another replica's lock ran out) | Serve the current token if it still has min_ttl left, else 503 vendor-unavailable. Never refresh over it |
| ACTIVE with a newer generation (a concurrent refresh won) | Serve that token, even if it has less than min_ttl left. The audit line adds short_ttl: true. Refreshing again cannot give a longer-lived token |
| Otherwise | Refresh (below) |
If the lock does not come within LOCK_TIMEOUT_S, the resolve re-reads
once. It serves an ACTIVE token with at least min_ttl left, else 503
vendor-unavailable.
One refresh (refresh.py, shared by resolve and sweeper):
- No refresh token: go STALE without a vendor call (not counted toward mass-STALE).
- Redis profile: CAS-write a
REFRESHINGmarker. A CAS loss ends the attempt as a lost race. - Call the vendor token endpoint (
grant_type=refresh_token, client auth,resourcewhen set). - The outcome:
| Vendor answer | Result |
|---|---|
| New token | CAS-write ACTIVE gen+1. Audit broker.refresh (generation_from, generation_to, path: proactive from the sweeper, scope_widened when set) |
invalid_grant or bad_refresh_token |
CAS-write STALE with tokens blanked. Audit broker.stale. Resolve returns 404 needs-consent |
Unreachable, timeout, 5xx, any other error code (for example invalid_client), malformed answer, no access_token |
503 vendor-unavailable. The entry keeps its tokens. In the redis profile the marker is CAS-written back to ACTIVE at once, instead of waiting for the takeover TTL |
| CAS lost on the outcome write | Throw the result away. Resolve re-reads and serves an ACTIVE entry, else 503 vendor-unavailable |
A CAS loss while writing STALE is a lost race, not a STALE: it is not audited or counted.
When a refresh starts:
- Lazy: a resolve finds the token inside
REFRESH_BUFFER_S(default 300 s). - Proactive: the sweeper refreshes ACTIVE entries with
REFRESH_BUFFER_S < remaining ≤ PROACTIVE_REFRESH_S(default 300–900 s), and takes over abandoned REFRESHING markers (§8.1).
Mass-STALE. When a refresh fails with invalid_grant, the entry goes
STALE. If one vendor has ≥MASS_STALE_THRESHOLD (default 3) such STALEs
inside MASS_STALE_WINDOW_S (default 60 s), the broker emits
broker.stale.mass with page: true and security_event: true, once
per window per vendor. This signals a possible org-level uninstall.
Monitoring must route it to on-call. If the coordination store is down,
the per-entry broker.stale is still written but the burst is not
counted.
8.1 The sweeper
The sweeper is a background loop that retries pending revocations and
refreshes tokens before callers need them. It runs every
SWEEP_INTERVAL_S (default 60 s, 0 = off).
flowchart TD
T["tick: SWEEP_INTERVAL_S<br/>(redis: ±20% jitter)"] --> L{"holds the<br/>sweep lease?"}
L -- "no, or Redis down" --> SKIP["skip this tick"]
L -- yes --> LIST["list subjects of every<br/>enabled vendor"]
LIST --> BATCH["take up to SWEEP_MAX_ENTRIES<br/>from the round-robin cursor"]
BATCH --> READ["read each entry"]
READ --> RP{"state?"}
RP -- REVOKE_PENDING --> RTRY{"refresh lock<br/>free now?"}
RTRY -- yes --> REV["re-read, revoke,<br/>delete if version unchanged"]
RTRY -- no --> NEXT["skip entry"]
RP -- "ACTIVE in band, or<br/>abandoned REFRESHING" --> TRY{"refresh lock<br/>free now?"}
TRY -- yes --> REF["re-read, then one refresh<br/>path: proactive"]
TRY -- no --> NEXT
RP -- "anything else" --> NEXT
The pass cost is bounded, and every entry is still reached over consecutive passes.
- Lease. The memory profile is always the leader, on a fixed interval.
In the redis profile only the holder of
vtb:sweep-leasesweeps (§14). If Redis is down, the tick is skipped. - Budget. A pass visits at most
SWEEP_MAX_ENTRIES(default 500) entries, starting where the previous pass stopped. A vendor whose listing fails (custody down) is skipped for that pass. Entries the budget skips are still refreshed lazily by resolve. - REVOKE_PENDING retry. The sweeper takes the entry's refresh lock
without waiting and skips the entry if someone holds it. Under the lock
it re-reads the entry and calls the vendor revoke with that pair. On
success, or when the vendor has no revocation endpoint, it re-reads once
more and deletes the entry only if its version is unchanged, then drops
it from every cache and audits
broker.revokewithpath: sweep-retry. Custody has no conditional delete, so this check plus the lock (which consent also holds while it writes) keeps the sweeper from deleting a grant written meanwhile. If the vendor is still down, it leaves the entry for the next pass. - Refresh band. An ACTIVE entry with
REFRESH_BUFFER_S < remaining ≤ PROACTIVE_REFRESH_S, or a REFRESHING marker older thanREFRESHING_TTL_S, is refreshed under the lock. The sweeper takes the lock without waiting and skips the entry if a resolve holds it. It re-reads under the lock and refreshes only an ACTIVE entry or an abandoned marker. - Errors. A failure on one entry is audited as
broker.sweep.errorand the pass goes on. A failure of the whole pass is audited the same way and the loop goes on.
9. Failure modes
The broker fails closed, and callers can tell the failures apart.
(Outage sequences: token-lifecycle.md §9. Replica-death recovery:
Token lifecycle §5.)
| Failure | How it is detected | What the broker does |
|---|---|---|
| User revoked the grant at the vendor | invalid_grant on refresh |
Entry → STALE. The next resolve returns needs-consent |
| Org app uninstalled at the vendor | Burst of STALEs for one vendor | Mass-stale page. Each entry behaves as usual |
| Refresh token expired from disuse | invalid_grant after dormancy |
STALE → re-consent. Expected, not an incident |
| Custody unavailable | Read/write errors (VAULT_TIMEOUT_S, default 3 s) |
Fail closed. No grace beyond the in-memory cache (CACHE_TTL_S, default 60s). 503 vault-unavailable on API routes, a 503 "Credential store unavailable." page on callbacks |
| Hub JWKS unreachable | Key fetch fails | 503 hub-unavailable (never 401). Also on authorize when hub discovery fails |
| Coordination (redis) unavailable | Lock/store errors | Problem JSON 503 coordination-unavailable from: resolve when it needs a refresh or a consent link (so absent, STALE, or insufficient-scope resolves get 503, not 404/409), authorize, DELETE, and the hub callback's vendor-step start. The browser callbacks otherwise answer with the page "Coordination store unavailable." (status 503, no problem title). Resolves served without a refresh still return 200. The sweeper skips its tick |
| Vendor AS outage | Timeouts, 5xx, or any token-endpoint error other than invalid_grant/bad_refresh_token (for example invalid_client) |
503 vendor-unavailable. The entry keeps its tokens. In the redis profile its REFRESHING marker is CAS-written back to ACTIVE |
| Vendor revocation fails | Revoke call fails on DELETE | Entry parked REVOKE_PENDING, 502 revoke-pending. The sweeper retries |
| Refresh lock not free in time | LOCK_TIMEOUT_S passes |
Resolve: re-read once, serve if usable, else 503 vendor-unavailable. DELETE: 503 vendor-unavailable |
| Custody write fails after a consent code was redeemed | Write error | The new grant is revoked at the vendor (best effort). 503 page, the user retries consent |
state replay / mismatch |
Server-side store check | 400 + security alert (possible CSRF or binding attack) |
| Replica dies mid-refresh | Lock TTL + takeover of the abandoned REFRESHING marker | Waiting requests retry. CAS prevents any stale write. Worst case: the family burns → STALE → re-consent |
10. Security requirements
The broker must meet all of these:
- No token issuing. No access-token issuance and no JWKS endpoint. A route audit in CI checks this.
- Signing only toward vendors. The broker may sign vendor client-authentication assertions with keys from custody.
- No token material in output. Never in logs, errors, or traces. A token-in-log grep in CI checks this.
- Pinned hub algorithms. Only PS256/384/512, ES256/384/512 and EdDSA can be configured. RS256 and every HMAC algorithm fail startup.
- Per-user entries only. No shared vendor service accounts through this path.
- Scope ceilings. The broker enforces registry scope ceilings at authorize time and caps recorded scopes to them.
- Bound consent. Consent is bound to the user the hub signed in and to the browser that started it (§6).
- Private callback. The callback host is on the private tier only.
- Pen testing. Include the consent dance in deployment penetration testing: CSRF, mix-up, and code injection (per RFC 9700 §4). This repo provides no pen-test report.
- No business data. The broker stores credentials, not business data. DLP (data loss prevention) at the egress gateway, not the broker, is the control that stops sensitive data from reaching vendors.
11. Audit events
The broker writes one JSON line per event to stdout:
{"audit": <event>, "ts": <unix time>, …fields}. Events hold ids,
states, and generations only.
The event names are wire-frozen:
| Event | Records |
|---|---|
broker.resolve |
decision + path, hub_jti, sub, vendor. May add min_ttl_clamped_from, short_ttl, missing, generation, error; the scope-ceiling denial adds required and ceiling. A malformed body or a sub mismatch is denied with reason (invalid_request, sub_mismatch, plus claimed_sub) instead of path |
broker.consent.start, broker.consent.complete, broker.consent.fail |
consent steps. complete has vendor_user_id (and scope_widened when set). fail has reason and security_event |
broker.refresh |
generation transition (generation_from, generation_to, optional path, scope_widened) |
broker.stale |
entry went STALE (generation) |
broker.stale.mass |
mass-STALE signal (count, window_s, page, security_event) |
broker.revoke |
revocation: outcome (revoked, unsupported, pending), optional path (sweep-retry, reconsent). hub_jti on a self-service DELETE that finished (revoked or unsupported) |
broker.admin.deny |
admin access denied |
broker.sweep.error |
sweeper error |
broker.custody.renew_failed |
the broker's own custody token could not be renewed (ttl_remaining_s) |
You can trace every vendor-side action end to end: hub jti → resolve →
gateway record → vendor audit log via vendor_user_id.
12. SLOs and capacity
These are design targets, not measured results.
- Resolve p99 ≤ 25 ms (cache hit) / 400 ms (inline refresh).
- Availability 99.9%. This is the egress hot path. Gateways treat 503 as retriable.
- Sizing: about tens of vendor refreshes per user per day. Cache-served resolves make up the hot path.
One modest replica pair is for high availability, not for throughput.
13. Sunset criteria (per vendor)
Before you retire the broker for a vendor, a proposed migration needs:
- compatible clients, identity provider, and resource authorization server, and
- evidence that the replacement gives the needed downstream vendor access.
Check parity in a controlled dual-run before you retire consent and drain grants.
ema_status is for tracking only. No registry flag automates ID-JAG
exchange, consent shutdown, or scheduled draining. Those features need a
separate implementation and rollout plan.
14. Deployment profiles
Run one replica with no extra infrastructure, or many replicas with Redis.
flowchart TB
subgraph mem["memory: exactly 1 replica"]
direction LR
B1["broker<br/>asyncio locks · local consent<br/>state · fixed-interval sweep"]
K1["custody KV-v2<br/>(CAS)"]
B1 --> K1
end
subgraph red["redis: N replicas"]
direction LR
LB["any balancer"]
A2["broker A"]
B2["broker B"]
R2["redis 8<br/>locks · consent state · sweep<br/>lease · invalidation pub/sub"]
K2["custody KV-v2<br/>(CAS = correctness backstop)"]
LB --> A2
LB --> B2
A2 --> R2
B2 --> R2
A2 --> K2
B2 --> K2
end
classDef broker fill:#6366f133,stroke:#6366f1
classDef custody fill:#10b98133,stroke:#10b981
classDef coord fill:#ef444433,stroke:#ef4444
classDef neutral fill:#94a3b833,stroke:#94a3b8
class B1,A2,B2 broker
class K1,K2 custody
class R2 coord
class LB neutral
Single replica (COORD_BACKEND=memory, the default)
- In-process asyncio lock per
{vendor, sub}. Locks are reference-counted and removed when idle, so the table holds only entries with a refresh in flight or waiters. - Consent records kept on the replica. Expired ones are removed every 60
s by a cleanup timer of their own, which runs even when the sweeper is
off (
SWEEP_INTERVAL_S=0). - Fixed-interval sweeper.
REFRESHINGis never persisted.- Zero extra infrastructure.
Correct only while replicas = 1.
Multi-replica (COORD_BACKEND=redis)
Redis (7.4 or later; the stack runs 8.10) provides:
- the distributed single-flight lock (
LOCK_TTL_MS, default 20000; it must be at least(VENDOR_TIMEOUT_S + 2 × VAULT_TIMEOUT_S) × 1000or startup fails), - shared single-use consent state, so consent may start on one replica and call back on another (no session affinity needed),
- persisted
REFRESHINGwith takeover of abandoned markers, - the sweep leader lease (+ jittered interval),
- the shared mass-STALE window,
- best-effort pub/sub cache invalidation (§7). Without it, the worst case
stays the per-replica
CACHE_TTL_S(default 60s).
Redis only helps availability. The KV-v2 CAS remains the correctness
guarantee. See docs/adr/0001-redis-coordination.md.
Both profiles pass the same acceptance suite. The multi profile also
passes tests/integration/test_multi_replica.py.
Token Lifecycle
Every path a connection takes, step by step: connect, refresh, disconnect, outages.
source: docs/token-lifecycle.md
This page shows every path a user's vendor connection (a "grant") can take through the broker. It starts at first consent and ends at deletion. It also shows what happens when things fail.
- A grant is the stored pair of vendor tokens for one user and one vendor (for example, one person's GitHub tokens).
- Timings are the defaults from
config.py. - For the rules behind these diagrams, see
design.md. To try these same paths by hand, seesmoke-tests.md. - If you use the broker through the MCP gateway, see the MCP gateway page first. This page covers the engine underneath.
Cast (the same in every diagram):
| Actor | Role |
|---|---|
| Client | The trusted gateway that calls the broker. It sends the user's hub JWT (a signed sign-in token from the hub, your company's sign-in service). It is not the MCP client. |
| Broker | This service. One replica unless the diagram says otherwise. |
| Vendor AS | The vendor's authorization server, which issues vendor tokens. GitHub-class: it rotates refresh tokens. |
| Custody | OpenBao/Vault KV-v2, where tokens are stored. Each entry's KV version is the CAS handle (CAS = compare-and-swap: a write succeeds only if the version has not changed). |
| Redis | The coordination backend. Used only in the multi-replica (redis) profile. |
0. Orientation — the states and which diagram covers each transition
A grant is always in one of these states (NoGrant means no stored
entry). The table below the diagram points to the diagram that shows each
change.
stateDiagram-v2
classDef live fill:#10b98122,stroke:#10b981
classDef transient fill:#6366f122,stroke:#6366f1
classDef dead fill:#ef444422,stroke:#ef4444
[*] --> NoGrant
NoGrant --> ACTIVE : consent, gen=1
ACTIVE --> ACTIVE : refresh ok (memory profile), gen+1<br/>or scope re-consent, gen=1
ACTIVE --> REFRESHING : refresh starts<br/>(redis profile)
REFRESHING --> ACTIVE : refresh ok, CAS gen+1<br/>or vendor down, restored
REFRESHING --> STALE : invalid_grant
ACTIVE --> STALE : invalid_grant (memory profile)<br/>or no refresh token
STALE --> ACTIVE : re-consent, fresh gen=1
ACTIVE --> REVOKE_PENDING : DELETE, vendor revoke fails
REFRESHING --> REVOKE_PENDING : DELETE, vendor revoke fails
REVOKE_PENDING --> ACTIVE : re-consent, old grant<br/>revoked first, gen=1
ACTIVE --> NoGrant : DELETE, revoked<br/>or unsupported
REFRESHING --> NoGrant : DELETE, revoked<br/>or unsupported
STALE --> NoGrant : DELETE, nothing to revoke
REVOKE_PENDING --> NoGrant : sweeper retry<br/>or repeated DELETE
class ACTIVE live
class REFRESHING transient
class STALE,REVOKE_PENDING dead
What to notice:
- A refresh can go STALE from a resolve (§3) or from the sweeper (§4).
- If the vendor revoke fails, DELETE parks whatever state it read as
REVOKE_PENDING(in practice ACTIVE or an abandoned REFRESHING marker). A STALE entry holds no tokens, so DELETE removes it with no vendor call. - Consent overwrites any state with a fresh
gen=1. Only aREVOKE_PENDINGpredecessor is revoked at the vendor first.
| Transition group | Diagram | What happens |
|---|---|---|
| §1 | Birth | The consent flow writes ACTIVE gen=1. |
| §2–§4 | Steady state / refresh | The broker serves from cache, or refreshes on demand or ahead of time (gen+1). |
| §5 | Multi-replica | A stored REFRESHING marker, takeover by another replica, and the CAS backstop. |
| §6 | Scope step-up | Re-consent for the combined scopes writes a fresh gen=1. |
| §7 | STALE | invalid_grant, or no refresh token, leads to re-consent. A burst of invalid_grant pages on-call. |
| §8 | Revocation | The broker revokes at the vendor first, then deletes. If the vendor revoke fails, the entry waits in REVOKE_PENDING until the sweeper retry or a repeated DELETE removes it. |
REFRESHING exists only in the redis profile. Callers never see it:
/v1/grants reports it as ACTIVE.
1. Birth — first-time consent dance
This runs when a resolve finds no entry, or a STALE one, for {vendor, sub}.
(sub is the user's id from the hub JWT.)
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant C as Client
participant UA as Browser
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(148,163,184,0.14) Hub
participant H as Hub (IdP)
end
box rgba(234,179,8,0.16) Vendor
participant V as Vendor AS
end
box rgba(16,185,129,0.16) Custody
participant K as Custody
end
C->>B: POST /v1/tokens/resolve (hub JWT)
Note over B: check hub JWT again:<br/>pinned algorithm (default PS256/ES256), issuer,<br/>exactly one tier audience, mcp_contract
B->>K: read entry
K-->>B: not found
B-->>C: 404 needs-consent + authorize_uri(txn)
C->>UA: open authorize_uri
UA->>B: GET /v1/authorize/{vendor}?txn=…
Note over B: link used up (single use, ≤ 5 min old)<br/>set binding cookie (HttpOnly, SameSite=Lax)
B-->>UA: 307 → hub login (PKCE, nonce,<br/>login_hint=sub unless HUB_LOGIN_HINT=none)
UA->>H: user signs in
H-->>UA: 302 → /v1/callback/_hub?code&state
UA->>B: GET /v1/callback/_hub (with binding cookie)
Note over B: login state valid, binding cookie matches,<br/>iss = HUB_ISSUER if present, then use up state
B->>H: POST /token (code + PKCE verifier)
H-->>B: ID token
Note over B: check ID token: signature, iss, aud, exp, nonce<br/>signed-in sub MUST equal the link's sub<br/>(else 403 + security event, no vendor step)
Note over B: create vendor PKCE verifier (S256) and<br/>single-use state {sub, vendor, issuer,<br/>scopes ≤ registry ceiling, binding}<br/>+ iss_required if the AS advertises RFC 9207
B-->>UA: 307 → vendor authorize<br/>(client_id, code_challenge, state,<br/>resource if the registry sets one)
UA->>V: user consents as themself
V-->>UA: 302 → /v1/callback/{vendor}?code&state&iss
UA->>B: GET /v1/callback/{vendor}?code&state&iss
Note over B: 1. state exists, unused, vendor matches<br/>2. iss equals recorded issuer (exact match,<br/> missing iss = mix-up if supported)<br/>3. binding cookie = the starting browser<br/>4. use up state, BEFORE redeeming the code<br/>5. a REVOKE_PENDING predecessor is revoked first
B->>V: POST /token (code + PKCE verifier<br/>+ client auth¹ + resource if set)
V-->>B: access token + refresh token (rotating)
B->>V: GET user endpoint → vendor_user_id<br/>(best effort, "unknown" on failure)
Note over B: take the entry's refresh lock,<br/>write anyway on timeout
B->>K: write entry state=ACTIVE gen=1 (cas=None, re-consent overwrites)
Note over B: drop cached entry on every replica
B-->>UA: "Connected — return to your client."
C->>B: retry resolve
B-->>C: 200 access_token
¹ Client auth follows the registry's token_endpoint_auth_method:
client_secret_post, client_secret_basic, or private_key_jwt. For
private_key_jwt, the broker signs an assertion with the key from
vendor-clients/{vendor} (RFC 7523). Its audience is the vendor's token
endpoint and it lives 300 seconds.
The resource parameter (RFC 8707) is for vendors whose tokens are bound
to one MCP server, such as Linear, Atlassian and Cloudflare. When the
registry entry sets resource, the broker sends it on the vendor authorize
redirect, on the code exchange, and on every refresh (§3, §4), unless the
registry sets resource_on_refresh: false (Linear refuses it on refresh,
design §2.1).
If the custody write fails after the code was redeemed, the broker revokes the new grant at the vendor (best effort) and shows a 503 page. The user connects again.
How this flow protects the user:
- Only the user the link was made for can finish consent, and only in the browser that opened the link. The hub login and the binding cookie enforce this.
- The broker redeems the code on the server only.
- The browser sees only opaque state handles.
- The PKCE verifier appears only in server-side token requests.
- A replayed callback (state already used) gets a 400 and a
security_event: trueaudit line. - A changed or missing
issis rejected before the state is used up. So the real callback can still finish.
Standards: RFC 7523, RFC 9207.
2. Steady state — cache-hit resolve
This runs when the token has plenty of time left:
expires_at − now ≥ max(min_ttl_s, REFRESH_BUFFER_S=300).
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant C as Client
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(16,185,129,0.16) Custody
participant K as Custody
end
C->>B: resolve (hub JWT)
alt per-replica cache fresh (≤ CACHE_TTL_S, default 60s)
Note over B: serve from memory, no custody read
else cache miss/expired
B->>K: read entry
K-->>B: ACTIVE, expires_at, version
Note over B: cache {entry, version} for ≤ CACHE_TTL_S
end
B-->>C: 200 {access_token, expires_at, granted_scopes}
- This is the hot path. Target: p99 ≤ 25 ms.
- The 60-second cache is also the only grace the broker gives during a custody outage (§9).
3. Lazy refresh — single-flight inside the buffer
This runs when the entry is ACTIVE but close to expiry:
expires_at − now < max(min_ttl_s, REFRESH_BUFFER_S=300).
Some vendors rotate refresh tokens. If a used refresh token is sent again, they revoke the whole token family. So when many resolves arrive at once, the broker must make exactly one vendor call ("single-flight").
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) Callers
participant C1 as Caller 1
participant C2 as Caller 2 (concurrent)
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(234,179,8,0.16) Vendor
participant V as Vendor AS
end
box rgba(16,185,129,0.16) Custody
participant K as Custody
end
par both inside the buffer
C1->>B: resolve
and
C2->>B: resolve
end
Note over B: lock per {vendor,sub}. C1 gets it.<br/>C2 waits (memory) or polls 200ms±jitter (redis)
B->>K: re-read under lock (gen = N, version = v)
B->>V: refresh_token grant (RT gen N)
V-->>B: new AT + rotated RT (old RT now dead)
B->>K: CAS write gen N→N+1 (cas = v)
K-->>B: ok, version v+1
Note over B: audit broker.refresh {gen N → N+1}
B-->>C1: 200 AT(gen N+1)
Note over B: lock released. C2 re-reads,<br/>sees gen N+1 and serves it
B-->>C2: 200 AT(gen N+1), same token,<br/>zero extra vendor calls
Note over K: memory profile shown. The redis profile first<br/>CAS-writes a REFRESHING marker (§5), so the result is v+2
What a waiting caller gets, best case first:
- The generation moved forward: it gets the winner's token.
- It waited more than 10s for the lock: it re-reads once. If the token is
still usable, it gets it. If not, it gets
503 vendor-unavailable(safe to retry). - The winner got
invalid_grant: the waiter finds STALE and getsneeds-consent.
In case 1 the waiter gets the winner's token even when it has less than
min_ttl_s left. The audit line then carries short_ttl: true. Another
refresh could not give a longer-lived token. (min_ttl_s itself is capped
at REFRESH_BUFFER_S, and a cap is audited as min_ttl_clamped_from.)
The refresh request carries the stored refresh token, client auth, and
resource when the registry sets one. If the vendor is unreachable or
answers any error other than invalid_grant/bad_refresh_token (for
example invalid_client), the caller gets 503 vendor-unavailable and
the entry keeps its tokens.
4. Proactive refresh — the sweeper
The sweeper is a background loop. It refreshes tokens before callers need them.
- It runs every
SWEEP_INTERVAL_S=60(0turns it off). In the redis profile, only the lease holder runs it, with ±20% jitter. In the memory profile the single replica always runs it, on a fixed interval. - It targets ACTIVE entries with 5–15 minutes left
(
REFRESH_BUFFER_S < remaining ≤ PROACTIVE_REFRESH_S, 300–900 s), and REFRESHING markers abandoned forREFRESHING_TTL_S(30 s). - Entries with 0–5 minutes left are left to lazy refresh (§3).
- It retries pending revocations (§8).
sequenceDiagram
autonumber
box rgba(99,102,241,0.20) Broker
participant S as Sweeper (lease holder)
participant B as Broker internals
end
box rgba(239,68,68,0.14) Coordination
participant R as Redis
end
box rgba(234,179,8,0.16) Vendor
participant V as Vendor AS
end
box rgba(16,185,129,0.16) Custody
participant K as Custody
end
S->>R: SET vtb:sweep-lease NX PX 2×interval
R-->>S: leader (others skip this pass)
S->>K: list subjects per enabled vendor
loop each entry, up to SWEEP_MAX_ENTRIES per pass (round-robin cursor)
S->>K: read entry
alt state REVOKE_PENDING
S->>B: try_lock (no waiting, skip if held)
S->>K: re-read under lock
S->>V: retry revoke
Note over S: success or unsupported: delete only if<br/>the version is unchanged, audit broker.revoke<br/>{path: sweep-retry}. Vendor still down: next pass
else REFRESHING abandoned (≥ REFRESHING_TTL_S) or ACTIVE and 300 < remaining ≤ 900
S->>B: try_lock (no waiting, a resolve may hold it)
B->>K: re-read under lock
B->>V: refresh_token grant
V-->>B: new AT + RT
B->>K: CAS write gen+1
Note over S: audit broker.refresh {path: proactive}
else anything else
Note over S: skip, not in the band
end
end
One failed entry never stops the loop. The sweeper logs
broker.sweep.error and moves on.
- The sweeper never waits for a lock. If a resolve holds it, the entry is skipped this pass.
- A proactive refresh ends like a lazy one:
gen+1, STALE oninvalid_grantor a missing refresh token (§7), or no change when the vendor is down. - If Redis is down, the whole tick is skipped. If custody cannot list one vendor's entries, that vendor is skipped for the pass.
5. Multi-replica refresh: persisted REFRESHING, takeover, CAS backstop
This applies to the redis profile only.
- The lock is a speed-up. The custody CAS is what keeps data correct.
- A replica that dies mid-refresh can never corrupt the stored token pair.
sequenceDiagram
autonumber
box rgba(99,102,241,0.20) Broker replicas
participant A as Replica A
participant Bb as Replica B
end
box rgba(239,68,68,0.14) Coordination
participant R as Redis
end
box rgba(234,179,8,0.16) Vendor
participant V as Vendor AS
end
box rgba(16,185,129,0.16) Custody
participant K as Custody
end
A->>R: SET vtb:lock:{vendor}:{sub} NX PX 20000
R-->>A: acquired
A->>K: CAS write state=REFRESHING + owner=A + started_at (v → v1)
A->>V: refresh_token grant …
Note over A: 💀 replica A dies here
Bb->>R: SET NX … (retry 200ms±jitter, re-reading each try)
Note over Bb: gives up after LOCK_TIMEOUT_S = 10s.<br/>That resolve gets 503 vendor-unavailable (entry REFRESHING)
Note over R: lock expires after 20s TTL
Bb->>R: a later resolve: SET NX …
R-->>Bb: acquired
Bb->>K: re-read: REFRESHING, owner=A
alt marker fresh (< REFRESHING_TTL_S = 30s)
Note over Bb: never refresh over a fresh marker.<br/>Serve the old AT if ≥ min_ttl,<br/>else 503 retry
else marker abandoned (≥ 30s)
Bb->>K: CAS write its own REFRESHING marker (v1 → v2)
Bb->>V: refresh with the stored RT
alt A's call never reached the vendor
V-->>Bb: new AT + RT, clean takeover
Bb->>K: CAS write ACTIVE gen+1 (cas = v2)
else A's call used the RT before dying
V-->>Bb: invalid_grant (replay burns the family)
Bb->>K: CAS write STALE (cas = v2)
Note over Bb: next resolve → needs-consent.<br/>Family lost, custody consistent.
end
end
Note over A,K: if A comes back and writes its result,<br/>its CAS (against v1) fails and the old pair is dropped
6. Scope step-up — 409 needs-reconsent-scope
This runs when a grant exists but lacks some of the tool's
required_scopes. The caller can never ask for more than the registry
ceiling (the most scopes the registry allows for that vendor).
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant C as Client
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(234,179,8,0.16) Vendor
participant V as Vendor AS
end
box rgba(16,185,129,0.16) Custody
participant K as Custody
end
C->>B: resolve {required_scopes: [read, write]}
Note over B: ceiling check first: required ⊄ ceiling<br/>→ 403 scope-exceeds-ceiling (hard stop)
B->>K: read entry, granted: [read]
Note over B: missing = [write] →<br/>re-consent scopes = (held ∪ required) ∩ ceiling
B-->>C: 409 needs-reconsent-scope<br/>{missing_scopes: [write], authorize_uri}
Note over C: gateway turns this into a step-up<br/>authorization challenge
C->>B: user re-runs the §1 dance via authorize_uri
B->>V: … consent for [read, write] …
B->>K: write fresh entry gen=1 (union of scopes)
C->>B: retry resolve {required_scopes: [read, write]}
B-->>C: 200
- The re-consent scopes keep the ceiling's order, and never drop a scope the user already granted.
- If the vendor's
scope_ceilingis empty (a GitHub App, where the app's permissions decide), the broker ignoresrequired_scopes, so this path never happens. - If the vendor grants more than the ceiling, the extra scopes are left
out of the entry and listed as
scope_widenedin the audit line.
7. Death by vendor — STALE and the mass-STALE page
invalid_grant on refresh (GitHub says bad_refresh_token, treated the
same) means the grant is gone at the vendor. Common causes:
- the user revoked access at the vendor
- the refresh token expired from disuse
- the org uninstalled the app
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant C as Client
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(234,179,8,0.16) Vendor
participant V as Vendor AS
end
box rgba(16,185,129,0.16) Custody
participant K as Custody
end
box rgba(239,68,68,0.14) Ops
participant O as On-call
end
C->>B: resolve (inside buffer)
B->>V: refresh_token grant
V-->>B: 400 invalid_grant
B->>K: CAS write state=STALE, both tokens blanked
Note over B: audit broker.stale {sub, vendor, generation}
B-->>C: 404 needs-consent + authorize_uri
Note over C: user fixes it: re-run the §1 dance
rect rgba(239,68,68,0.12)
Note over B,O: mass event: ≥ 3 STALEs for ONE vendor<br/>within 60s = looks like an org uninstall
B->>O: audit broker.stale.mass {page: true, security_event: true}
Note over O: signal sent once per window per vendor<br/>(deduped). Each entry is handled as usual.
end
- One STALE is normal and not an incident.
- An entry with no refresh token goes STALE when it reaches the refresh
buffer, with no vendor call. It is audited as
broker.stalebut never counted toward the burst. - If the STALE write loses its CAS (a concurrent refresh or re-consent moved the entry), nothing is marked or counted.
- If Redis is down, the per-entry
broker.staleis still written, but the burst is not counted. - To notify on-call about a burst, route the burst signal through your monitoring.
8. Death by choice — revocation, vendor-first
Order matters: the broker tries to revoke at the vendor before it deletes from custody.
- The MCP gateway's
disconnect_<service>tools call this same DELETE with the person's hub JWT (see the MCP gateway page). - If a vendor has no revocation endpoint, the broker deletes the entry
locally only. It reports
vendor_revocation: "unsupported". - In that case, access at the vendor can outlive the deletion.
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant U as User or gateway
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
participant L as Refresh lock
end
box rgba(234,179,8,0.16) Vendor
participant V as Vendor AS
end
box rgba(16,185,129,0.16) Custody
participant K as Custody
end
U->>B: DELETE /v1/grants/{vendor}/{sub} (hub JWT)
Note over B: sub in path MUST equal JWT sub<br/>(else 403 forbidden, users can only delete their own)
B->>L: wait for the lock (≤ LOCK_TIMEOUT_S = 10s)
Note over B,L: timeout → 503 vendor-unavailable<br/>Redis down → 503 coordination-unavailable
B->>K: read entry (version v)
loop up to 3 rounds
B->>V: revoke: access token, then refresh token²
B->>K: re-read
Note over B,K: same version v → done.<br/>A newer pair landed → revoke that one too
end
alt revoked, or vendor has no revocation endpoint
B->>K: delete entry (all versions + metadata)
B-->>U: 200 {revoked: true} (+ vendor_revocation: unsupported)
else vendor revoke fails
B->>K: CAS write the read entry as REVOKE_PENDING (cas = v)
B-->>U: 502 revoke-pending (will retry)
end
B->>L: release the lock
What to notice:
- The lock stops a refresh from rotating the pair between the revoke and
the delete. The re-read loop covers a lost lock. If the pair still
changes after 3 rounds, the answer is 503
vendor-unavailableand nothing is deleted. - A parked entry cannot be used: resolves get 409
revoke-pending. If the park CAS loses, the broker re-reads once and parks the newer pair. - Every finished delete drops the entry from every replica's cache.
The sweeper then retries the parked revocation on each pass (§4):
sequenceDiagram
autonumber
participant S as Sweeper
participant V as Vendor AS
participant K as Custody
loop each sweep pass until the vendor recovers
Note over S: take the entry lock without waiting,<br/>skip this pass if it is held
S->>K: read entry, state REVOKE_PENDING
S->>V: retry revoke
end
V-->>S: 200
S->>K: re-read, version unchanged?
S->>K: delete entry
Note over S: audit broker.revoke {path: sweep-retry}.<br/>If a new grant landed meanwhile,<br/>the sweeper leaves it alone
If the user connects again while the entry is REVOKE_PENDING, the
consent callback revokes and removes it first (path: reconsent). If the
vendor is still down, consent stops with a 503 page and the user tries
again later.
² GitHub does this differently. It uses its grant-deletion API with basic
auth instead of RFC 7009 (revocation.type: github_grant in the registry).
Other vendors get RFC 7009 calls for the access token, then the refresh
token. The refresh token's answer decides (for a grant that never expires
and has no refresh token, the access token's answer). A STALE entry holds
no tokens, so it is deleted with no vendor call.
9. Outages — fail closed, distinguishably
When a backend is down, the broker refuses rather than guesses ("fails closed"). Each backend fails in its own, recognizable way.
Neither failure ever looks like "no grant". That would send every user into re-consent at once.
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant C as Client
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(239,68,68,0.14) Failed backends
participant K as Custody (down)
participant R as Redis (down)
end
rect rgba(234,179,8,0.10)
Note over C,K: custody outage, fully fail-closed
C->>B: resolve (uncached sub)
B--xK: read fails (3s timeout)
B-->>C: 503 vault-unavailable (retriable)
Note over B: only grace: entries already in the<br/>per-replica cache (CACHE_TTL_S) keep serving
end
rect rgba(59,130,246,0.10)
Note over C,R: redis outage, every path that needs redis (redis profile)
C->>B: resolve, entry ACTIVE, sufficient scopes, NOT near expiry
B-->>C: 200, cache/custody path does not use redis
C->>B: resolve, entry inside refresh buffer
B--xR: lock acquisition fails
B-->>C: 503 coordination-unavailable (retriable)
Note over B: consent links need redis too. Absent, STALE or<br/>too-few-scopes resolves also get 503, not 404/409
Note over B: no lock means no refresh. Refreshing without<br/>single-flight could burn a rotating RT family.<br/>Custody CAS still keeps data correct.
end
Other paths during a Redis outage (redis profile):
| Path | Answer |
|---|---|
GET /v1/authorize/{vendor} |
problem JSON 503 coordination-unavailable |
DELETE /v1/grants/{vendor}/{sub} |
problem JSON 503 coordination-unavailable |
| Hub callback, starting the vendor step | problem JSON 503 coordination-unavailable |
Browser callbacks (/v1/callback/_hub, /v1/callback/{vendor}) reading or using up state |
HTML page "Coordination store unavailable." with status 503, no problem title |
| Sweeper | skips the tick |
- Normal service resumes when the backends come back.
- Custody stays the lasting store of credentials.
- If Redis loses consent records, users must start a new browser flow.
Reading the audit trail
Every transition above writes JSON log lines with ids only, never token
material. One action can log more than one event. For example, a lazy
refresh logs both broker.refresh and broker.resolve. Every line has
audit (the event name) and ts (Unix time).
How to link events together:
| Event | Emitted in | Joins on |
|---|---|---|
broker.resolve {decision, path} |
§§2,3,6,7,9 | hub_jti, sub, vendor |
broker.consent.start / .complete / .fail |
§1, §6 | sub (when known), vendor. Only .complete carries vendor_user_id |
broker.refresh {generation_from → to, path?: proactive} |
§§3,4,5 | sub, vendor |
broker.stale / broker.stale.mass |
§§3,4,7 | sub/vendor. Mass carries page: true |
broker.revoke {outcome: revoked|unsupported|pending, path?: sweep-retry|reconsent} |
§8 | sub, vendor. hub_jti only on a self-service DELETE that finished (revoked or unsupported), not on pending |
You can trace a vendor-side action from end to end: hub jti →
broker.resolve → gateway record → vendor audit log via vendor_user_id.
Smoke Tests
A hands-on tour that checks each protection with curl and a browser.
source: docs/smoke-tests.md
This page walks you through key security and lifecycle checks by hand. You
run them against the self-contained test stack, using only curl and a
browser.
- The outputs below show what to expect. Ids and timestamps will differ.
- For dated suite results, see the verification record.
- The automated suite (
tests/unit,tests/integration) checks the same properties on every run. - These tests exercise the broker directly. For the MCP gateway that sits in front of it, see the MCP gateway page.
Conventions used on this page. The commands are written to fit narrow terminals:
echo '{"vendor":"mockhub"}' > /tmp/req.json # resolve request body
# hub JWT lives in /tmp/tok; every resolve is:
# curl -s -X POST localhost:8300/v1/tokens/resolve \
# -d @/tmp/req.json -H "Authorization: Bearer $(cat /tmp/tok)"
The stack under test
The test stack runs in Docker Compose. Your terminal and browser play the part of the trusted gateway and the user.
flowchart LR
subgraph host["Your machine"]
T["Terminal / Browser<br/>(acts as the trusted gateway + user)"]
end
subgraph compose["tests/stack compose"]
B["vtb-broker :8300<br/>the service under test"]
H["vtb-hub-stub :8320<br/>fake company sign-in (IdP)<br/>JWKS + /_test/token mint"]
M["vtb-mock-vendor :8310<br/>hostile vendor AS<br/>60s tokens, rotating RTs"]
V["vtb-openbao :8210<br/>KV-v2 token storage"]
R["vtb-redis<br/>coordination (COORD_BACKEND=redis)"]
I["vtb-openbao-init<br/>one-time setup job"]
end
T -- "resolve / consent" --> B
T -- "mint test JWTs" --> H
B -- "check hub JWT (JWKS)" --> H
B -- "OAuth steps" --> M
B -- "read/write grants (CAS)" --> V
B -.-> R
I -- "mounts + ACL policy +<br/>scoped token (never root)" --> V
classDef broker fill:#6366f133,stroke:#6366f1
classDef vendor fill:#eab30833,stroke:#eab308
classDef custody fill:#10b98133,stroke:#10b981
classDef coord fill:#ef444433,stroke:#ef4444
classDef neutral fill:#94a3b833,stroke:#94a3b8
class B broker
class M vendor
class V,I custody
class R coord
class T,H neutral
Terms used on this page:
- Hub — your company's sign-in service (identity provider). Here, a stub plays it. A hub JWT is the signed token it issues for a user.
- Custody — where the broker stores vendor tokens (OpenBao KV-v2).
- Grant — one user's stored vendor tokens for one vendor.
The mock vendor is hostile on purpose, in the ways that matter:
- Access tokens last 60 seconds, so every resolve lands inside the refresh buffer.
- Refresh tokens rotate. Sending a used one again revokes the whole family (GitHub-class).
- It checks PKCE for real.
- It publishes RFC 8414 metadata and adds the RFC 9207
issto redirects. - It supports RFC 7009 revocation.
Test 1 — stack up, scoped custody token
Start the stack and check that the broker does not use a root token.
docker compose -f tests/stack/docker-compose.yml up -d --build --wait
curl -s http://localhost:8300/healthz # -> {"ok":true,"custody":"ok"}
docker logs vtb-openbao-init
The setup job prints:
==> Waiting for OpenBao at http://openbao:8200...
==> Writing vendor client credentials to vendor-clients/...
==> mockhub-jwt (private_key_jwt) client configured
==> OpenBao initialized (scoped broker token ready)
What this proves:
- The broker uses a custody token limited by policy, never root. It can
read and write
vendor-tokens/*and only readvendor-clients/*. - Vendor client credentials live in custody, not in the registry or the
environment. This includes the
private_key_jwtsigning key.
Test 2 — hub-JWT validation at the door, consent challenge
Get a hub JWT from the hub stub, then ask the broker to resolve a token:
curl -s -X POST localhost:8320/_test/token -d '{"sub":"wf-smoke"}' -o /tmp/mint.json
sed 's/.*"access_token":"\([^"]*\)".*/\1/' /tmp/mint.json > /tmp/tok
curl -s -X POST localhost:8300/v1/tokens/resolve -d @/tmp/req.json \
-H "Authorization: Bearer $(cat /tmp/tok)"
Expected:
{"type":"urn:vendor-token-broker:needs-consent","title":"needs-consent",
"detail":"no usable grant for mockhub",
"authorize_uri":"http://localhost:8300/v1/authorize/mockhub?txn=RiHcUB2s..."}
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant C as Client (curl)
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(16,185,129,0.16) Custody
participant V as OpenBao
end
C->>B: POST /v1/tokens/resolve (hub JWT)
Note over B: check JWT again: signature (JWKS),<br/>alg in HUB_ALGORITHMS (default PS256, ES256),<br/>issuer, exp/iat/sub/jti, mcp_contract,<br/>exactly one tier audience
B->>V: read vendor-tokens/mockhub/sub-b64.{encoded subject}
V-->>B: not found
B-->>C: 404 needs-consent + authorize_uri(txn)
Note over C: a gateway would pass this on as the<br/>custom consent prompt (see the MCP integration guide)
What this proves: the broker accepts a valid hub JWT, but access is
per user. With no grant, the user gets no token. Instead the broker
returns a one-time consent link (txn).
- The link can start consent for 5 minutes.
- Consent state lasts 10 minutes by default.
Test 3 — the consent dance (terminal and real browser)
Follow the authorize_uri. You can do this in a terminal or a real
browser.
In a terminal, use a cookie jar. The flow is tied to the browser that
starts it, so cookies must carry across the redirects:
curl -sL -c /tmp/jar -b /tmp/jar "<authorize_uri>".
In a real browser (what a real user sees):
curl -s -X POST localhost:8300/v1/tokens/resolve -d @/tmp/req.json \
-H "Authorization: Bearer $(cat /tmp/tok)" -o /tmp/challenge.json
open "$(sed 's/.*"authorize_uri":"\([^"]*\)".*/\1/' /tmp/challenge.json)"
The browser goes broker → hub login → broker → vendor → broker. Output
captured with curl -sL -D - (codes, states, nonces and the cookie value
are hidden):
307 location: http://localhost:8320/authorize?client_id=vtb-broker&response_type=code
&scope=openid&redirect_uri=http%3A%2F%2Flocalhost%3A8300%2Fv1%2Fcallback%2F_hub
&state=…&nonce=…&code_challenge=…&code_challenge_method=S256&login_hint=wf-smoke
set-cookie: vtb_consent_mockhub=…; HttpOnly; Max-Age=600; Path=/v1/callback; SameSite=lax
307 location: http://localhost:8300/v1/callback/_hub?code=…&state=…&iss=http%3A%2F%2Fhub-stub%3A8320
307 location: http://localhost:8310/authorize?client_id=mcp-lab-broker&response_type=code
&redirect_uri=http%3A%2F%2Flocalhost%3A8300%2Fv1%2Fcallback%2Fmockhub
&scope=issues%3Aread+issues%3Awrite&state=…&code_challenge=…&code_challenge_method=S256
307 location: http://localhost:8300/v1/callback/mockhub?code=…&state=…&iss=http%3A%2F%2Fmock-vendor%3A8310
200 <h1>Connected — return to your client.</h1>
The hub stub signs the browser in automatically as the login_hint user.
A real hub shows its own sign-in page, or reuses an existing session.
The flow ends on the page "Connected — return to your client." The
broker log shows the matching audit pair (your ts values will differ):
{"audit": "broker.consent.start", "ts": 1784286281.07, "sub": "wf-smoke", "vendor": "mockhub"}
{"audit": "broker.consent.complete", "ts": 1784286284.52, "sub": "wf-smoke", "vendor": "mockhub", "vendor_user_id": "mock-4217"}
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant UA as Browser
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(148,163,184,0.14) Hub
participant H as Hub stub (IdP)
end
box rgba(234,179,8,0.16) Vendor
participant M as Mock vendor AS
end
box rgba(16,185,129,0.16) Custody
participant V as OpenBao
end
UA->>B: GET /v1/authorize/mockhub?txn=…
Note over B: link used up (single use, ≤ 5 min)<br/>set binding cookie for this browser
B-->>UA: 307 → hub login (PKCE, nonce, login_hint)
UA->>H: sign in (auto: as the login_hint)
H-->>UA: 307 → /v1/callback/_hub (code, state, iss)
UA->>B: GET /v1/callback/_hub (binding cookie)
B->>H: POST /token (code + PKCE verifier)
H-->>B: ID token
Note over B: ID token valid and its sub == the link's sub<br/>create single-use vendor state record<br/>{sub, vendor, PKCE verifier, issuer, iss_required,<br/>scopes ≤ registry ceiling, binding}, TTL 10 min
B-->>UA: 307 → vendor authorize<br/>(client_id, code_challenge S256, state)
UA->>M: GET /authorize (auto-consent as mock-4217)
M-->>UA: 307 → broker callback (code, state, iss)
UA->>B: GET /v1/callback/mockhub?code&state&iss
Note over B: check state (exists, unused,<br/>vendor match) → check RFC 9207 iss<br/>→ binding cookie = this browser<br/>→ use up state (single use)<br/>→ THEN redeem code
B->>M: POST /token (code + PKCE verifier + client auth)
M-->>B: access token (60s) + rotating refresh token
B->>V: write entry state=ACTIVE gen=1
B-->>UA: "Connected — return to your client."
Now the retry succeeds:
curl -s -X POST localhost:8300/v1/tokens/resolve -d @/tmp/req.json \
-H "Authorization: Bearer $(cat /tmp/tok)"
{"access_token":"mock-at-nV2-JGwg…","expires_at":1784286345.61,
"granted_scopes":["issues:read","issues:write"]}
What this proves:
- The PKCE verifier appears only in server-side token requests. The
browser sees
statevalues only as opaque handles. The broker redeems the vendorcodeon the server only. - Consent is tied to the user and to the browser:
- The hub login must match the link's
sub. A link forwarded to someone else, or stolen, is refused before the vendor is involved (reason: login_sub_mismatch). - The binding cookie must match. A step finished in another browser is
refused before any code is redeemed (
reason: browser_mismatch). - Both refusals write
security_event: trueaudit lines. - The authorize link works once. Opening it again returns 400
invalid-transaction. - The broker compares the
issin the callback URL, as an exact string, with the issuer it recorded when the vendor leg started (after the hub sign-in). It does this before redeeming the code (this stops mix-up attacks). If the vendor says it supportsissbut leaves it out, the broker rejects the callback the same way. - Replay demo: reload the "Connected" page. You get
"Invalid or expired authorization state." The state was already used.
The replay writes a
security_event: trueaudit line (reason: state_invalid_or_replayed). - The stand-in gateway (your terminal) now holds a vendor token. The hub JWT never goes to the vendor.
Standards: RFC 9207.
Test 4 — single-flight refresh and the generation counter
The mock's 60-second tokens force a refresh on every resolve. The Test 3 retry already refreshed gen 1→2. Resolve once more, then read the audit trail:
curl -s -X POST localhost:8300/v1/tokens/resolve -d @/tmp/req.json \
-H "Authorization: Bearer $(cat /tmp/tok)" # new mock-at-… each time
docker logs --since 5m vtb-broker 2>&1 | grep broker.refresh | tail -3
Expected (your ts values will differ):
{"audit": "broker.refresh", "ts": 1784286285.61, "sub": "wf-smoke", "vendor": "mockhub", "generation_from": 1, "generation_to": 2}
{"audit": "broker.refresh", "ts": 1784286312.08, "sub": "wf-smoke", "vendor": "mockhub", "generation_from": 2, "generation_to": 3}
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) Callers
participant C1 as Caller 1
participant C2 as Caller 2 (concurrent)
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(234,179,8,0.16) Vendor
participant M as Mock vendor
end
box rgba(16,185,129,0.16) Custody
participant V as OpenBao
end
par inside the refresh buffer
C1->>B: resolve
and
C2->>B: resolve
end
Note over B: one lock per {vendor,sub}<br/>C1 wins, C2 waits, no second refresh
B->>M: refresh_token grant (RT gen N)
M-->>B: new AT + rotated RT
B->>V: CAS write gen N→N+1 (fails if version moved)
B-->>C1: 200 AT(gen N+1)
B-->>C2: 200 AT(gen N+1), same token, zero extra vendor calls
What this proves:
- Every refresh raises
refresh_generationby one. It never goes down. - The KV-v2 compare-and-swap (CAS) on that version keeps data correct. Two racing refreshes can never both be saved. A writer that loses the race cannot overwrite a newer stored version.
- Limit: if the broker crashes after the vendor accepts a refresh token, the family can still be burned. The user must then consent again.
- The automated suite sends 20 resolves at once. It checks for exactly one vendor refresh and zero RT replays, including across two replicas behind a load balancer.
- The audit lines hold ids, states and generations only, never token material.
Test 5 — forged-algorithm token rejected at the door
The hub stub can make an RS256 token signed with the same trusted RSA key that is in its JWKS. The signature is valid and the key can be found. Only the algorithm is wrong:
curl -s -X POST localhost:8320/_test/token -d '{"sub":"wf-smoke","kind":"rs256"}' -o /tmp/mint2.json
sed 's/.*"access_token":"\([^"]*\)".*/\1/' /tmp/mint2.json > /tmp/badtok
curl -s -X POST localhost:8300/v1/tokens/resolve -d @/tmp/req.json \
-H "Authorization: Bearer $(cat /tmp/badtok)"
Expected:
{"type":"urn:vendor-token-broker:invalid-hub-token","title":"invalid-hub-token",
"detail":"The specified alg value is not allowed"}
flowchart TB
A["Bearer token"] --> K{"signing key in the<br/>hub JWKS, by kid?"}
K -- "JWKS unreachable" --> U["503 hub-unavailable"]
K -- yes --> D["One decode step checks all of:<br/>alg in HUB_ALGORITHMS<br/>signature<br/>iss == HUB_ISSUER<br/>aud has HUB_TIER_AUDIENCE<br/>exp, iat, sub, jti present<br/>not expired"]
D -- "all pass" --> G{"mcp_contract == 1.0?"}
G -- yes --> E{"exactly one<br/>mcp://tier/* audience?"}
E -- yes --> OK["resolve proceeds"]
K -- no --> X["401 invalid-hub-token"]
D -- "any check fails" --> X
G -- no --> X
E -- no --> X
The checks in the middle box run together in one jwt.decode call, so
the first failure it finds names the detail. Here that is the
algorithm. The contract and tier-count checks run after it, in that
order.
You get the same 401 for every bad token the stub can make:
wrong_issuer, external_tier, two_tiers (a token for two tiers),
expired, no_jti, wrong_contract, no_contract. A good token that
has really expired also gets it, with
"detail":"Signature has expired".
What this proves: the broker does not trust the gateway. It checks every incoming hub JWT itself, and accepts only the allowed algorithms.
Test 6 — the no-issuance wall
The broker has no endpoints for issuing tokens. Probe the usual paths:
for p in /token /oauth/token /keys /v1/tokens/issue \
/.well-known/jwks.json /.well-known/openid-configuration; do
curl -s -o /dev/null -w "$p -> %{http_code}\n" localhost:8300$p
done
Expected:
/token -> 404
/oauth/token -> 404
/keys -> 404
/v1/tokens/issue -> 404
/.well-known/jwks.json -> 404
/.well-known/openid-configuration -> 404
What this proves: the broker stores tokens. It does not issue them.
- Its whole API is seven routes:
/healthz,/v1/tokens/resolve,/v1/authorize/{vendor},/v1/callback/{vendor},/v1/grants/{vendor}/{sub},/v1/grants,/v1/admin/vendors/{vendor}. - It has no access-token issuance endpoint and no JWKS endpoint.
- It can hold vendor client-authentication keys and sign
private_key_jwtassertions. tests/unit/test_routes.pychecks the actual route table on every push.
Test 7 — self-service revocation, vendor-first
A user deletes their own grant. The broker revokes at the vendor first, then deletes the stored entry.
curl -s localhost:8310/_test/state | grep -o '"revoke":[0-9]*' # "revoke":0
curl -s -X DELETE localhost:8300/v1/grants/mockhub/wf-smoke \
-H "Authorization: Bearer $(cat /tmp/tok)" # {"revoked":true}
curl -s localhost:8310/_test/state | grep -o '"revoke":[0-9]*' # "revoke":2
curl -s -X POST localhost:8300/v1/tokens/resolve -d @/tmp/req.json \
-H "Authorization: Bearer $(cat /tmp/tok)" # needs-consent again
sequenceDiagram
autonumber
box rgba(148,163,184,0.14) User side
participant U as User (curl)
end
box rgba(99,102,241,0.20) Broker
participant B as Broker
end
box rgba(234,179,8,0.16) Vendor
participant M as Mock vendor
end
box rgba(16,185,129,0.16) Custody
participant V as OpenBao
end
U->>B: DELETE /v1/grants/mockhub/wf-smoke (hub JWT)
Note over B: sub in path MUST match JWT sub<br/>(anyone else → 403 forbidden)<br/>then hold the refresh lock until done
B->>M: POST /revoke (RFC 7009): access token
M-->>B: 200, family revoked at the vendor
B->>M: POST /revoke (RFC 7009): refresh token
M-->>B: 200, this answer decides
B->>V: re-read (unchanged), then delete<br/>vendor-tokens/mockhub/sub-b64.{encoded subject}
B-->>U: {"revoked": true}
Note over B,M: if the vendor were down: entry waits as<br/>REVOKE_PENDING (502), cannot be resolved,<br/>sweeper retries until the vendor recovers
What this proves:
- This mock supports revoking at the vendor first. For vendors without revocation support, the broker deletes the entry locally only.
- Users can delete only their own grants.
- The lifecycle returns cleanly to
needs-consent.
The full lifecycle we walked
stateDiagram-v2
classDef live fill:#10b98122,stroke:#10b981
classDef transient fill:#6366f122,stroke:#6366f1
classDef dead fill:#ef444422,stroke:#ef4444
[*] --> NoGrant
NoGrant --> ACTIVE : consent, gen=1
ACTIVE --> ACTIVE : refresh ok (memory profile), gen+1<br/>or scope re-consent, gen=1
ACTIVE --> REFRESHING : refresh starts<br/>(redis profile)
REFRESHING --> ACTIVE : refresh ok, CAS gen+1<br/>or vendor down, restored
REFRESHING --> STALE : invalid_grant
ACTIVE --> STALE : invalid_grant (memory profile)<br/>or no refresh token
STALE --> ACTIVE : re-consent, fresh gen=1
ACTIVE --> REVOKE_PENDING : DELETE, vendor revoke fails
REFRESHING --> REVOKE_PENDING : DELETE, vendor revoke fails
REVOKE_PENDING --> ACTIVE : re-consent, old grant<br/>revoked first, gen=1
ACTIVE --> NoGrant : DELETE, revoked<br/>or unsupported
REFRESHING --> NoGrant : DELETE, revoked<br/>or unsupported
STALE --> NoGrant : DELETE, nothing to revoke
REVOKE_PENDING --> NoGrant : sweeper retry<br/>or repeated DELETE
class ACTIVE live
class REFRESHING transient
class STALE,REVOKE_PENDING dead
The tests walked consent (Test 3), refresh (Test 4) and DELETE (Test 7).
STALE happens on a lazy refresh or a sweeper refresh, and also when an
entry has no refresh token left (no vendor call). If the vendor revoke
fails, DELETE parks whatever state it read as REVOKE_PENDING (in
practice ACTIVE or an abandoned REFRESHING marker).
For every path in detail, see the token lifecycle.
Scorecard
| # | Property | Verified by |
|---|---|---|
| 1 | Scoped custody token, never root | setup job output |
| 2 | Hub-JWT re-check + per-user consent challenge | 404 needs-consent |
| 3 | PKCE + single-use state tied to sub + RFC 9207 iss |
browser flow → "Connected"; reload → replay rejected |
| 4 | Single-flight refresh, generation CAS, id-only audit | gen 1→2→3 in broker.refresh |
| 5 | Only allowed algorithms / contract shape at the door | RS256 & expired → 401 invalid-hub-token |
| 6 | No token-issuing endpoints | six issuer paths → 404 |
| 7 | Revoke at vendor first, users delete only their own | RFC 7009 counter 0→2 (access token, then refresh token), entry gone |
Tear down with:
docker compose -f tests/stack/docker-compose.yml down
Redis Decision
Why Redis coordinates several brokers running side by side.
source: docs/adr/0001-redis-coordination.md
Status: accepted · Date: 2026-07-16
This record explains why the broker uses Redis 7 to coordinate more than one replica.
Update 2026-09-30: the test stack and the multi-replica deploy shape moved to Redis 8.10.2. The commands the broker uses (SET NX PX, GETDEL, Lua, sorted sets, pub/sub) are unchanged, and the redis and multi-replica suites pass on it.
Context
The source lab ran the broker as one replica. There:
- An in-process asyncio lock per
{vendor, sub}made sure only one refresh ran at a time (single-flight refresh). - Consent transactions and
staterecords lived in in-process dicts. - The sweeper always ran.
This was documented as correct only for one replica. Running more than one replica needs all of these together:
- a single-flight lock shared across replicas,
- a persisted
REFRESHINGmarker, - a sweeper with jitter, run only by an elected leader,
- shared, single-use consent state (or session affinity at the load balancer),
- honest docs about the token cache being per replica.
The key constraint: the lock is an optimization. It protects rotating refresh-token families from being burned. The correctness guarantee is the KV-v2 compare-and-swap (CAS): a write succeeds only if the entry's KV version has not changed since it was read. The entry's refresh_generation only goes up, so a CAS loser can never write an older pair. So the coordination backend needs speed and TTL (auto-expiry) support, not durability.
Decision
Use Redis 7 behind a Coordination protocol (coordination.py). COORD_BACKEND selects the backend. memory stays the default. It keeps the lab's exact single-replica behavior and needs no extra infrastructure.
How each job maps to Redis (with the default settings):
- Lock:
SET vtb:lock:{vendor}:{sub} <token> NX PX 20000. - The PX TTL is
LOCK_TTL_MS(default 20000). Startup fails if it is shorter than(VENDOR_TIMEOUT_S + 2 × VAULT_TIMEOUT_S) × 1000, so the lock outlives the slowest refresh it protects. - Waiters retry every 200ms ± 25%, for up to
LOCK_TIMEOUT_S(10s). - On each retry, a waiting resolve re-reads the entry. Once the generation moves forward, it serves that token without taking the lock.
- Release uses a compare-and-DEL Lua script, so only the holder ever releases a lock. If the release fails, the PX TTL frees the lock.
- The sweeper takes the lock without waiting and skips a held entry.
- DELETE and the consent write also wait for this lock. The sweeper's revoke retry takes it without waiting.
- Consent transactions and states:
SETEX(TXN_TTL_S, 600), consumed once with atomicGETDEL. An authorize link must be used within 300 s (AUTHORIZE_LINK_TTL_Sinconsent.py). - For states, the order is peek, check, then consume. So a rejection for an issuer (iss) mismatch does not burn the state the real callback needs.
- Persisted
REFRESHING: lives in the storage (custody) entry, not in redis, withrefresh_ownerandrefresh_started_at. The broker writes it with CAS before calling the vendor. If the vendor is unreachable or errors, the broker CAS-writes the entry back to ACTIVE at once. The next lock holder takes over markers older thanREFRESHING_TTL_S(30s), and never refreshes over a younger one. - Mass-STALE window: a per-vendor ZSET, plus
SET NX PXto send each page only once. - Sweeper: leader lease
SET vtb:sweep-lease <instance> NX PX 2×interval, with ±20% jitter on the interval. The holder renews it with a compare-and-PEXPIRE Lua script, so it never extends a lease another replica took over. - A stopped leader may keep its lease until it expires (120 seconds by default).
- Successors wait for that lease before taking over.
- Cache invalidation: best-effort pub/sub on channel
vtb:invalidatewhen an entry goes STALE, is parked REVOKE_PENDING, is deleted, or is replaced by a completed consent. The listener re-subscribes after a dropped connection. Without it, the worst case is still the per-replicaCACHE_TTL_S(default 60s).
If Redis is lost, the broker fails closed on every path that needs redis. These answer with problem JSON 503 coordination-unavailable:
- resolves that need a refresh,
- resolves that need a consent link (so absent, STALE, or insufficient-scope entries get 503, not 404/409),
- authorize,
- DELETE,
- the hub callback when it starts the vendor step.
The browser callbacks (/v1/callback/_hub and /v1/callback/{vendor}), when they cannot read or use up a state, answer with the HTML page "Coordination store unavailable." and status 503. That page carries no problem title. The sweeper skips its tick.
Resolves that need no refresh keep serving. This mirrors the storage fail-closed behavior, with a problem title you can tell apart. The mass-STALE window is not counted while Redis is down, but each broker.stale event is still written.
Alternatives considered
- Postgres advisory locks: support fencing and are durable. But they add a relational database to a service with no relational state. TTL and pub/sub would have to be emulated.
- DynamoDB conditional writes: workable for AWS-only estates. Higher latency on the lock hot path. No pub/sub.
- Storage backend only (CAS, no lock): correct, but burns rotating refresh-token families under concurrency. That is exactly the kind of incident the broker exists to prevent.
Consequences
- Multi-replica deployments gain one infrastructure dependency. Single-replica deployments gain none (
memorydefault). - Consent flows need no session affinity at the load balancer.
- All coordination state can be rebuilt. You can flush redis with no effect on stored tokens: in-flight consents restart and locks form again.
tests/integration/test_multi_replica.pyproves:- exactly one refresh across replicas, with zero refresh-token replays,
- consent that crosses replicas,
- a single sweep leader,
- takeover of abandoned markers,
- redis-down behavior,
- cache invalidation.