Benchmark

This document contains two different measurements. The live-read benchmark was executed against a running app. The bug-fix pilot is a reproducible runner and manifest, but its comparative agent runs have not been executed yet, so it does not support a claim about bug-fix success. Prompt-token figures below are client-reported deltas: they measure what those client versions reported, not the full serialized size or guaranteed context occupancy of tools/list.

Two questions get asked about a hub that hands an agent 58 tools. What does that cost the agent in context, and does the agent actually come back with the right answer about the running app. Both are measurable, so neither is argued here.

Everything below was measured against a real Expo app on a booted iOS simulator, with a device connected to the hub, across the three CLIs the hub targets. The numbers come from each client's own usage reporting, not from an estimate.

What the hub costs an agent in context

Attaching the hub added between 150 and 800 tokens to the prompt usage reported by these client versions. On a 1M context window that is under a tenth of a percent if the reported delta represents context occupancy. The tool list also sits at the very front of the request, where it is the most cacheable part of the prompt, and tools/list is byte-stable across calls, so the cache prefix holds.

The cost scales with the number of tools, and it is close to linear.

13.3, 13.1 and 12.9 tokens per tool at 10, 33 and 65 tools. The remaining question is what those tokens are: the names, or the schemas.

Inflating every description until the payload is 4.2 times larger changes the client-reported injected cost by zero tokens. Shortening the tool names does move it. This is consistent with that client loading schemas on demand and injecting names, but usage reporting alone cannot prove the internal mechanism: the client may account for schemas outside the reported prompt-token field.

In this client version, trimming descriptions did not change the reported prompt-token field. The experiment cannot establish that schemas are free: a client may load, cache or account for them outside that field. Tool count and serialized schema bytes must therefore be reported separately.

Whether an agent reads the app correctly through the hub

Nine questions about the running app, each answerable by one read-only hub tool: the slowest endpoint by p95, how many endpoints are tracked, total calls, how many endpoints have errors, the crash count, the connected device name, the declared expo and react-native versions, the registered debug actions.

Every client answered every retained, completed round correctly.

The interesting part is what it took to measure this honestly. A first run scored 50 to 65 percent, and the failures were identical across all three clients, which is not how three different models fail. They were not failures: the app kept running, and the reference values captured at the start had gone stale. The slowest endpoint had moved, the crash history had rolled over on an app restart, the endpoint count had changed. The agents were reporting the truth at the moment they were asked, and the benchmark was grading them against the truth from forty minutes earlier.

So ground truth is now captured directly from the hub immediately before and immediately after each round. When it moves during the round, that round is discarded for all three clients at once, because the fact changed rather than the agent being wrong. One round out of 18 was discarded that way.

That volatility is visible in the retained data and is the point: between the two repetitions the slowest endpoint went from GET /v1/express/config to POST /v1/auth/login and the tracked endpoint count went from 11 to 14. Every client tracked the change both times. A hub that returned a cached or stale view would have scored well against a fixed reference and badly here.

Two devices at once

The questions above were asked of a single connected device. Adding a second one changes the problem: a hub that holds two apps has to be asked which app, and an agent has to work that out.

So a physical Samsung Galaxy A16 5G (SM-A165M), Android 16 was added next to the iPhone 16 Pro Max, iOS 18.4, both running the same app, both connected to the same hub. The Android side is a development build (the app depends on expo-dev-client and native modules Expo Go does not carry), installed over USB with Metro and the hub reached through reversed ports.

With two devices attached, every device-scoped tool stops answering and says so, naming both candidates:

2 devices are connected, so the target is ambiguous: pass deviceId.
Connected: s-SimulatoriOS-...-ios (@example/mobile on iPhone 16 Pro Max),
           s-SM-A165M-...-andr (@example/mobile on Galaxy A16)

That refusal is the feature under test. Ten questions were asked, each naming its target device, with no hint in the prompt that disambiguation would be needed. The agents had to hit the refusal, resolve the device themselves and retry scoped.

The two devices held different data, which is what makes a wrong answer visible: the hub had statistics for 13 endpoints on the simulator and 1 on the phone. An agent that scoped to the wrong device answers 13 where 1 is correct, and none did.

The single miss is real rather than a scoring artifact: on one round Cursor answered that one device was connected when two were. It is the only genuine wrong answer across both accuracy experiments, and it is left in the count.

Nothing drifted during these 20 rounds, so none were discarded.

Method

App: an Expo delivery app, React Native 0.86, Expo SDK 57. Connected to the hub over WebSocket from an iPhone 16 Pro Max simulator and, for the two-device section, from a physical Samsung Galaxy A16 5G on Android 16 running an EAS development build installed over USB.
Clients: Claude Code 2.1.251, Codex CLI 0.147.0, Cursor CLI 3.9.16, each connected to the same hub over the local MCP endpoint.
Injection cost: same trivial prompt with and without the MCP server declared, 3 runs per arm, median reported. Prompt tokens are the sum of fresh, cache write and cache read from each client's usage object.
Tool count and schema size: a controlled MCP server serving a chosen slice of the hub's own tool definitions, so the names and schemas are the real ones.
Accuracy: 9 questions, 2 repetitions, three clients per round, run in parallel against one hub. Agents were restricted to read-only tools and to a single call, and were told not to read project files.
Two devices: 10 questions, 2 repetitions. The single-call restriction is lifted, because resolving a device takes a call of its own, and the prompt says nothing about disambiguation.
Scoring: the expected pattern is derived from the captured value, never written by hand.

The exact model identifiers and the raw transcripts were not retained with the published document. The CLI versions and aggregate results are therefore auditable only as reported here, not independently reproducible. Future runs must store the prompt, model identifier, raw client output, ground truth before and after the round, scoring output and any discarded attempt together.

Limits

These are single-tool lookups, not multi-step debugging. They measure whether the hub delivers correct live facts to an agent and whether the agent reads them, which is the hub's contract. They do not measure whether an agent can diagnose a bug.

With 17 retained rounds per client, a 100% score carries a 95% interval reaching down to about 82%. The result rules out a broken path, not a rare failure mode. One Cursor run out of 18 exceeded a 260 second deadline and was counted as missing rather than wrong. Cursor's conditional answer accuracy was therefore 16/16, while its end-to-end completion rate on retained rounds was 16/17 (94.1%). Both numbers are required: excluding the timeout from the accuracy denominator must not make the operational result look like 100%.

The Wilson intervals treat answers as Bernoulli observations, although answers are clustered by app, machine, round and underlying hub state. They are descriptive intervals, not evidence that 17 independent app or environment samples were tested.

All of it is one app on one machine. Another app with different traffic would be a different sample, and the injection numbers belong to the client versions listed above, which change often.

The scoring code is itself a source of error, and was twice: an early version graded three questions against a pattern built from a non-numeric value, which turned three correct answers into failures across every client at once. It now refuses to build a matcher it has no rule for instead of quietly producing one that can never match. Both accuracy sections were scored again from the same recorded answers after that fix; no agent was re-run to improve a number.