Overlap is real
Similar tools can compete for the same request, especially at semantic boundaries.
Independent evidence for agent/tool reliability
Find where agents choose the wrong MCP tool.
Start with a free metadata scan. Get real-model selection evidence for one MCP, then measure whether a description change helps on the same frozen cases.
Start with a free metadata scan.
For a quick description-derived routing check, paste a public MCP URL. This existing scan reads tools/list, generates description-derived routing cases, and never executes your MCP tools. It is generated-readiness evidence, not untouched natural-language or real-model routing accuracy.
31measured subjects
24S2-backed
12multi-batch
17preserved misses
Why this matters
Tool access is not the same as tool selection.
Agents can have many tools available and still choose the wrong one before any tool is executed. Descriptions can overlap semantically, so evidence about actual selection adds context that a directory listing cannot provide.
Descriptions are not results
Tool metadata explains an interface; it does not show how a model selected among alternatives.
Evidence travels further
Frozen benchmark measurements give users a traceable basis for comparison.
What we measure
A simple evidence trail.
Inputs are fixed first.
Tool schemas, prompts, expected tools, and model settings are frozen before measurement.
Selection comes before execution.
The model selects a tool; target business tools are not executed as part of this measurement.
The record stays auditable.
Correct selections, semantic misses, provider failures, provenance, and uncertainty remain distinct.
Evidence strength
Two clear evidence levels.
Measured on a frozen constructed benchmark.
Measured against independently sourced external gold cases.
S1 and S2 describe the evidence basis for selection measurements. Neither is a certification of overall MCP quality, transport, business correctness, or safety. See the methodology notes.
Browse the evidence
Reliability Explorer
Browse measured subjects, filter by evidence strength, inspect multi-batch history and observed misses, distinguish live from version-bound evidence, and compare 2–5 subjects.
DeepWiki MCP
deepwiki-mcpMulti-batchS2-backedObserved missesMixed provenance
Mixed live and version-bound evidence
- Independent batches
- 6
- Measured cases
- 52
- Evidence
- S1, S2
- Observed strict accuracy range
- 57.1%–100.0%
- Observed semantic misses
- 5
- Latest measurement
- 2026-09-10 19:13:49.842728 UTC
data.gouv.fr
datagouv-mcpMulti-batchS2-backedObserved missesMixed provenance
Mixed live and version-bound evidence
- Independent batches
- 6
- Measured cases
- 39
- Evidence
- S2
- Observed strict accuracy range
- 83.3%–100.0%
- Observed semantic misses
- 1
- Latest measurement
- 2026-09-11 00:07:39.419700 UTC
NixOS
nixos-mcpMulti-batchS2-backedObserved missesVersion-bound
Version-bound source snapshot
- Independent batches
- 3
- Measured cases
- 21
- Evidence
- S2
- Observed strict accuracy range
- 83.3%–100.0%
- Observed semantic misses
- 1
- Latest measurement
- 2026-09-11 00:07:39.419700 UTC
Exa MCP
exa-mcpMulti-batchS2-backedMixed provenance
Mixed live and version-bound evidence
- Independent batches
- 6
- Measured cases
- 55
- Evidence
- S1, S2
- Observed strict accuracy range
- 100.0%–100.0%
- Observed semantic misses
- 0
- Latest measurement
- 2026-09-11 00:07:39.419700 UTC
Credibility through preservation
Misses are not removed from the record.
When the model chooses the wrong tool, that result remains part of the evidence. The Explorer keeps observed semantic misses attributable to their benchmark, batch, and case while keeping provider failures separate.
Descriptive comparison
Compare evidence, not marketing claims.
Compare measured cases, independent batches, S1/S2 evidence, observed score ranges, semantic miss history, and provenance. The comparison describes evidence amount and history; it does not declare a winner.
Evidence discipline
Built for traceability.
- Frozen inputs and immutable measurement receipts preserve what was measured.
- S2 uses independent external gold cases.
- Provider failures remain separate from semantic misses.
- Results are retained as measured evidence without post-result cherry-picking.
For developers
Use structured evidence in your own workflow.
Public reliability evidence is available through the API and OpenAPI document.