← Back to Reliability Explorer
Measured selection evidence
GitMCP Docs
gitmcp-docs
Measured historical tool-selection reliability from frozen benchmark evidence.
Multi-batchS2-backedObserved missesMixed provenance
Mixed live and version-bound evidence. Historical evidence does not establish current endpoint availability or current production behavior. Check observation freshness and schema changes.
3independent benchmark batches
35measured frozen cases
S1, S2evidence level(s)
3observed semantic misses
First measurement: 2026-09-10 13:23:12.077611 UTC · Last measurement: 2026-09-10 17:45:38.245198 UTC · Observed strict accuracy range: 66.7%–100.0%
Batch history
Each immutable benchmark batch is shown separately. No pooled accuracy is calculated here.
| Benchmark / batch | Type | Frozen cases | Strict score | Wilson 95% CI | Provider-valid coverage | Infrastructure failures | Evidence / measured |
|---|---|---|---|---|---|---|---|
selection-evidence-expansion-v1selection-evidence-expansion-v1 |
CONSTRUCTED_FROZEN_SELECTION_BENCHMARK | 24 | 24/24 100.0% |
Not recorded–Not recorded | 24 / 24 (100.0%) | 0 | S1 2026-09-10 13:23:12.077611 UTC |
external-gold-expansion-v2-batch2external-gold-expansion-v2-batch2 |
EXTERNAL_INDEPENDENT_GOLD_BENCHMARK | 5 | 4/5 80.0% |
37.6%–96.4% | 5 / 5 (100.0%) | 0 | S2 2026-09-10 16:49:53.346446 UTC |
external-gold-expansion-v2-batch3external-gold-expansion-v2-batch3 |
EXTERNAL_INDEPENDENT_GOLD_BENCHMARK | 6 | 4/6 66.7% |
30.0%–90.3% | 6 / 6 (100.0%) | 0 | S2 2026-09-10 17:45:38.245198 UTC |
Observed semantic misses
These are preserved adjudicated historical observations, attributable to a benchmark, batch, and case.
| Benchmark / batch | Case ID | Expected tool | Selected tool | Boundary family | Adjudication |
|---|---|---|---|---|---|
external-gold-expansion-v2-batch2external-gold-expansion-v2-batch2 |
gitmcp-hard-005 | search_generic_documentation |
fetch_generic_documentation | DOCUMENTED_BEHAVIOR_VS_IMPLEMENTATION | SEMANTIC_MISS_GOLD_REMAINS_VALID |
external-gold-expansion-v2-batch3external-gold-expansion-v2-batch3 |
b3-gm-005 | fetch_generic_url_content |
fetch_generic_documentation | DOCUMENTATION_VS_REFERENCED_URL | GOLD_REMAINS_VALID; semantic miss preserved in strict score. The public source supports the requested neighboring operation, and the selected tool is a plausible but incorrect neighboring retrieval mode. |
external-gold-expansion-v2-batch3external-gold-expansion-v2-batch3 |
b3-gm-006 | search_generic_documentation |
fetch_generic_documentation | DOCUMENTED_BEHAVIOR_VS_IMPLEMENTATION | GOLD_REMAINS_VALID; semantic miss preserved in strict score. The public source supports the requested neighboring operation, and the selected tool is a plausible but incorrect neighboring retrieval mode. |
Evidence provenance
Technical identities are secondary detail for auditability.
GitMCP Docs
- Canonical subject ID
gitmcp-docs- Provenance type
- CONSTRUCTED_FROZEN_SELECTION_BENCHMARK
- Identity mode
- LIVE_TOOLS_LIST_SNAPSHOT
- Identity source
LIVE_TOOLS_LIST
GitMCP
- Canonical subject ID
gitmcp-docs- Provenance type
- EXTERNAL_INDEPENDENT_GOLD_BENCHMARK
- Identity mode
- LIVE_TOOLS_LIST_SNAPSHOT
- Identity source
CURRENT_LIVE_TOOLS_LIST_FROZEN_BY_EXISTING_IMMUTABLE_SNAPSHOT
GitMCP
- Canonical subject ID
gitmcp-docs- Provenance type
- EXTERNAL_INDEPENDENT_GOLD_BENCHMARK
- Identity mode
- VERSION_BOUND_SNAPSHOT
- Identity source
CURRENT_OR_VERSION_BOUND_SNAPSHOT_REUSED_FROM_QUALIFIED_CORPUS
selection-evidence-expansion-v1 immutable identities
- case_freeze_sha256
5d248e845fc1b52ef53585682a97a592fecd0d3c1b8c9e561c758a4222273aed- freeze_commit_sha
d2937086b419ba0bf77517da772540de1975c45f- measurement_artifact_sha256
27f281b49b50fe5218102274c57db0110f3119091ca5dd53ebe954997f4fe826- model_lock_sha256
a8df78457b76e0dea73f66e2a75907feab3c42885db23658caada6973c5cb478- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d
external-gold-expansion-v2-batch2 immutable identities
- case_freeze_sha256
ce21fc085bd9a0185c9e37d0d795221f286397cbc477ee941e876be24f7ba1b4- final_freeze_receipt_commit
d3c55d2f96a4e45bf2577225ea16929877f2ded1- freeze_input_commit
61f6511c41562c24019cd441b37bc6ee07283951- measurement_artifact_sha256
dd13aec8a90b2b5490c934c9ad9650c37733c65f18d824e2ae6066202176ced9- measurement_commit
f0b6cd33be044d8bad800c20b4f2684e6d8431ce- model_lock_sha256
153647f94eed0ee3d3069229545846eef864fd43fa821f2d311dc3fb5d265ba9- policy_evaluation_commit
f0b6cd33be044d8bad800c20b4f2684e6d8431ce- policy_evaluation_sha256
736f154a5d5c3b8f0d77cd58fa0644ca97ad94c7a6b46af98d394c1dc1578f9a- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d- raw_predictions_sha256
8c4364ab598241ddbe4adb16453ba2e12b0917a1d1852caedb07884d1a54570b
external-gold-expansion-v2-batch3 immutable identities
- case_freeze_sha256
c1595743f48fe8ae6b3f70f2ea401731052b3bd7614297feabaa8172ce190ae5- final_freeze_receipt_commit
a309bda7e8dd2867ccc7894dc7d42efbfc48395e- freeze_input_commit
8450ed889756784a12ca4ebb8d84974d73b31778- immutable_measurement_receipt_commit
9d8290160849a5124d1896fc318d53470585ef41- measurement_artifact_sha256
7dcab9d4b71e7de59094c07cbca830ebf674b601ac6de024db0d3e677e7c0067- measurement_commit
b81cc816158f7c0fbe7f8ac0a6fcf2b3f38f4c86- model_lock_sha256
48b5ed9c98f16cc2b3b76bc0da2e7ec8072ecd89ba1914683181b1c20074a3a8- policy_evaluation_sha256
3ea5b62044b4da6e92812831ebb00205970849accffe763a270248a24a0adef4- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d- raw_predictions_sha256
c5cfe765f6b72489220cc73f689acb1c1c2c9284ffd8d093de0e597b645f6fbf