# Conformance and measurement
Status: experimental, project-run tests. No external certification or independent implementations are claimed.

## Separate the claims
1. Schema-valid: a profile has the required shape and supported version.
2. Binding-valid: its operation references satisfy the limited OpenAPI binding.
3. Behavior-tested: named client/service versions pass specified failure scenarios.
4. Independently implemented: a separate team has implemented the profile and published its own results.

The web validator checks the first two when an OpenAPI document is supplied. It performs no endpoint calls. The browser lab uses the production reference client algorithm against an in-memory simulated service; it does not create actual support tickets or persist browser state.

## Reproduce
Use Node 24 and Python 3.11 or later. Install the repository's npm dependencies and reference/python/requirements.txt. Run npm test and npm run benchmark.

Tests exercise committed-write response loss, stale reads, unavailable status, incorrect resource identity, pending outcomes, expired ledgers, changed-input key reuse, unsafe operation references, actual loopback HTTP reply loss, concurrent service deduplication, and process termination followed by SQLite recovery in both languages.

Both consumers are authored in this project. Their agreement tests portability but does not count as two independent adopters.

## External-client runner
Run `npm run pilot -- --adapter pilots/adapters/node-reference.json` for seven real HTTP scenarios, including process restart. The [runner guide](PILOT-RUNNER.md) documents the process interface for separately implemented clients. The runner observes submissions, reads, receipts, and stored resource counts. It rejects a claimed success with no service interaction.

[The published reference run](https://ruagentic.org/reports/reference-pilot.json) is project-authored evidence for release `v0.1.0-draft.2`. It is a local single-principal behavior subset, not production or independent compatibility evidence. The runner always leaves independent authorship unverified for later review.

## Benchmark
reports/benchmark.json is generated by scripts/benchmark.ts. It runs deterministic scenarios and reports ticket counts, duplicates, calls, and outcomes. Each strategy receives an isolated service with the same fault.

Baselines include:
- Blind retry: retries with a new ID.
- Idempotent retry: retries with the same ID and reports success from a successful HTTP response.
- Existing verification workflow: separately written orchestration that performs status and resource checks, without consuming an Agentic profile.
- Agentic: the profile-driven reference consumer.

A competent existing workflow is expected to match Agentic's reliability in this example. Any benefit from a shared profile is an interoperability and implementation-effort hypothesis, not a proven advantage in the benchmark.

No production traffic, independent security audit, statistical reliability estimate, adoption number, token saving, or performance claim is established. Deterministic runs are not independent random trials. Latency measurements from a browser simulation would not represent network performance, so none are advertised.

## Known gaps
- Strict duplicate-member detection in JSON text is not implemented.
- The Node consumer supports the documented binding subset. The Python consumer is a smaller implementation limited to the shipped ticket binding.
- Host authorization and account isolation are integration obligations; the reference service is anonymous and loopback-only.
- Browser demo storage is deliberately ephemeral. Durable behavior is tested in the local SQLite implementations.
- Optional WebMCP tools are feature-detected. They require a supporting browser; their browser registry behavior has not been verified in a supported WebMCP context in this release.
- Arazzo/MCP adapters, cross-host leases, signing, generalized retries, and production authentication are not implemented.