Benchmarks

All docs

Every figure on this page was measured on the live hosted deployment with bench/benchmark.py. Nothing is modelled, extrapolated, or rounded to something friendlier. Where a mode could not be measured, it says so and reports nothing.

Reproduce it yourself:

CAIN_API_KEY=<your key> python3 bench/benchmark.py --samples 300 --concurrency 16

Latency#

Measured 2026-08-17 against https://cainstudio.online, deployment in enforce mode. 1200 samples per mode after 5 discarded warm-up requests.

mode	median	p95	p99	n	ok
OFF (no decision path)	10.51 ms	15.58 ms	18.78 ms	1200	100%
ENFORCING (full path)	20.79 ms	32.00 ms	44.40 ms	1200	100%
SHADOW	*not measured*
ENFORCING + ActionProof	91.27 ms	130.06 ms	181.92 ms	1200	100%

Fabric overhead: 10.28 ms median, 16.42 ms p95. ActionProof adds 70.48 ms median on top of that, when a plan is submitted.

p99 is reported here because 1200 samples is enough to place it. At 300 samples it was withheld as n/a rather than printed — see below.

Throughput#

mode	concurrency	rps	successful rps	non-200
ENFORCING	24	99.5	99.5	none

1007 requests over 10 seconds, every one a 200. Note that raising concurrency from 16 to 24 did not raise throughput (108 rps → 99 rps): the decision path is not concurrency-limited at this level, so adding load adds queueing rather than work. That is worth knowing before sizing a deployment, and it is why the figure is published with its concurrency attached rather than as a bare "rps". The successful-rps column is reported separately from total rps on purpose: a throughput number with the errors hidden is the most common way a benchmark lies, and 400 rps where a third of responses are 429 is not 400 rps of useful work.

How to read these numbers honestly#

OFF is not "CAIN uninstalled". It is a request to the same host on a route that does not enter the decision path. That isolates fabric cost from network latency to this box, which is the number you actually want when deciding whether to adopt. It is *not* a measurement of your agent calling its own tools locally with no CAIN at all — that number depends entirely on your network, not ours.

p99 is withheld when there are too few samples. The benchmark refuses to print a percentile unless it has at least 5× the samples needed to place it, because a "p99" from 300 observations is roughly the third-worst one wearing a formal name. The figures above come from 1200 samples, which is why p99 appears. Run with --samples 300 and it correctly reports n/a instead.

SHADOW is not measured, and not estimated. Shadow versus enforcing is a live property of the deployment, set by a global switch. Measuring shadow would mean flipping that switch on production traffic, so the benchmark reports not_measured with the reason rather than printing a plausible number.

ActionProof is the expensive stage, and that is expected. It discharges real SMT queries about a submitted plan. 68 ms to prove a two-step plan cannot violate your constraints is a different kind of work from a policy lookup, and it only runs when you submit a plan. If you never send one, you never pay for it — the stage reports skipped.

Failure behaviour#

The benchmark also measures what happens when a stage has nothing configured. A verification stage with no profile returns not_configured promptly; it does not hang waiting for a verifier that has no constraints to check. That matters because a control which stalls under a missing configuration becomes an outage rather than a refusal.

What is not benchmarked yet#

• Sustained load over hours. These are seconds-long windows. Nothing here

says anything about behaviour under sustained pressure, memory growth, or connection-pool exhaustion over a day.

• Concurrency beyond 24. Two levels measured (16 and 24); throughput did not

improve, so the useful ceiling is somewhere at or below 16 concurrent callers on this hardware. Nothing here characterises 100+.

• Self-hosted MCPGate. Its performance depends on your hardware, so a number

measured on ours would be misleading rather than useful.

• Cold start. Warm-up requests are discarded, so these are steady-state

figures. The first request after a deploy pays TLS setup and is slower.

These gaps are listed rather than quietly omitted, and they are on what CAIN does not do yet.