AGENTIX · ENGINEERING LOGPRODUCTION / 2026
Production5 min read

Read-only first: how an AI agent earns a production cutover

Shadow mode plus a CI gate on precision and calibration is the honest way to ship. Orions v2 proves it with ~200 real PRs and an independent judge.

By Agentix Tech · Engineering log

Orions v2 shadow mode pipeline: production traffic splits to live v1 and scored v2, a CI gate on precision and calibration decides the cutover

Most teams ship an AI agent by flipping a flag and hoping. The new thing goes live, users become the test suite, and regressions surface as support tickets. That is not a release process — it is an incident waiting to happen.

Orions is the incident-diagnosis agent that runs for a busy marketplace. When we rebuilt its reasoning from a linear four-agent pipeline into a checkpoint-driven loop that holds competing hypotheses, the change touched the most sensitive part of the system: the decision that names the root cause of an outage. We did not flip a flag. We ran the new pipeline in shadow, beside the live one, on every real incident, and forced it to earn the cutover with evidence.

~200fixed PRs as ground truth
7golden YAML fixtures gate every merge
~80%proven verdict, 0 unsupported

Shadow mode is not a vibe — it is a data pipeline

The v2 pipeline went live behind a DIAGNOSIS_V2 flag. Every production incident was diagnosed by both versions simultaneously. The v1 output served users. The v2 output was scored against the known root cause, stored, and never posted. No human compared outputs by eye — that does not scale. The comparison is automated, continuous, and the gate lives in CI.

Production incidents v1 — live serves users v2 — shadow scored, stored, silent CI gate precision drop < 5 pts calibration err < 0.15 cut over
Users keep the current behaviour. The candidate earns the cutover on evidence already gathered.

A system that is confidently wrong is worse than one that is uncertain and says so. The gate checks precision and calibration, and it runs before the code lands, not after.

The gate that makes shadow mode mean something

Watching two outputs side by side is not a gate; it is a vibe. The gate has to live in CI, and it has to say no on its own.

For Orions, every merge request that touches diagnosis or prompt files is scored against seven golden YAML fixtures — real incidents with known root causes — and the merge is blocked if:

Metric Threshold Measured against
Precision drop vs v1 baseline < 5 percentage points 7 golden fixtures
Expected calibration error (ECE) < 0.15 ~200 fixed PRs
Unsupported verdict rate 0 full ground-truth set

The ground-truth set is drawn from approximately two hundred already-fixed production PRs. The judge is a second, independent model with tools disabled and separate billing, so the engine can never grade its own homework. This is the read-only principle applied to evaluation: the judge only reads, never writes, and it is structurally separated from the system it grades.

Read-only first is a structural choice, not a policy

Orions v1 already had a read-only boundary: 15 typed MCP tools, a two-layer allowlist, and it refuses to boot unless read-only mode is enforced — there is no write code path to exploit. The v2 rebuild preserved that guarantee. The shadow pipeline inherits the same boundary: it can read logs, metrics, and deployment history; it cannot post to Slack, open tickets, or restart services.

Boundary v1 v2 (shadow)
Transport 15 read-only MCP tools same 15 tools
Write paths 0 (structural) 0 (structural)
Allowlist two-layer, booted at startup same
Act-as-user N/A (read-only) N/A (read-only)

The read-only architecture means the shadow pipeline can run on real production traffic without a safety review for every incident type. The worst it can do is store a wrong diagnosis — which the CI gate catches before it ever reaches a human.

Calibration is the metric that catches confident hallucinations

Precision alone is not enough. A model that says “database” for every incident might hit 60% precision if databases cause 60% of outages, but its confidence is meaningless. Calibration measures whether the stated confidence matches the empirical accuracy.

Confidence bin Predicted Actual accuracy Gap
0.9–1.0 23% of verdicts 91% +2 pts
0.7–0.9 34% of verdicts 78% -1 pt
0.5–0.7 28% of verdicts 54% -4 pts
< 0.5 15% of verdicts 38% +7 pts

The expected calibration error (ECE) across the full set is 0.08 — well under the 0.15 gate. When v2’s ECE drifted to 0.18 during development, the CI gate blocked the merge. That drift came from a prompt change that made the model overconfident on network-partition hypotheses. The gate caught it; a precision-only gate would have missed it.

The sequence is the honest one

The order matters. You do not run shadow mode after you ship. You build the candidate, run it in shadow on real traffic, grade it against known-correct outcomes with an independent judge, gate the merge on precision and calibration, and only then cut over. By the time traffic shifts, the decision has already been made by the data.

See the shadow-mode and CI-gate details in the Orions case file, or tell us what you are shipping at cal.com/agentix-tech.

Work with us

Building a system
that has to hold up?

Tell us what you're building. We'll tell you how we'd architect it, what the eval harness would cover, and what production deployment involves.

AGENTIX TECH · engineering log · Production