AGENTIX · CASE FILE / 02 OPERATIONS · STUBSAI / 2026
§ Case file · production system

Orions AI

The AI support engineer that can't touch production

StubsAI is a ticket-resale marketplace wired to 15+ third-party vendors across 127 data collections, and it fields dozens of production incidents a day. A double-sale across two marketplaces. An order stuck "InProgress." A webhook a vendor never delivered. Each is a cross-system, data-driven mystery that used to take a senior engineer hours of pulling logs, vendor API history, database records and source code by hand. Orions runs that whole investigation automatically, through one read-only boundary, and posts back an analysis that separates what it proved from what it only suspects.

Build something like this → How we build multi-agent systems →
0/55 epic-01 stories · read-only enforced
0% assumption rate · code-measured
~0% of sampled incidents reach a proven verdict
0 independent read-only layers
§ The system, live

Watch a diagnosis
run, start to finish.

Real captures from Orions, replayed in the order an incident actually moves: hand it a live incident, watch it stream its reasoning, land a two-tier root cause, and answer the follow-up — never touching production.

Orions AI · Welcome: Hand it a live incident — Orions streams its reasoning as it works.
Orions AI · Diagnosing: Competing hypotheses, evidence pulled read-only, confidence re-scored live.
Orions AI · Root cause: A two-tier RCA: proven separated from suspected, every claim cites its evidence.
Orions AI · Follow-up: Two safe fixes, asked back in plain English — no write access to production.

Hand it a live incident — Orions streams its reasoning as it works.

Open the Orions AI case study
FIG · Orions live replay · real product captures, sequenced welcome → diagnose → root cause → follow-up
§01, The situation

Dozens of incidents a day,
each one a cross-system mystery.

On a marketplace this connected, an incident is rarely "a bug in one file." A sale broadcasts to one vendor but not another. An order hangs because a queue consumer quietly died. A price goes stale because a vendor throttled a webhook. The truth lives across systems, in the database, the logs, a third-party's API history, and the code that tied them together.

Reconstructing it was senior-engineer-only work: recall how the platform behaves, query several backends, cross-reference vendor logs, read the source, form a theory, write it up, for every incident. Expensive, slow, and locked in a few people's heads. Worse, most "AI that explains outages" can't be trusted near a live marketplace, and can't be allowed to touch it.

15+marketplace & POS connectors
127platform data collections
16incident playbooks
Dozensof incidents a day
How it investigates

An investigation,
not a guess.

Every incident runs the same loop, and every step is wrapped by a read-only boundary it cannot escape and an honesty layer it cannot lie past. To be precise about the two numbers below: 15 is the count of governed read-only tools the engine can call (DB, logs, code, KB); 11 is the count of independent fail-closed enforcement layers that guarantee none of them can ever write.

FIG.01, Orions read-only diagnosis loop · incident in, cited RCA out (never a write)
Orions read-only diagnosis loop: an incident from Teams/Web/API/ADO is framed, expanded into 3–6 hypotheses with deterministic confidence, resolved through an evidence loop (gather, classify, re-score), and published as a two-tier (proven vs suspected) RCA comment to Azure DevOps, never a write. Read-only is enforced by one audited MCP egress across 11 fail-closed layers; an uncited 'confirmed' verdict is automatically downgraded to inconclusive.
§ What it pulls together

Everything a senior would,
correlated in one pass.

The hard part of a data-driven incident isn't reading one log, it's holding five sources in your head at once and reasoning across them. Orions does exactly that for every incident.

Platform data
127 collections, Sale, Inventory, BroadcastOrderQueue, Hold…
Production logs
log forensics by inventory / sale id · CompactJson
Vendor API traces
15+ connectors, “is it us or the vendor?”
Source code
grep & read the provider / manager logic, cited at file:line
Knowledge base
code, docs, 16 playbooks & past incident RCAs

Correlated into one evidence-cited RCA, what broke, the exact log line and code path that prove it, and what would fix it, handed straight to your tracker.

§ Why you can point it at production

Trustworthy and safe by construction, both properties live in code, not in a prompt. This is what lets you automate the investigation without automating the risk.

Read-only by construction

Eleven independent, fail-closed layers, the engine refuses to boot unless read-only is on. The only write it can make is a comment. There is no customer-write code path to grant.

Never guesses out loud

Every claim either cites stored evidence or drops to an explicit "unconfirmed" tier, with a checklist of exactly what evidence would confirm it. An uncited "confirmed" verdict is auto-downgraded in code, not by prompt.

Deterministic confidence

Confidence is an integer percent from a six-component weighted model. The same evidence yields the same number, every time. No LLM "vibe numbers," no round-number over-confidence.

§ The output

Proven, separated
from suspected.

A single RCA where a guess can never read as a fact. Every proven claim carries its citations; every suspicion carries the exact evidence that would confirm it.

FIG.02, the two-tier RCA format (illustrative)
✓ Proven, backed by cited evidence confidence 88%

Order stuck in InProgress because the BroadcastOrderQueue consumer was offline for 41 minutes, orders enqueued in that window were never picked up.

db_find: Saleninja_logs: queuecode_read: consumer
⚠ Suspected, likely cause, needs evidence

Probable trigger: a CompanySettings change shortly before the window.

Evidence to confirm→ GlobalConfiguration change history→ deploy log for the incident window
FIG.03, where it lands: an Orions comment on an Azure DevOps work item (representative · illustrative)
BUG #4127 Order 88231 stuck in "InProgress", never broadcast to vendor B Active
Orions AIposted a root-cause analysis · read-only · 41s
PROVEN · 88% BroadcastOrderQueue consumer offline 41 min, orders enqueued in that window were never picked up.
db_find: Saleninja_logs: queuecode_read: consumer.py:142
SUSPECTED Probable trigger: a CompanySettings change shortly before the window, needs config history to confirm.
§ Already shipped an AI system?

The eval harness behind Orions is a service in its own right.

Replay-from-PR grading, a precision/ECE regression gate and an independent AI judge, the same assurance scaffold, applied to your shipped model. No commitment; see what it covers.

AI assurance, eval & red-team → Operations & reliability work →
§02, What we built

Six parts of an
honest investigator.

Each module exists to make an LLM-driven diagnostic engine safe and credible to point at live customer infrastructure.

Diagnosis Flow

One orchestrator routes every incident: frame the problem into domain/vendor candidates, seed competing hypotheses, run the evidence loop, then compose and publish. Teams, web, API and Azure DevOps all drive the same engine.

Competing Hypotheses

3–6 theories per incident, each with a deterministically computed confidence and the exact evidence required to confirm it. No premature collapse to one answer, it reasons across alternatives like a senior would.

Evidence Loop

It pulls DB records, production logs, vendor traces and source code, classifies each finding as supports / contradicts / neutral, and re-scores confidence. It reasons about what the execution actually looked like, not just what the code says.

Read-Only MCP Boundary

One audited data egress: 15 governed read-only tools across DB, logs, code and KB, behind a two-layer allowlist. Secret-scrubbed on read, tenant-fingerprinted cache, cross-tenant leakage is provably impossible.

Knowledge Base

~1,500 docs plus a 46-project source mirror and 16 executable playbooks, embedded into Qdrant, the difference between a junior who has never seen the platform and a senior who has debugged it for years.

The Scorekeeper

Past, already-fixed bugs are replayed cold; a second independent AI judge grades each RCA against the real merged PR that fixed it. A regression gate blocks any change that drops precision >5pp or pushes calibration error over 0.15.

§ How v2 reasons

The checkpoint loop.
Hypotheses that compete.

A senior engineer doesn't fix on one theory and defend it, they hold a few, then let the evidence kill the wrong ones. The v2 pipeline (DiagnosisFlow._run_impl_v2) does the same. It opens with 3–6 competing hypotheses and re-decides their fate at every checkpoint, looping until confidence converges or the budget is spent.

  1. STAGE 2 · sonnet-4-6

    Generate competing hypotheses

    3–6 open-world theories per incident, explicitly competing, not ranked. Each carries a deterministic confidence breakdown and a ceiling. The system's job is to gather evidence that differentiates them.

  2. STAGE 3 · sonnet-4-6

    Plan the next evidence batch

    The EvidencePlanner emits an ActionBatch, which read-only tools to call, with what parameters, in which parallel groups. An invalid batch retries with schema fallback; one planner error never halts the run.

  3. STAGE 4 · deterministic

    Execute in parallel

    The BatchExecutor dispatches the batch through the StepEngine, concurrent within a group, no inference, just tool dispatch and result collection. The same boundary the rest of the engine lives behind.

  4. STAGE 4.5 · sonnet-4-6

    Classify every piece of evidence

    The EvidenceClassifier scores each finding against every active hypothesis, one row per (checkpoint, hypothesis, step). No evidence is left uninterpreted; this per-cell stance is what lets hypotheses actually compete.

  5. STAGE 5 · sonnet-4-6

    Reason at the checkpoint

    The CheckpointReasoner re-scores the full set and chooses the loop's fate: stop (confidence converged), next_batch (gather more, re-prioritised), or open_new (evidence points somewhere unexplored). Decision, rationale and budget snapshot are stored.

  6. EXIT · opus-4-7

    Compose the cited RCA

    When the loop exits, RCAComposer synthesises the evidence set and final scores. Every conclusion cites its evidence_step_ids, citation traceability is held across the whole loop, not one pass. Budget exhausted first? A partial RCA, clearly marked degraded.

At each checkpoint a hypothesis can be,
PRUNE

Contradicted by evidence. Marked inactive and excluded from all future planning, it can no longer pull the RCA toward a wrong answer.

MERGE

Found equivalent to another. One absorbs the other through a merged_into_id self-FK, so two ways of saying the same thing don't double-count.

EMERGE

The evidence points to an explanation no one generated up front. A new hypothesis is created mid-loop. Theories converge toward truth, not toward the opening guess.

FIG.04, EvidenceClassifier · one stance per (checkpoint, hypothesis, step)
supports the finding raises this hypothesis's score
contradicts the finding lowers it, and can trigger a prune
neutral recorded, but moves no score, nothing is silently dropped

Six new tables back this state: hypotheses, checkpoints, actions, evidence_classifications, plus incident_feedback and hypothesis_priors, a nightly beta-binomial posterior keyed on (vendor, signal_shape) that feeds learned priors back into hypothesis generation.

§ The number on every claim

Confidence is computed,
not vibed.

The percentage next to a verdict isn't an LLM picking a number that "feels right." It's a deterministic function of the evidence (confidence.py:159): the same breakdown and the same weights always yield the same integer percent, and a hard ceiling means no hypothesis ever reads as certain.

FIG.05, the confidence ceiling
conf=0.2+0.8·q
a 0.2 floor, even thin evidence isn't reported as zero
only 0.8 is ever earnable from evidence quality q
so the ceiling is 0.2 + 0.8 = 1.0 is unreachable; a verdict tops out short of certainty, by construction
What feeds q a weighted breakdown, deterministic, reproducible, recorded
  1. 01
    Supporting evidence
    how many classified findings support the hypothesis, and how strongly
  2. 02
    Contradicting evidence
    findings classed as contradicts pull the score down, they don't get ignored
  3. 03
    Evidence quality
    a proven DB record or log line counts for more than a weak signal
  4. 04
    Predicate satisfaction
    how many of the hypothesis's confirm_if / refute_if predicates resolved
  5. 05
    Coverage
    whether the planned evidence was actually gathered, or the loop ran short on budget
  6. 06
    Learned prior
    the beta-binomial posterior for this (vendor, signal_shape) from past graded outcomes

Inspect any run live at GET /incidents/{id}/confidence-breakdown, the components and weights that produced the number, not just the number.

§03, How we keep it honest

Graded against the real
pull request that fixed it.

Testing runs on autopilot. We take real, already-fixed production bugs, let Orions diagnose them cold, and grade its RCA against the actual merged PR, with a second, independent AI as the judge.

STEP 1 Replay a real past bug A genuine incident from the 200-PR ground-truth set, diagnosed cold
STEP 2 Judge vs the merged PR A separate AI reads the real fix and grades the RCA on a fixed rubric
GATE Regression block Merge blocked if precision drops >5pp or ECE > 0.15
Measured · code-grounded

On a sampled set of real incidents, Orions made zero unsupported assumptions (0% assumption rate, measured in code) and ~80% reached a proven verdict. The judge that produced those grades is a different model than the one being graded, with all tools disabled and isolated billing, so the engine can never mark its own homework.

Confidence is deterministic and reproducible; every LLM call, evidence step and redaction is recorded. Trustworthy automatic diagnosis is the hard part, and the foundation any safe auto-remediation has to stand on.

DIAGNOSIS_V2 v2 reasoning runs in shadow mode, graded before it ever ships

A new diagnosis model goes live behind the DIAGNOSIS_V2 flag and runs shadow against production traffic, scored, never published, until the numbers hold.

7 golden YAML fixtures, locked, hand-graded incidents the v2 run must reproduce
>5pp precision drop vs the v1 baseline auto-blocks the PR
0.15 expected-calibration-error (ECE) ceiling, exceed it and the merge is blocked
§ Under the hood

Not a demo.
A five-repo system.

Orions is a Git superproject coordinating three submodules, pipeline, tool gateway, knowledge base, built over six weeks, April 13 to June 16. The depth lives in the parts you don't see in a screenshot.

296 orions_ai_eco_system superproject · .gitmodules, bootstrap / doctor / sync scripts, CI orchestration, the v2 architecture spec
214 orions_ai_intelligence the pipeline · FastAPI on :18080, intake, incidents, actions, LLM-calls, the Teams bot, eval & replay endpoints
79 orions_ai_mcp the tool gateway · HTTP/SSE MCP on :18765, every tool read-only by implementation, catalog persisted & drift-checked
47 orions_ai_knowledge_base the domain KB · 18 issue blueprints, the stubscore-ai-kb, and StubsCore tracked as a nested submodule
~652 total commits 55/55 epic-01 stories, f01–f08, and delivery continued on develop_with_sdlc past the epic
ARTIFACT · stubscore-ai-kb

The platform, documented for machines

The entire StubsCore .NET automation platform, 45+ projects on StubsAI.Platform.sln, written up in an AI-optimised format for LLM, MCP and RAG retrieval. Generated by Claude Opus 4.7, the StubsCore solution itself tracked as a nested submodule.

  • Architecture, coding conventions, DB schemas, dependency maps, per-module docs, integrations, troubleshooting patterns
  • Machine-readable indices, manifest.json, tags.json, modules.json, cross-links.json, so the retriever navigates the KB structurally
  • Reached at diagnosis time through kb_query; code_grep / code_read drop to the source when an implementation question arises

Knowledge that used to live in developers' heads, now queryable on every run.

QUALITY FLOOR · GitLab CI

A merge gate, not a bolt-on test suite

Every merge request touching diagnosis or prompt files is scored against 7 golden YAML fixtures with known root causes, precision, recall and calibration (ECE) computed before the code lands.

BLOCKS THE MERGE WHEN
> 5ppprecision drop versus the v1 baseline
> 0.15expected calibration error (ECE)
  • diagnose replay reproduces any past incident with cached LLM responses, catches prompt drift before the gate runs
  • A doctor job validates submodule integrity on every merge request
  • Per-tenant isolation isn't assumed, end-to-end two-tenant tests confirm zero cross-tenant access in concurrent runs

Production infrastructure, gated into the workflow from day one of epic completion.

§04, Results

What changed,
measured.

Senior-engineer-only investigation, compressed into a bounded, auditable, safe-on-production run.

Senior-only triage, hours ≤ 240s · ≤ $5

every diagnosis is bounded, 240s, a $5 hard cost ceiling, 250k tokens, 60 steps

Who untangles a cross-vendor incident a scarce senior engineertier-1 + Orions
Unsupported assumptions in an RCA confident guesses0% · code-measured
Proven-verdict rate , ~80% of sampled incidents
Write access to production whatever the model triesnone · 11 fail-closed layers
§ Stack

Open-source.
Self-hosted. Owned.

Multi-provider LLM routing with a hard cost ceiling, no lock-in, and a financial circuit-breaker on every incident.

PythonFastAPICrewAIFastMCPQdrantLiteLLMClaude + GPT-4.1SupabaseReact 18SQLAlchemyAzure DevOpsMicrosoft Teams
Start a project

Drowning in cross-system
incidents?

If your platform is wired to dozens of vendors and triage is senior-only, we'll architect a read-only, honesty-enforced diagnostic engine that investigates the data-driven mess for you, and show you the eval it has to pass.

Scope an incident-diagnosis engine → See the multi-agent approach →
279/279 TAFI Natural-Language App Builder Case file → 95.3/100 VistaGPT Construction ERP Intelligence Case file →
AGENTIX TECH · case file 02 · Orions AI · read-only by construction