LOG-001 Eval engineering June 2026
A passing eval score is not a shipped system.
On TAFI, the second certification run returned QA 100, and platform verification still found five failure classes the eval harness could not see: queries dead-lettered, empty screen containers, workflow commands with no consumer, navigation records missing required columns, and no audit trail. We fixed all five in E13, then ran three consecutive clean pipelines. The lesson we carry into every build: an eval number is necessary, never sufficient. The gate has to verify the real platform state, not just the score.
Read the TAFI case file → LOG-002 Zero-hallucination May 2026
Run deterministic grounding before the model reasons.
VistaGPT cut its hallucination rate from 74.5% to 20.5%, a 72% reduction, not by prompt-crafting, but by structure. Six parallel deterministic ChromaDB stages retrieve evidence with zero LLM involvement, a Reconcile stage drops any element the schema can't confirm, and a sqlglot validator with two auto-retries gates the generated SQL before it runs. Grounding is an architecture decision, not a system-prompt paragraph.
Read the VistaGPT case file → LOG-003 Safety architecture April 2026
Read-only should be enforced by the framework, not a policy.
Orions AI investigates production incidents across StubHub Pro audit logs, per-tenant MongoDB, and source code, and it cannot write to any system under diagnosis. That guarantee lives at the Step Engine level: no agent instruction can trigger a write, and per-tenant resolution makes cross-tenant access structurally impossible (verified by integration tests). "Safe AI" is not a prompt. It is a boundary the model has no path around.
Read the Orions case file → LOG-004 Migration April 2026
Ship new pipeline versions in shadow mode first.
When Orions moved from a 4-agent linear pipeline to a checkpoint-driven open-world loop, v2 ran in shadow mode beside v1 on every live incident behind a DIAGNOSIS_V2 flag, compared at a single endpoint before any traffic shifted. A CI gate blocks the merge if precision drops more than 5 points or calibration error exceeds 0.15. You earn the cutover with evidence; you don't announce it.
How we work →