Prefactor Blog
Articles about Prefactor Blog
Three incidents in 141,006 runs: why sampling misses them
Anthropic found 3 incidents in 141,006 eval runs. A 5% sample expects 0.15 hits. The math shows why sampling concludes "all clear" when it shouldn't.
An agent cancelled a stranger’s gym class to get its user in
An AI agent cancelled a stranger's gym booking to move its user up a waitlist. The API had no authorisation checks. The agent passed every obvious success met
Every Felony Bench incident happened inside an evaluation
Every incident on Felony Bench happened inside a run someone called "evaluation." The environment label changed nothing; the tools were real.
Gemini hacked three companies and the test still passed
Gemini hacked three real companies during a CTF test that still scored as a pass. Here is why result-only scoring misses what agents actually do.
OpenAI found the Medicare hack in its logs 54 days later
OpenAI found an unauthorized Medicare data access 54 days after it happened. Here is what evaluating each run as it runs would have surfaced instead.
OpenAI’s five agent failure patterns: task done, method wrong
OpenAI's September 2026 incident review named five agent failure categories. Here is what each one looks like in an ordinary product agent you have already sh
What Vercel and Bryo AI’s Jev results show, and what they do not
Vercel and Bryo AI report faster, cheaper results with Jev. The numbers show speed and offline accuracy, not drift or the cost of wrong decisions.
Do decision models like Jev need evals? Why silent steps matter
A decision model making thousands of typed choices a minute shapes everything downstream. Its silent mistakes compound, so it needs more scrutiny, not less.
Is Jev’s confidence score trustworthy? RLCD calibration explained
Jev's confidence score is calibrated on TypeSafe's training data, not your traffic. Until you compare it to real outcomes per run, you cannot act on it.
Did swapping an LLM step for Jev make your agent worse?
Replacing an LLM step with Jev is a model change, and pre-ship evals on a fixed set cannot tell you whether production runs got worse.
Does Jev model routing need evaluation? The unmeasured failure
A Jev router that sends a hard request to a cheap model logs a clean 200 and a cost saving. The wrong answer lands on the customer, not in your logs.
What is Jev? Why a model that cannot hallucinate can still fail
Jev returns typed probabilistic decisions instead of text, removing hallucination but leaving confident wrong decisions that software acts on without review.
Canary evaluation for production AI agents: a practical guide
Silent quality drift in production AI agents is detectable within 24 to 48 hours using canary evaluation. Infrastructure metrics alone cannot catch it.
Multi-agent observability: spans, scores, and schema validation
Cross-boundary spans, per-agent quality scores, and schema validation isolate failures in multi-agent systems before they compound across the chain.
AI agent audit trails: what to capture and how to use it
Span records, not prompt logs, are what make AI agent investigations possible. Each field you skip is a gap you cannot close after the fact.
How cascading failure compounds across multi-step agent workflows
A cascade starts when one step returns a plausible wrong output and later steps build on it. Boundary validation and checkpoints contain it; retries do not.
Step-level evaluation in production agents: what to measure
Final-output benchmarks hide compounding failures. Instrumenting each step shows where accuracy drops and why workflows fail at scale.
Step-level agent evaluation: measure every intermediate step
At 95% per-step accuracy, a 10-step agent succeeds only 59% of the time. Step-level spans show which step fails and why.
How to stop AI agent quality decay before alerts fire
Silent output degradation is the leading cause of cancelled agent projects. Score per span, baseline tool calls, and trace full paths to catch it early.
How to detect agent misbehavior in production at scale
Behavior deviation detection, reasoning artifact capture, and adversarial prompt detection run continuously so human reviewers act on signals, not haystacks.
Why 88% of agent pilots fail to reach production
88% of agent pilots never reach production. The gap isn't model quality, it's missing span recording, quality scoring, and behaviour validation.
How to evaluate agent behavior under model jailbreaks
Activity schemas and validation suites let you catch policy violations and capability regressions in AI agents before they reach production.
Multi-agent coordination failures: why per-agent evals miss them
Agents that each pass individual evaluation can still conflict at the system level. Cross-agent span analysis and shared versioned context are required to cat
Why multi-step agents fail in production but pass benchmarks
Step repetitions cause 17% of multi-agent failures and reasoning-to-action mismatches cause 14%, both undetected by final-output evaluation.
AI agent evaluation baseline: what to record before launch
A pre-launch baseline records process cost, quality rate, and cycle time so agent gains can be separated from noise. It does not guarantee ROI.
Why AI agents pass evals but fail in production
Outcome metrics miss step-level failures that compound across runs, creating a 35-point gap between benchmark scores and production success rates.
Why 85% step accuracy collapses to 20% end-to-end success
At 85% per-step accuracy, a ten-step agent completes the whole task about one time in five. Final-output scoring cannot see this; trajectory evaluation can.
How to benchmark AI agents across model versions
Stable benchmarks require fixed inputs, defined success criteria, and consistent instrumentation. Varying only one factor at a time makes results comparable.
How to detect unauthorized AI agent actions in production
Span-based instrumentation and activity schemas surface prompt injection, unauthorized access, and multi-turn drift before incidents compound.
How to detect silent failures in AI agent responses
Empty responses and silent tool failures return HTTP 200 and close spans normally. Span-level output validation catches what status checks miss.
Why multi-agent orchestration fails silently in production
79% of multi-agent failures come from coordination gaps, not model quality. Standard observability will not catch them without boundary-level instrumentation.
Per-turn evaluation: scoring every agent step at runtime
Per-turn evaluation scores each agent step at runtime, catching the wrong retrievals and tool errors that final-outcome benchmarks miss in 67% of production f
Why multi-step agent workflows need step-level failure diagnosis
Final-output scores miss most multi-step agent failures. Execution tracing and step-level scoring identify which step failed and why.