Production Notes
Articles about Production Notes
AI Agent Evals in CI/CD: Flakes, Retries, and Missing Results
AI agent evals in CI/CD only help if red means a real regression. Opt in, fail missing tasks, pin trial id to the trace, and separate flakes from agent bugs.
Multi-Agent Handoff Testing: Catch Context Loss Before Production
Multi-agent handoff testing treats the brief as a contract: settled facts, open gaps, no transcript dump. Catch context loss before the parent overflows.
How to Test AI Agent Loops Without Overfitting
AI agent loop detection that bans tool names overfits. Cap failures, score new evidence, and unit-test the gate without a live model.
Deterministic Checks vs LLM-as-a-Judge for Agent Evals
Use deterministic graders for facts a model must not invent. Use LLM-as-a-judge for tone and usefulness. Never plant tool scripts in the prompt.
The AI Agent Testing Pyramid: What Belongs in CI?
An AI agent testing pyramid: fast bound checks at the base, fact contracts in the middle, few live trials on top. Keep long LLM runs out of every push.
AI Agent Evaluation: Why the Grader Must Be Separate
AI agent evaluation fails when the same model writes and grades. Separate the grader, build for the trial OS, treat the prompt as the interface.
Same Incident, Five Models: Rank Investigators Without Mixing the Exam
Random CI reruns mix model skill with incident difficulty. Freeze one checkout outage, swap only the bound model, and rank on Correct first, then wall time and new input tokens.
Session Efficiency Should Not Beat Accuracy
We labeled a score 0-100 and watched a perfect, cheap, fast agent run print 125. Cap budget bonuses at 1.0 so efficiency never grades above how often you were Correct.