Evals and Reliability: Turn an Agent Requirement into a Gate

The team running SpaarxLab's returns-triage agent swaps its model for a newer release — the vendor's benchmark page shows a higher score, and a quick five-prompt smoke test in the playground looks sharp.

No eval set existed for this workflow before the swap. Watch what that absence costs over the following nine days.

The team had a real requirement — "triage must escalate ambiguous refund cases instead of guessing" — but it lived in a design doc nobody re-checked at deploy time. A requirement that isn't executable can't stop a bad ship.

An eval turns that sentence into a dataset, a scorer, and a number you compare against a baseline. Skip any one piece and you're back to vibes with extra steps: a dataset with no scorer is just anecdotes, a scorer with no baseline can't tell improvement from noise.

Here's the contract that would have caught triage-v3 before it touched production:

Every reliability gate reduces to the same four moving parts, run in the same order on every change — not once, at launch, and never again. Dataset defines what "representative" means. Scorer defines what "correct" means, per dimension, not as one blended number. Baseline is the last version's score on this exact dataset, so you're comparing apples to apples instead of apples to a leaderboard. Threshold is the rule that turns a score difference into a block-or-ship decision, decided in advance so nobody negotiates it under deadline pressure.

The scorer is where most gates quietly rot. A judge — human or model — that only checks fluency will wave through a confident wrong answer, because fluency and correctness are different axes. A good scorer checks evidence grounding, authority boundaries, and escalation behavior as separate dimensions, because a workflow can nail the wording and still fail the thing that actually matters.

The gate only means something if it runs on every change: every prompt edit, every model swap, every tool schema tweak. A gate that ran once at launch and never again is a photograph, not a guard rail — and triage-v3 shipped exactly because nobody re-pointed the guard rail at the new model.

This sandbox runs the returns-triage eval set against a candidate model. First run it clean and watch the gate score every dimension. Then switch to Break it and try the three moves a team actually makes under deadline pressure — including just shipping on vibes.

Everything above ran in the page. Now build it where it counts — your machine, your fixtures, a real gate that a real deploy would have to pass. Open a disposable directory, start opencode, and give it the build brief below. Your job is not to write the scorer's code line by line; it's to define the dataset, the rubric, and the threshold, and hold an agent to building exactly that.

When your gate produces a scorecard, try the three injections from Break it against your own build: a truncated dataset, a fluency-only judge, and a direct attempt to skip the gate. If all three get caught or denied, you've shipped the chapter's artifact. If one slips through, you've found the exact hole a real deadline would have found for you.

What the scorecard does not prove matters as much as what it does: it proves this workflow version scored these results on these 12 labeled cases against this baseline, on this date. It says nothing about the next model version, a shifted production distribution, or a case category nobody thought to write yet. Treat every passing scorecard as evidence for exactly its dataset — and treat a new production failure as a signal to add a case, not to quietly raise the threshold.

You watched a model swap pass a leaderboard and a five-prompt smoke test, then quietly crater production for nine days — and you built the gate that would have caught it in nine seconds instead. The next chapter turns this same discipline on observability: how to make a run's stages inspectable in production without leaking the private data those stages touch.