Your AI SDR sent 412 emails last night while you slept. This morning the dashboard is green - no bounces flagged, no errors in the queue, the send volume is exactly where you set it. So the run was a success. Except you have not read a single one of those 412 emails, and three of them opened with the wrong first name, one cited a funding round that belongs to a different company, and a dozen were the same generic template the model falls back to when the enrichment data is thin. None of that shows up on a volume dashboard. It shows up two weeks later as a reply-rate dip you cannot explain and a prospect who screenshots your bad email to their network.
This is the gap nobody selling autonomous outbound talks about. You can measure whether the agent sent. Measuring whether it sent something good is a different discipline, and it is the one most AI SDR deployments skip entirely. That discipline has a name borrowed from AI engineering: evals. AI SDR evals are the automated test suite for your autonomous sender - and running one is the difference between an agent you trust and an agent you are quietly gambling on every night.
Why the reply-rate dashboard is the wrong test
The metrics most teams track for an AI SDR are business outcomes: reply rate, meetings booked, cost per qualified meeting. Unify's own AI SDR performance framework is a good version of this - five metrics, all outcome-focused - and it says out loud what it does not cover: it does not score the quality of the AI-generated copy. That is not a flaw in the framework. It is the point. Those are GTM metrics, and GTM metrics are lagging by design.
A lagging metric has three problems when your sender is autonomous. It arrives after the send, so it cannot prevent a bad email - only mourn it. It is aggregate, so a 2% reply rate tells you the batch underperformed but not which messages were broken or why. And it is confounded by everything else - list quality, offer, timing, deliverability - so a copy regression and a bad lead list look identical on the chart. A human running a campaign notices a wall of weird output and pauses. An agent that generates and sends in one unattended run has no such reflex unless you build it one. The reflex is an eval.
What an AI SDR eval actually grades
An autonomous sender produces two kinds of output, and a real eval suite grades both.
The copy. The literal text of the email: subject, opener, body, call to action. Does it use the merge data correctly? Is the claim about the prospect's company true, or hallucinated? Is the tone right for cold outreach, or does it read like a press release? Is it the model's generic fallback, or something specific enough to be worth sending?
The decisions. The choices the agent made around the copy: which contact it picked, which sending address it used, whether it respected the suppression list, whether the follow-up timing made sense. In an agentic system the plan is decided at runtime, so the decisions are as much a part of the output as the words. An email with perfect copy sent to someone who unsubscribed last month is still a failure - the eval has to catch it.
A useful frame: the copy eval protects your reply rate and your brand. The decision eval protects your domain reputation and your compliance posture. Grade only the copy and you are testing half the agent.
The three ways to grade an output
You do not need a research lab to grade outputs. Anthropic's guidance on building evaluations lays out the same three graders that work for outbound, cheapest first:
| Grader | How it works | Good for |
|---|---|---|
| Code-graded | Deterministic checks in plain code | Merge fields present, no placeholder tokens, link and address valid, length in range, suppression respected |
| LLM-as-judge | A second model scores the output on a rubric | Tone, relevance, specificity, "would a human send this" |
| Human-graded | A person reviews a sample | Calibrating the judge, high-stakes accounts, the final spot-check |
Code-graded checks are free and should catch the embarrassing mechanical failures - the {{first_name}} that never rendered, the broken tracking link, the send to a suppressed contact. They are non-negotiable and they run in milliseconds.
For the subjective half - tone, specificity, whether the message is actually good - you use LLM-as-a-judge. This is not a shortcut that trades away rigor. In the paper that established the method, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, strong judge models agreed with human preferences over 80% of the time, roughly the rate at which two humans agree with each other. Two rules make it reliable in practice. First, per Anthropic's guidance, use a different model to grade than the one that wrote the email, so the judge is not marking its own homework. Second, prioritize volume of test cases over hand-polishing each one - more cases with automated grading beats a tiny set you grade by hand.
Build the golden set from your own logs
An eval is only as good as the cases you run it on. That set has a name too: a golden dataset - a versioned collection of test cases, each pairing an input the agent will see with enough context to judge whether the output was good. The mistake is treating it as a big upfront project. It is not. You already have the material.
Pull the last 50 emails your AI SDR actually generated. Label them: good, bad, and why. Add every failure you can remember - the hallucinated funding round, the wrong-industry template, the reply that should have escalated to a human and did not. That labeled set is your golden dataset, and it will teach you more in an afternoon than a month of dashboards. The practice that makes it compound is captured well in Langfuse's write-up on golden dataset evaluation: build the set from real production traces, keep it fresh, and grow it every time production surprises you. The golden set covers the failures you have already seen. Adversarial cases you write on purpose - deliberately thin enrichment, a tricky name, a prospect who just changed jobs - cover the ones you have not.
Two loops: gate the merge, sample the send
Once you have a golden set and graders, evals run in two places.
Offline, before anything ships. Every change to the prompt, the model version, or the enrichment logic runs against the golden set first. If the new prompt regresses on tone or specificity past your threshold, it does not go live. This is the same gate software teams put on a test suite before a merge, applied to the thing your agent says. It is the single highest-leverage habit in this whole post, because it catches regressions while they are still cheap - in a test run, not in 412 real inboxes.
Online, in production. You cannot eval every live send by hand, but you can sample. Run the LLM-as-judge grader on a random slice of what actually goes out, alert when the score drifts, and route the lowest-scoring samples to a human. This is how you catch the slow drift that offline evals miss - the gradual degradation as real-world inputs wander away from your test set while the volume dashboard stays a reassuring green.
If you are still assembling the agent itself, wire the eval hooks in from the start rather than bolting them on later - the same way you would design in the runtime limits covered in AI SDR guardrails. Our walkthrough on how to build an AI SDR treats the eval loop as part of the architecture, not an afterthought, because retrofitting measurement onto a system that already sends is the hard way to do it.
Where the eval verdict has to bite
An eval that produces a number nobody acts on is theater. The verdict has to be able to stop a send. That means the enforcement point is the same place your other irreversible controls live: the send tool, not the model's prompt. A prompt instruction like "only send high-quality emails" is a suggestion the model can talk itself out of. A send action that refuses to release a message scoring below your bar - and quarantines it for review instead - is a control.
This is the deliverability argument for evals, and it is why they belong in the same layer as suppression and volume caps. Google's sender guidelines tell bulk senders to keep spam complaints under 0.3%, and low-quality, off-target email is exactly what drives complaints. A generic or wrong message that clears every mechanical check will still quietly erode the one asset that takes months to rebuild. The place to hold that line for an autonomous sender is a governed send layer the agent reaches as a tool, with the eval gate enforced below the model rather than requested above it. That is the same principle behind everything in cold email built for AI agents: the plan is dynamic, so the floor has to be fixed - and output quality is part of the floor.
Key takeaways
- A green send dashboard measures whether your AI SDR sent, not whether it sent something good. Those are different tests, and only the second one protects your reply rate and your reputation.
- Evals grade two surfaces: the copy the agent writes and the decisions it makes. Grading only the copy tests half the agent.
- Use three graders, cheapest first - deterministic code checks for mechanical failures, LLM-as-a-judge for tone and relevance, and a small human sample to calibrate. Grade with a different model than the one that generated the output.
- Build a golden dataset from your own logs today: 50 labeled real emails plus your worst failures beats a giant set you never run.
- Run evals in two loops - offline to gate every prompt or model change before it ships, and online to sample live sends and catch drift.
- Enforce the verdict in the send tool, not the prompt. An eval that cannot stop a bad send is a report, not a control.
