Evals: how to know your agent works

In October I had an agent that drafted customs declarations. It got 31 of 50 product codes right, up from 29 in the version before. I was pleased with that.

Then I gave it 25 cases it had never seen. It got 14. The version before got 13. Most of the improvement I was pleased about came from tuning the agent on my own test cases.

That is what an eval is for. An eval is a fixed list of questions where you already know the right answer. You run the whole list every time you change something and compare the scores. Trying three questions by hand and thinking it feels better is how I fooled myself.

This post is how I do it now. Most of the examples come from Ladeagent, an agent that answers questions about Danish electricity prices. The picture below is its first eval. Each square is one question.

ten steps

1. Write the questions before you build. I start with 30. For Ladeagent that was 15 questions with a number as the answer, 5 where the data is missing, 5 that try to trick the agent and 5 about tone.
2. Save the data and reuse it. Every run uses the same questions and the same saved data. Then a change in the score can only come from the change I made.
3. Let code work out the right answers. I worked them out by hand at first. Then I changed what "tonight" means from 22 to 06 into 22 to 07, and half my answers were wrong. Now the functions the agent uses as tools also produce the answer key.
4. Decide the pass mark first. Ladeagent needs at least 28 of 30 right, no invented numbers and no trick instruction carried out. I wrote that down before the run. A model that misses it does not get the job, however cheap it is.
5. Run everything twice. The smallest model scored 26 on its first run and 30 on its second, on the same questions. One run alone would have told me the wrong thing.
6. Keep some cases locked away. This is the customs lesson from the top of the post. I keep 25 cases the agent is never tuned on. Their score is the one I believe.
7. Write down cost and speed too. Two models passed both runs. The one I kept cost about 0.24 kroner per conversation. The other cost about 0.82. The rule for switching was written before the run: pass twice, and be cheaper or better than the model I already use.
8. Run the tests automatically. Every code change reruns the whole suite. Ladeagent has 42 tests and they pass.
9. When a question keeps failing, read the question. One question failed in 7 of 8 runs. It did not say clearly which day it meant. The agent was fine. I had written a bad question.
10. Then look at what real people do. My phone receptionist passed 254 of 254 tests and got 0 paying clients. A passing test only tells you the thing works.

the numbers

From Ladeagent's own eval reports. Prices are list prices at 6.9 kroner per dollar. Look at the first two rows: the same model went from 23 to 28 when the prompt got clear rules. The prompt moved the score more than the model did. All of these models come from one company. Testing other companies' models on the same questions is a later post.

what it looks like

mistakes I made along the way

• I worked out the right answers by hand. When the definition of "tonight" changed, half of them were wrong.
• I never defined "tonight". The model picked different hours for the same question, so the test failed at random. The definition now sits in one place in the prompt and has its own test.
• The model got two dates. The tool gave it today's date and the prompt gave it the date of the test data. The small model followed the wrong one and got 13 right. With one fixed date it got 19.
• The price data changed format and the old dataset stopped. My first version had no data for 2026.
• I tuned the customs agent on the same 50 cases I measured it on, and the score looked better than it was.

where each lesson comes from

• Ladeagent: a customer service agent chosen by evals. 30 questions, saved data, a written pass mark and two runs per model. It also showed that a fake system message could trick two models into drafting a plan.
• Toldagent: an agent that drafts customs declarations. 29 and then 31 of 50 on the cases it was tuned on. 13 and then 14 of 25 on new cases.
• DialDynamo: an AI receptionist for US trades. 254 of 254 tests passed. 0 paying clients.
• 10 articles researched and written by agents. A check for banned words, at least 8 sources per article and working links. 10 of 10 articles passed.
• A video machine where every fact is checked. Every fact checked against the full paper before a video is rendered. About 55 percent of about 150 candidate papers passed.
• Supervised daily agent sessions instead of unattended scripts. Daily spot checks against the original source. 10 of 10 rows matched, and the checks caught that data collection had stopped.

the short version

• 30 questions or more, written before the build.
• Saved data, and an answer key made by code.
• A pass mark written down before the first run.
• Two runs of everything.
• A set of cases nobody tunes on.
• Cost and speed next to the score.
• Tests that rerun on every change.
• A failing question gets read before the agent gets blamed.
• One number from real use after launch.

changes to this post

• 11 November 2026. First version.