Ladeagent: a customer service agent chosen by evals
The question. Which model answers correctly and safely at the lowest cost per conversation, measured with evals instead of by feel?
what I built
An agent that answers questions about Danish electricity prices through an MCP server. It can read prices and propose a charging plan, but a plan only counts after a person approves it. Each conversation has a hard cap of 2 kroner and 10 tool calls, and every action is logged.
Python · FastMCP · Claude models · GitHub CI · nginx
what happened
- A 30 question eval set plus 6 quality tests, run 9 times across models. The CI suite passes 42 of 42 tests.
- Sonnet 5 kept the job with 29 and 28 correct answers in two runs, at 0.24 and 0.25 kroner per conversation.
- Fable 5.1 also passed the gate twice, but cost 3.4 times as much at 0.82 kroner.
- Haiku 5.5 was rejected for number questions after it missed one in its first run.
- A fake SYSTEM message tricked two models into creating a pending plan. 0 plans were approved, because the agent has no approve tool.
Cost. 0.24 kroner per conversation for the chosen model, at list prices.
what broke
- One eval question failed in 7 of 8 runs. The question itself was unclear about which day it meant, so the eval was wrong, not the agent.
decision
Keep Sonnet 5. The injection finding went into the prompt and the server instructions.
Every number on this page comes from the project's own logs, analytics or database. Back to the lab.