AI Evals Playbook: Test Cases & Metrics for Production AI

This guide is for founders, developers, and consultants who can already get an AI application working and now have to prove it works well enough for production. Maybe the demo was great and a client or testing team is now asking how you know it will hold up across 1,000 real cases. Maybe a model upgrade broke something. Maybe a cut-off response took down a downstream process and you couldn't tell why. AI applications are non-deterministic, so traditional testing doesn't fit them well. Most AI projects that get labeled failures didn't fail technically. Nobody agreed upfront on what success looked like or how to measure it. Inside, you'll get a practical process for eval engineering. You'll learn how to write a success definition your client will sign off on. You'll build a golden dataset before you write your first prompt. You'll design test cases for odd behavior such as language switching, off-script callers, and ten-minute tangents. You'll set up tracing so every change can be compared against what came before. You'll also catch silent production killers like a max_tokens limit that truncates JSON. Finally, you'll plan an upgrade path and an escalation path for the outliers that will always show up. I'm Edward Honour, founder of Maludbms and CEO of Kinetic Seas, and author of Oracle How-To. I'm running into these exact problems on real enterprise AI projects right now. The examples here come from that work, including a production bug I shipped myself, where about 20% of LLM calls failed because I let Claude Code set the max_tokens value. You'll get the lessons without repeating the mistakes.