Building an Offline Eval Harness Before You Change the Prompt
Building an evaluation harness catches silent regressions before they reach users.
Every team running an LLM feature in production hits the same wall eventually: someone wants to change the prompt, and nobody can say with confidence whether that change will help or hurt. This piece is about building an offline evaluation harness before that moment arrives, not scrambling to build one after. Done right, the harness turns "I think this new prompt is better" into a number you can put in front of a skeptical reviewer and defend.
Most teams skip this step. They ship the feature, watch it work in a demo, and move on to the next thing on the roadmap. Three months later someone tweaks the system prompt to fix a specific complaint, ships it same-day because it's "just wording," and support tickets spike for a reason that has nothing to do with the original complaint. Nobody catches it for two weeks, because nobody was measuring anything before the change went out, so there's no baseline to compare against. I sat in the retro for a version of this: a one-line tweak meant to fix a formatting complaint, something like "always return the date in ISO format," quietly cut accuracy on multi-part requests by double digits. Nobody had a test for multi-part requests. Nobody had a test at all. It happens because manual testing is expensive enough, in time and tedium, that people talk themselves into skipping it. "It looked fine in the playground" becomes the entire QA process.
Why "it looks good to me" doesn't scale
Prompt engineering has a seductive quality. You write a prompt, run it against a handful of examples, read the outputs, and they look right. That's fine at the prototype stage. It falls apart the moment the feature touches real traffic, because the space of inputs a production system sees is wider and stranger than the five examples a developer tried by hand at 4pm on a Friday.
Take a support ticket classifier built on an LLM. During development the prompt gets run against ten sample tickets, all clearly worded, all single-issue. In production it sees tickets that mix three complaints into one email, tickets that are sarcastic, tickets written in broken English by someone typing on a phone, tickets that reference a screenshot the model never sees. A prompt that scores well on ten clean examples has no guarantee of holding up against that mess, and the only way to find out is to test against a sample that actually looks like production, not like the version of production that lives in a developer's head.
This is the same lesson software engineering learned decades ago with unit tests and regression suites, just applied to a function whose output is probabilistic instead of deterministic. A traditional function either passes a test or it doesn't. An LLM call might phrase things slightly differently every run, might hallucinate a field that was never in the schema, might quietly degrade on a subset of inputs that share some property nobody thought to check. An evaluation harness is what makes that variability visible instead of anecdotal.
What the harness actually is
Cut through the tooling vocabulary and an offline evaluation harness is three things: a dataset, a scoring method, and a process for running both prompts against both and comparing the results.
The dataset comes first, and it has to look like production, not like a wish list. Pull real inputs from logs where they exist. If the feature is new and there's no log history yet, write synthetic inputs that approximate what you expect the traffic to look like, and go back and refresh that assumption once real data starts coming in, because the first guess is usually wrong in some detail. A dataset of 100 to 300 labeled examples tends to catch meaningful regressions for a narrow-scope feature, in my experience; a feature with more edge cases and more failure modes needs more than that, and the exact number is something each team has to earn through its own history of things going wrong. The labels matter more than the count. Without a known correct answer, or an acceptable range of answers, for each example, there's nothing to score against.
Scoring is where most of the design effort actually goes, and it splits into two camps. Exact-match or rule-based scoring works when the task has a clean right answer: extracting a date from an email, sorting a ticket into one of six categories. Write an assertion, run it against the output, get a pass or fail. Start here if the task allows it. It's cheap, it's fast, and a script can grade a thousand examples in the time it takes to get coffee.
The harder case is open-ended generation: summarization, drafting, anything without a single correct output. Here teams reach for one of two tools. LLM-as-judge uses a separate model call to score output against a rubric. Human review has someone actually read a sample of outputs and grade them by hand. Both have documented failure modes. LLM judges tend to favor longer responses regardless of quality, and there's research showing they rate outputs from their own model family more favorably than they should. Human review is accurate, but it doesn't scale past a few dozen examples per run before it turns into someone's entire week. Most mature teams run LLM-as-judge across the bulk of the dataset and spot-check with a human periodically, mostly to catch the judge drifting over time as the underlying model gets updated out from under them.
Running the comparison
Once the dataset and the scoring method exist, the test itself is almost anticlimactic. Run the old prompt against every example, log the scores. Run the new prompt against the same examples, log those. Compare the two sets.
The comparison needs to go past the aggregate average, though, because an average is exactly the kind of number that hides a regression instead of revealing it. Say the old prompt scores 84 out of 100 and the new one scores 86. Looks like a win, ship it. But if the new prompt picked up ground on easy, high-frequency cases while losing ground on something rare and expensive, a refund request, a legal threat, a churn risk buried in the wording, the aggregate is lying by omission. Slice the dataset by input type, by length, by whatever dimension actually matters to the product, and the regression usually surfaces immediately. This is the same reasoning behind stratified sampling in statistics: an average taken across a lumpy, uneven population tells you less than it appears to.
Latency and cost belong in the same report, and they get forgotten until the invoice shows up. A new prompt that tacks on a paragraph of instructions, or that needs a longer chain of reasoning to hit the same accuracy, can cost real money per call once it's running at scale. That tradeoff should sit next to the accuracy numbers in the same table, not get discovered three weeks later by whoever owns the API budget.
The counterargument, and why it doesn't hold
The objection is predictable, and honestly, reasonable on its face: building a harness takes time, and most prompt tweaks are small enough that formal testing feels like using a sledgehammer on a thumbtack. There's some truth in it. Not every one-word wording fix needs a full evaluation run, and a team that treats every comma change as a research project will ship nothing, ever.
But the cost runs the other direction more often than people admit. Building a harness once, even a modest one covering 150 examples with rule-based scoring, costs a few days of engineering time. Shipping a silent regression, discovering it through a spike in angry tickets, and rolling it back after the fact costs more than that in engineering hours alone. It also costs something harder to get back: user trust, and internally, the credibility of whoever approved the change. Skipping evaluation doesn't remove the cost. It just defers it and adds interest.
Where this fits in the broader shift
None of this is exotic. It's the same discipline that turned software engineering from artisanal guesswork into something closer to a repeatable trade, now applied to a technology that behaves less predictably than the deterministic code that came before it. Model behavior drifts, prompts interact with each other in ways nobody predicts on paper, and without infrastructure to measure that, a team is making ship decisions on gut feel dressed up as confidence.
The teams shipping reliable LLM features built the unglamorous infrastructure first: a dataset that actually looks like their traffic, a scoring method that's honest about where it can be wrong, and the discipline to run the comparison before the prompt goes out, not after the complaints start coming in.
