When to evaluate a model — and when training quietly makes it worse

01When evaluation earns its cost — and when it doesn't

Human evaluation is expensive per item and slow relative to an automated score. That's fine when the output of the review changes a decision. It's wasted when it doesn't. The distinction is less about the task's difficulty and more about what happens after the review is done.

Evaluate closely when:

The output is going to be used as training or preference data — anything a model will learn from deserves more scrutiny than anything a user will simply read once, because an error here doesn't cost one bad answer, it costs a pattern the model repeats. The task is high-stakes or hard to reverse (medical, legal, financial, safety-adjacent content, anything shipping to a large audience at once). Automated metrics disagree with spot checks you've already done, or disagree with each other. The domain is new enough that you don't yet know which failure modes exist, so you have no rubric to automate against. And whenever a model is asked to explain or justify its own output — the explanation is a second claim that needs its own check, not a courtesy that comes for free with a correct answer.

Skip or thin out evaluation when:

The task is stable, low-stakes, and an automated metric has already been validated against human judgment on that exact task — re-checking it by hand is theater, not quality control. The review queue is growing faster than anyone can act on the findings — a backlog of correctly identified problems that nobody has time to fix isn't evaluation, it's an archive. And a metric or a review layer is being kept purely out of habit, with no one able to say what decision it currently changes.

Rule of thumbAsk what decision the review result feeds into. If you can't name one, the review is a cost with no corresponding benefit — cut it or redirect the reviewer's time to something that does feed a decision.

02What training on more data actually buys you — and what it costs

"More training data" is treated as an unqualified good often enough that it's worth stating the trade plainly, because both sides are real.

The upside is genuine: broader coverage of inputs the model will actually see in production, better calibration on the edge cases that a small hand-curated set never included, more consistent behavior across similar prompts, and — specific to RLHF — the ability to encode judgment that's hard to write down as a rule. A well-built preference dataset can teach a model the difference between "technically answers the question" and "answers it the way a careful person would," which no amount of prompt engineering reliably produces on its own.

The downside shows up when the pipeline that produces the new training data isn't independent of the model being trained. Reward hacking is the well-known version: a model learns to satisfy the reward signal instead of the underlying intent behind it. A subtler version is homogenization — outputs converge toward whatever the majority of raters preferred, and the model quietly gets worse at the handful of styles or approaches that used to be a minority but were still correct. Both are real costs of training on more data, and neither shows up in an aggregate accuracy number, which is exactly why they tend to go unnoticed until someone looks at the distribution instead of the average.

03The loop that makes a model worse: training a model on itself

There's a specific, sharper failure inside "training on more data can backfire," and it deserves its own name because it's easy to walk into by accident. Researchers call it model collapse; on a review team you'll more often hear it described the way it actually feels to watch happen — a bullshit loop. A model generates outputs, those outputs (or summaries, labels, or captions derived from them) get folded back into the next round of training data with too little real, diverse, human-verified material to anchor against, and the model trains on an increasingly concentrated version of its own voice.

The mechanism is not mysterious. Each generation of a model tends to sand off the tails of a distribution — the rare phrasing, the unusual but correct answer, the honest "it depends." Train the next generation on that output, and the tails get thinner again. Repeat it a few times without fresh, real data breaking the cycle, and you get a model that is fluent, confident, and steadily less correct about anything that isn't the statistical center of what it's already said before. It doesn't announce itself as a crash — it looks like gradual drift, not a bug report.

A constructed illustration, in the shape I'd actually flag it in review: a support-summarization model is used to generate short summaries of past tickets, and — to save annotation cost — a later fine-tuning round uses those summaries as ground truth instead of the original tickets. The first generation's summaries were 90% good and 10% subtly over-confident (a hedge like "the customer mentioned intermittent issues" flattened into "the customer's device was defective"). Trained on that mix, the next generation learns the confident version as normal. Run the cycle twice more and "intermittent issue reported" has quietly become "hardware fault confirmed" as the model's default framing — fluent, consistent, and wrong in the same direction every time, because nothing in the loop ever pointed back at an actual ticket.

PatternTraining data generated fully or partly by the model's own prior outputs, recycled through more than one generation without a large, fixed anchor of real, independently sourced examples. Aggregate accuracy can hold steady for a while even as distributional diversity and rare-case correctness quietly erode — which is exactly why it needs a dedicated check rather than trusting the headline metric. Recommendationkeep a real, human-sourced anchor set that never shrinks as a share of training data, and evaluate for distributional drift (rare-case accuracy, output diversity) — not just average score — every time synthetic or model-derived data enters the mix.

What this means for how you set up the loop

The two questions from the top of this post turn out to be one question wearing two hats: is this piece of the pipeline still connected to reality, or is it starting to check itself?

1. Reserve human evaluation for where it changes a decision. Especially before anything becomes training data — that's the one place an unnoticed error doesn't cost one wrong answer, it costs a pattern the model repeats at scale.
2. Treat "more data" as a question, not a default. Ask where the new data came from and how independent it is from the model that will train on it, before asking how much of it there is.
3. Keep a real anchor set and don't let it shrink. A fixed, human-sourced, independently verified core of examples is what a bullshit loop can't corrupt — as long as it stays a meaningful share of what the model actually trains on.
4. Watch the tails, not just the average. Rare-case accuracy and output diversity are the first things a feedback loop erodes, and the last things an aggregate metric will tell you about.

None of this argues against evaluation or against training on more data — both are how models get better. It's an argument for noticing which loop you're actually in: one that keeps touching real, independent ground truth, or one that's started grading its own homework.

— S.M.S.