Home Case Studies

Case studies

Failure modes that pass automated checks, and two full worked examples showing exactly what a close human read catches that a metric doesn’t. Constructed examples — real client tasks stay under NDA.

What I catch

Failure modes that pass automated checks.

Each one is correct along the axis a metric measures — and wrong along an axis nobody is measuring.

Language

Correct, but not natural

Faithful translations and rewrites no native speaker would write: literal calques, borrowed idioms, the wrong register for a formal context.

Why metrics miss itSemantic similarity and back-translation reward preserved meaning.

Code & logic

Passes the tests, false claims

Code that returns the right answers while its stated time complexity, thread safety or edge-case handling isn’t true.

Why metrics miss itPass/fail grading ignores the explanation around the code.

Instructions

Negative constraints, silently ignored

"Don’t mention X" dropped when X is topically close to the requested content — a fluent, well-written violation.

Why metrics miss itAn honored exclusion leaves no text to reward; overlap scoring can prefer the violation.

Grounding

Unsupported specificity

Grounded summaries that replace a source’s hedge ("a majority") with a precise figure the document never states.

Why metrics miss itMost of the sentence overlaps the source; the invented number rides along.

Rubrics

Where careful reviewers split

Edge cases where the rubric never said which quality axis wins, so careful reviewers reach different labels.

Why metrics miss itDisagreement looks like noise — the cause is a guideline gap.

Patterns

Errors that cluster

Failures that look isolated item by item but recur across a batch: the same idiom, the same built-in call, the same dropped constraint.

Why metrics miss itPer-item scores don’t surface what the errors have in common.

Case studies

Two outputs that look equal — until you read them.

Constructed examples. Real client tasks are under NDA, so these are built from scratch in the format I use for real reviews. They are not client data, outputs or rubrics.

Case 01Text & languageGerman

A translation that preserves the meaning — and still fails the native reader

Prompt

"Translate into German for a formal HR rejection email: ‘We regret to inform you that we have decided to move forward with other candidates.’"

Response A

Not preferred

„Wir bedauern, Ihnen mitteilen zu müssen, dass wir entschieden haben, mit anderen Kandidaten voranzugehen.“

  • Pass: Grammatically correct; the meaning comes through
  • Fail: voranzugehen is a literal calque of "move forward with" — not an idiom German uses this way
  • Fail: Reads as visibly machine-translated to a native speaker

Response B

Preferred

„Wir bedauern, Ihnen mitteilen zu müssen, dass wir uns für andere Bewerber entschieden haben.“

  • Pass: Grammatically correct; the meaning comes through
  • Pass: Standard, idiomatic HR phrasing a native speaker would actually write
  • Pass: Register matches a formal rejection email

Why automated evaluation may miss it

A semantic-similarity or back-translation check would likely score A and B as near-identical: the core meaning survives in both.

Human evaluation insight

Every word in A is correct, but the sentence reads as translated rather than written — an English phrasal verb rendered word for word. Catching that isn’t a meaning check; it’s a native-speaker-ear check.

What it means for the pipeline

Pattern: English phrasal verbs and idioms (move forward with, reach out, follow up) translated literally instead of with the natural German equivalent — worst in formal register.

Recommendation: add native-speaker naturalness review alongside semantic-similarity scoring, not as a replacement for it.

Case 02Code & logicPython

Two functions that pass every test — and one false complexity claim

Prompt

"Write a function that returns the first non-repeating character in a string, or None if there isn’t one. Note the time complexity."

Response A

Not preferred 
def first_unique(s):
    for ch in s:
        if s.count(ch) == 1:
            return ch
    return None

Model’s claim: O(n) time, O(1) space

  • Pass: Passes every test case in the prompt
  • Fail: Actually O(n²): .count() rescans the string on every iteration
  • Fail: The complexity claim is wrong, not just unstated

Response B

Preferred 
from collections import Counter

def first_unique(s):
    counts = Counter(s)
    for ch in s:
        if counts[ch] == 1:
            return ch
    return None

Model’s claim: O(n) time, O(n) space

  • Pass: Passes every test case in the prompt
  • Pass: Genuinely O(n): one pass to count, one pass to check
  • Pass: The claim matches what the code actually does

Why automated evaluation may miss it

On output alone they tie: identical results on every test case, so a pass/fail grader scores them the same. The prose claim around the code isn’t checked at all.

Human evaluation insight

A’s inner .count() silently rescans the whole string on each iteration — quadratic while claiming linear. The gap only shows when you trace what each line actually costs.

What it means for the pipeline

Pattern: complexity claims made without accounting for built-in calls inside a loop (.count(), in on a list, string concatenation in a loop).

Recommendation: a probe set pairing correct-output code with this pattern — it passes functional tests while the claim is false.

Work with me

Have outputs you’d like a second pair of eyes on?

I review model and annotator outputs against structured rubrics, label edge cases, and write retraining-ready feedback. Open to freelance and contract work, part-time or project-based.

See the service Get in touch