LLM Evaluation Interview Questions and Answers
Evals are where AI engineer interviews separate people who have shipped from people who have prototyped. These questions cover building eval sets, grading with code, humans and LLMs, gating releases on nondeterministic output, measuring retrieval and hallucinations, and closing the loop with production traces. All 14 are listed here, with model answers to five of them; the full set of answers comes as a free PDF.
14 questions · 5 sample answers · Updated
Get all 14 answers as a free PDFOn this page · 4 sections
Designing and grading an eval
1. What do you decide before you collect any eval data?
Sample answer
What failure costs, and to whom. I work from the user's goal to an observable success criterion, then list failure modes ranked by cost, and only then pick metrics that would detect those failures. I also write the decision rule up front: what score means ship, investigate, or block.
Skipping this leads to proxy metrics, things that are easy to score automatically, like similarity to a reference answer or length. Proxies track quality just often enough to fool you. For a summarizer, word overlap with reference summaries rewards copying the source, and it says nothing about the costly failure, an invented number inside a fluent summary. That needs its own check: does every number and claim trace back to the source?
An hour writing this down saves months of improving a number nobody downstream cares about.
- What they’re checking:
- Whether you derive metrics from the cost of failure instead of from what's easy to compute.
- Likely follow-up:
- Give me the failure modes, ranked, for a feature you've worked on.
- Avoid:
- Starting with a generic metric like BLEU or "helpfulness score" without saying what failure it detects.
2. Your eval set has 2,000 cases, but regressions still reach production. What's wrong with it?
- What they’re checking:
- Whether you know slice coverage, labeling, and the incident pipeline matter more than raw case count.
- Likely follow-up:
- How do you get eval cases for a feature that hasn't launched yet?
3. How do you write a rubric that two graders will agree on?
- What they’re checking:
- Whether you know rubric quality is measured by grader agreement, and why disqualifiers exist.
- Likely follow-up:
- The graders keep disagreeing on partially resolved cases. What do you change?
4. When do you score with code, and when do you need a grader?
- What they’re checking:
- Whether you prefer deterministic scoring and can name where it stops being enough.
- Likely follow-up:
- How would you restructure an open-ended task so more of it can be scored with code?
5. How do you build an LLM-as-judge you can trust, and how do you keep checking it?
Sample answer
I constrain how it grades, then keep measuring it against humans. It gets the rubric with example answers at each level. It cites evidence from the output before giving a score, because scoring first invites it to justify whatever number it picked. Its verdict comes back as structured output, and the judge model version is pinned.
I design against known biases: position bias in A/B comparisons, so I randomize order; a preference for longer answers; a tendency to favor outputs in its own family's style; and standards that shift when the judge model changes.
The check is a human-graded sample that stays in place permanently. I regrade it with the judge and measure agreement. If agreement drops, say after a judge model upgrade, I stop trusting new bulk results until the judge prompt or rubric is recalibrated against that sample.
- What they’re checking:
- Whether you treat the grader as a model that needs its own calibration and audit loop.
- Likely follow-up:
- Your judge agrees with humans on most cases but disagrees on one category. What do you do?
- Avoid:
- Using an LLM judge that has never been compared with human grades.
Regression testing and release gates
6. A test that checks summary quality fails about 5% of the time. What do you do with it?
- What they’re checking:
- Whether you can split deterministic tests from graded evals in a nondeterministic system.
- Likely follow-up:
- How do you test that your code handles a truncated model response?
7. The new prompt scores 3 points lower on your eval. Is that a real regression?
Sample answer
Maybe. I'd check three things before deciding. First, whether everything else was held fixed: same dataset, same graders, same model and settings. Comparing against historical numbers from a different harness measures the harness.
Second, case-level flips. A 3-point drop could be a few cases that went from pass to fail, mixed with others that went the other way. The flipped cases are what I read.
Third, slices, with counts and intervals. A slice that dropped to 7 out of 10 has a 95% confidence interval roughly from 40% to 89%, so it can't tell "worse" from "unlucky." I report "11 of 20" next to any percentage. If the interval is wide, I add cases to that slice. If it's narrow and below the bar, it's a real problem to fix before shipping.
- What they’re checking:
- Whether you reason about sample size, flips, and slices instead of trusting an aggregate delta.
- Likely follow-up:
- How many cases would a slice need before you'd trust a 5-point difference?
8. How would you write a release gate for an LLM feature?
- What they’re checking:
- Whether you make ship decisions with pre-agreed rules rather than post-hoc judgment.
- Likely follow-up:
- The gate fails by one point and the launch is tomorrow. What do you say?
9. Your eval score keeps climbing, but production quality doesn't move. What's going on?
- What they’re checking:
- Whether you know about held-out sets, contamination, and the visible-versus-held-out gap.
- Likely follow-up:
- Who should have access to the held-out set, and how often should it run?
10. How do offline and online evals work together?
- What they’re checking:
- Whether you see offline and online evaluation as one feedback loop.
- Likely follow-up:
- Which online metric would you watch for a support bot, and why not thumbs-up rate alone?
Retrieval and hallucinations
11. Which retrieval metrics do you report for a RAG system, and at what k?
- What they’re checking:
- Whether you can pick retrieval metrics for the product's need and measure at the right stage.
- Likely follow-up:
- Overall Recall@5 looks fine, but small tenants complain. What would you check?
12. How do you measure hallucinations in a RAG feature?
Sample answer
I split it into two measurements. First, could the answer be built from the retrieved context at all? A judge or a human labels each case's passages as fully, partially, or not sufficient. That's the honest baseline, because the model can't be grounded in evidence it didn't get.
Second, faithfulness: break the answer into claims and check each one against the supplied passages. A claim with no supporting passage counts as unsupported even if it happens to be true. Citations get checked for supporting the specific claim, since models miscite real sources too.
Then I cross the two. Unsupported claims on sufficient context point at generation. Confident answers on partial context mean the model should have abstained, so the instruction for "not in the material" needs work. Neither number comes from how confident the answer sounds.
- What they’re checking:
- Whether you separate retrieval sufficiency from answer faithfulness and measure at the claim level.
- Likely follow-up:
- How would you calibrate the LLM judge that labels sufficiency?
- Avoid:
- Measuring hallucination by asking the model whether it's confident, or by checking if two models agree.
Production observability
13. How do you connect user feedback to a specific quality regression?
- What they’re checking:
- Whether you can make quality regressions debuggable through version attribution on traces.
- Likely follow-up:
- What would you record on a model-call span so this query is possible?
14. Nothing was deployed, but quality is slowly getting worse. How do you detect and explain it?
Sample answer
That's drift, and it has four usual sources: inputs changing, the provider's model behavior shifting, data like the retrieval corpus aging, and outcome metrics sagging slowly. Each has a signal. For inputs, length, language mix, and topic clusters. For the provider, finish-state, refusal, and output-length rates moving with no deploy on our side. For data, retrieval score distributions and document staleness.
I set a baseline for each summary and alert on sustained deviation. A step change usually means a version change, so I check the version attributes on traces first. A slow slope points to inputs or data.
Threshold alerts miss slow drift, since no single window crosses the line. So I also compare this week against a pinned reference week. When I find a new input segment, it goes into the eval set.
- What they’re checking:
- Whether you know drift sources, their signals, and why thresholds alone miss them.
- Likely follow-up:
- Refusal rate tripled over six weeks with no deploy. Walk me through the investigation.
How to practice
Practice each answer out loud, then answer the follow-up cold. If you can, build a tiny eval harness (a dataset file, a scorer, and a comparison between two runs) so you can describe one from experience. The free 50 LLM interview questions cover the fundamentals behind these, the prompt engineering interview questions cover testing prompts, and the RAG interview questions go deeper on retrieval evals.
More interview questions
- LLM interview questions The core 50, from tokens to production.
- Prompt engineering interview questions Prompts, context engineering, structured outputs, injection.
- RAG interview questions Retrieval, chunking, evaluation, debugging and securing RAG.
- AI engineer interview questions Fundamentals, building, production and your projects.
- AI system design A framework for the design round, plus practice prompts.