AI Engineer Interview Questions and Answers
These are the questions that test whether you can build and run LLM features, grouped the way an AI engineer loop usually moves: fundamentals, building with model APIs, retrieval and agents, production, and your own experience. All 21 questions are listed, with model answers to five (each about a minute out loud); the full set of answers comes as a free PDF. For the core concept questions, start with the free 50 LLM interview questions.
21 questions · 5 sample answers · Updated
Get all 21 answers as a free PDFOn this page · 5 sections
LLM fundamentals
1. How do you choose a model for a new feature?
Sample answer
I treat it as a filtering problem with a test set, and I don't start from a leaderboard. First I write down the unit of work: input, expected output, tools, and what counts as success. Then I apply hard gates, things like context size, tool or schema support, data residency, and a latency ceiling. Any model that fails a gate is out, however good its benchmarks look.
The survivors run on a few hundred representative cases under identical prompts and settings, with exact version IDs recorded. I compare task success, p95 latency, and cost per successful task, and I check critical slices separately. A model that's great on average but weak on Spanish inputs might be disqualified. Usually I end up picking the cheapest model that clears every bar, and I rerun the same harness before any version upgrade.
- What they’re checking:
- Whether you pick models with your own evaluation and explicit constraints, or by reputation.
- Likely follow-up:
- Two models score within a point of each other on your set. How do you break the tie?
- Avoid:
- Naming a favorite model and defending it, with no mention of a task-specific eval set or cost per successful task.
2. Which parts of a task should the model do, and which should your code do?
- What they’re checking:
- Whether you understand that LLMs approximate rather than compute, and design around it with tools.
- Likely follow-up:
- How would you stop the model from doing the arithmetic itself when a calculator tool is available?
3. Your feature needs facts from last week. What are your options?
- What they’re checking:
- Whether you can separate the freshness problem (retrieval, tools) from the behavior problem (fine-tuning).
- Likely follow-up:
- The retrieved document and the model's own memory disagree. How do you make the model prefer the document?
4. How do you decide whether an LLM belongs in a workflow at all?
- What they’re checking:
- Whether you have a structured way to say no to an LLM, and what would change that answer.
- Likely follow-up:
- Pick a task from your current job and walk me through those five axes.
Building with models
5. What can go wrong with a single model API call, and how does your code handle it?
- What they’re checking:
- Whether you have written production integrations, and know finish reasons, retry classification, and the timeout side-effect trap.
- Likely follow-up:
- How would you test each of those failure paths without calling the real API?
6. Structured output returns valid JSON, but some values are wrong. What do you build?
Sample answer
A validation layer in code after generation, because schema enforcement only guarantees shape. Constrained decoding has no idea what's true, so a well-formed invoice number can still be invented.
I'd check, roughly cheapest first: ranges and plausibility, like dates that exist and amounts in a sane range. Then cross-field invariants, such as line items summing to the total. Then existence, meaning IDs resolve to real records. For extraction, I check that the extracted string appears in the source text. And I check authorization, since the model naming an account doesn't mean this user may touch it.
On failure, I retry once with the specific error fed back, like "line items sum to 1,240 but total says 1,420." After that it goes to a fallback, such as marking the record for manual review. All of that is plain functions I can unit test without a model.
- What they’re checking:
- Whether you know schema validity and correctness are separate layers, and where each check belongs.
- Likely follow-up:
- Your first-attempt failure rate is 30% but retries hide it in the success metric. How would you notice and what would you fix?
- Avoid:
- "We use structured outputs, so the data is reliable."
7. How does a tool that sends money differ from a tool that reads data?
- What they’re checking:
- Whether you design tool risk around reversibility, and know idempotency and exact-preview approvals.
- Likely follow-up:
- The approval UI shows "paying the approved invoice." What is wrong with that?
8. A nightly batch job and your chat feature share one API key. What can go wrong?
- What they’re checking:
- Whether you treat rate limits as a design input with pacing, isolation, and backpressure.
- Likely follow-up:
- One request with a huge context uses your whole tokens-per-minute budget. How do you handle that?
Retrieval and agents
9. How do you decide whether retrieval is good enough to ship?
- What they’re checking:
- Whether you set retrieval targets from consequences and downstream losses, rather than picking a number.
- Likely follow-up:
- The product owner asks whether 0.87 recall is good. How do you turn that into a decision they can make?
10. Your agent plans, executes, then verifies. Should every step see the same context?
- What they’re checking:
- Whether you design context and tool exposure per stage, for both attention and least privilege.
- Likely follow-up:
- Why would you hide the executor's reasoning from the verifier?
11. The median agent run is cheap, but the monthly bill is huge. What's going on?
Sample answer
The tail. An agent's step count is decided at runtime, and each step resends a growing transcript, so cost per run grows faster than the number of steps. Most runs finish in a handful of steps, while a few loop on a failing tool or keep re-planning and run for dozens. That small share can dominate spend, which is why the average misleads.
So I report cost per run as p50, p95, and p99, from traces, and look at the expensive runs individually. They're usually a correctness bug as well as a cost bug.
Then the controls: step, token, and time budgets with a "budget exhausted" end state, a per-run cost cap, retries counted inside the same budget, and prefix caching so the repeated transcript is cheaper. The cap turns runaway runs into visible outcomes someone triages.
- What they’re checking:
- Whether you understand agent cost as a long-tailed distribution and know the controls.
- Likely follow-up:
- Where would you set the per-run cost cap, and what happens to the user when it's hit?
12. Your model provider is degraded for 40 minutes. What does your agent system do?
- What they’re checking:
- Whether you plan provider failure modes, circuit breakers, and safe failover for agents.
- Likely follow-up:
- How would you decide which workflows are allowed to fail over?
Production
13. A benchmark says the new model is much faster. What do you ask before believing it?
- What they’re checking:
- Whether you can read latency claims critically and know what drives LLM latency.
- Likely follow-up:
- Time to first token is fine but total latency is bad. Where do you look?
14. Spend on an LLM feature doubled after a release. How do you find out why?
- What they’re checking:
- Whether you have cost attribution in your telemetry and think in cost per successful task.
- Likely follow-up:
- How would you design an alert that catches this the morning after the release?
15. How do you decide whether an AI feature is worth what it costs?
- What they’re checking:
- Whether you can reason about unit economics including human review, beyond token prices.
- Likely follow-up:
- Walk me through the expected cost of a workflow with a retry path and a human escalation path.
16. How do you upgrade to a new model version without surprises?
Sample answer
Model IDs live in config under named roles, like a fast model and a quality model, so a switch is a config change. I pin exact dated versions. A floating "latest" alias can change behavior overnight, and a slight shift in output format is enough to break a parser.
Before switching, the candidate runs through the same eval set, graders, and settings as the current model, against a gate agreed in advance. Newer doesn't mean better on our task. Then a staged rollout: a slice of traffic, compared live on success rate, cost, latency, and human interventions, with rollback as a config flip.
Every request logs the exact model that served it. When quality moves, "did we change something or did the provider?" should take one query to answer.
- What they’re checking:
- Whether you treat a model upgrade as a versioned behavior change with evals and rollback.
- Likely follow-up:
- The new version wins offline but loses on one user segment in production. What do you do?
- Avoid:
- Pointing production at a "latest" alias so you get upgrades automatically.
17. How do you know your guardrails work?
- What they’re checking:
- Whether you measure guardrails in both error directions and per slice.
- Likely follow-up:
- Your non-English slice misses three times as much as English. What do you do this quarter?
Your experience
18. Walk me through an AI feature you shipped.
Sample answer
A structure that works: the problem and who it was for, how you defined success, the design, how you measured it, what broke, and what you'd change. Keep it to about two minutes and let them dig in.
For example: "I built ticket triage for our support queue. Success meant correct routing and correctly flagging urgent tickets, since a missed urgent ticket was the costly failure. It's one structured-output call that returns a category, urgency, and a reason, then plain code routes it. I built a 300-ticket eval from real tickets, with incident cases added over time. The first version over-flagged urgency on angry but routine tickets, so I added boundary examples and a rubric. If I did it again, I'd instrument cost per ticket from day one."
Use your real numbers and your real mistakes. Interviewers probe exactly there.
- What they’re checking:
- Whether you can explain your own design decisions, how you measured success, and what you learned from failures.
- Likely follow-up:
- How did you know it was working after launch, and what would have told you it wasn't?
- Avoid:
- Describing the stack ("we used RAG with a vector DB") with no success metric, no eval, and no failure story.
19. Tell me about an AI failure in production and how you debugged it.
- What they’re checking:
- Whether you debug systematically from traces and fix root causes rather than prompt-tweaking.
- Likely follow-up:
- What did you not have in your traces that you wished you had?
20. How do you review AI-generated code?
- What they’re checking:
- Whether you anchor review on intent and know the specific failure patterns of generated code.
- Likely follow-up:
- How do you spot an invented API in a diff that compiles?
21. How do you use coding agents in your own work, and what do you keep for yourself?
- What they’re checking:
- Whether you use AI tools productively while staying accountable for what ships.
- Likely follow-up:
- The agent went down the wrong path for twenty minutes. What do you do?
What AI engineer interviews cover
Loops differ by company and level, but these round types come up often. Ask your recruiter which ones you’ll get.
| Round | What it usually tests |
|---|---|
| LLM concepts | Tokens, context windows, embeddings, sampling, hallucinations, and why models fail the way they do. The 50 LLM interview questions cover this. |
| Practical build or pairing | Calling a model API, structured outputs, tool calling, error handling, and testing code around a nondeterministic component. |
| AI system design | Designing a RAG assistant, an agent, or an extraction pipeline end to end: data flow, evals, latency, cost, security. See AI system design interview. |
| Coding | Ordinary software engineering. AI roles still expect clean, tested code. |
| Project deep dive | One AI feature you built: how you defined success, measured it, and handled failures. |
| Behavioral | Ownership, judgment, how you work with AI tools and with people who don’t. |
How to prepare in two weeks
- Days 1–2: fundamentals. Go through the 50 LLM interview questions and answer each one out loud, timing yourself at about a minute.
- Days 3–4: build with an API. Write a small script that returns a structured output and calls one tool. Handle finish reasons, retries, and validation.
- Days 5–6: retrieval. Index 50 documents, label 20 queries, and measure recall@5. Then practice the RAG interview questions.
- Days 7–8: agents and tools. Work through the agentic AI and MCP questions, and sketch how you’d limit an agent’s permissions.
- Days 9–10: prompts and evals. Write a 30-case eval for your script with one deterministic check and one rubric. Practice the prompt engineering and LLM evaluation questions.
- Days 11–12: system design. Do two full designs out loud from the AI system design interview guide, covering evals, cost, and failure modes in each.
- Day 13: your project story. Write the two-minute version of one feature you shipped, with real numbers and one real failure.
- Day 14: mock interview. Have someone ask you ten random questions from these pages. Then rest.
More interview questions
- LLM interview questions The core 50, from tokens to production.
- AI system design A framework for the design round, plus practice prompts.
- RAG interview questions Retrieval, chunking, evaluation, debugging and securing RAG.
- Agentic AI interview questions Agent loops, tools, state, memory and permissions.
- MCP interview questions Model Context Protocol: hosts, clients, servers and security.
- Prompt engineering interview questions Prompts, context engineering, structured outputs, injection.
- LLM evaluation interview questions Eval datasets, graders, LLM-as-judge and monitoring.