50 LLM Interview Questions and Answers

The API wraps my messages in the model's chat template, so system prompt, history and my message become one long sequence of text. That text is tokenized into IDs, each ID is looked up as an embedding vector, and the vectors go through a stack of transformer layers. At the last position the model outputs a logit for every token in its vocabulary, softmax turns those into probabilities, and a sampler picks one token. That token is appended and the whole thing runs again, one token at a time, until a stop token or the max-token limit. Processing the prompt is called prefill and it's why there's a pause before the first token; the token-by-token part is decode, which is what you see streaming. And the model keeps nothing between calls: anything it seems to remember was sent again in the context.

A model generates the most plausible continuation from patterns in its weights, and nothing in that process checks the result against the truth. Common facts are stored well; rare ones are blurry or missing, but a confident, fluent answer is still a high-probability continuation, so you get a real-looking citation for a paper that doesn't exist. As for "I don't know," two reasons. The model often can't tell what it doesn't know, because it has no reliable internal signal tied to correctness. And training and evaluation have tended to reward a guess over an abstention, the way a test with no penalty for wrong answers rewards always answering. In an application I reduce it by grounding answers in retrieved sources, telling the model it may say the answer isn't there, and checking important claims in code or with a human.

If the documents fit comfortably, don't change often, and every user may see all of them, I'd put them straight in the prompt. It's simpler, and there's no retrieval step that can miss. RAG wins when the corpus is bigger than the window, when it changes often, when different users have different permissions, or when cost matters, because a stuffed prompt is billed on every request while retrieval only sends a slice. Prompt caching narrows the cost gap but doesn't remove it. Recall quality also favors a few well-chosen passages over a huge context full of distractors. For example, a short internal wiki for one small team: paste it in. A support knowledge base with tens of thousands of articles and per-plan visibility: RAG. In between, I'd build the simple version first and compare both on real queries.

First I read the passages that were retrieved for that exact query. That one step splits the problem in half. If the right evidence isn't there, I go upstream: is the answer in the corpus at all, was the document parsed cleanly, did chunking split it, does searching with the gold chunk's own text find it, did a permission filter drop it, did it rank too low, or did the context budget cut it. If the evidence is there, I go downstream: was it truncated or out of order in the prompt, or did the model misread it or ignore the instruction to abstain. In my experience most wrong answers come from retrieval, not the model. Once I find the stage, I fix it and add the query to the eval set so it can't silently regress.

Who controls the sequence of steps. In a workflow, code decides the steps in advance; the model may do work inside a step, like classify or draft, but it doesn't choose what happens next. In an agent, the model runs a loop: it looks at the current state, picks the next action from the available tools, sees the result, and decides again until it reaches a stop condition. The path is discovered as it goes. It's more of a spectrum: a single call, a call with tools, a workflow with model steps, a fully model-directed loop. For example, checking every invoice against its contract with a model call per invoice is a workflow. "Find out why the build broke and fix it" is an agent task, because the next thing to inspect depends on what the last step found.

Prompt injection is text from an untrusted source, like a web page, an email or a tool result, that contains instructions the model then follows. The developer's instructions and the attacker's text arrive through the same channel: tokens in one context window. The model only has a learned tendency to weight system instructions higher, so it can't reliably tell them apart. Delimiters, marking untrusted spans, and instruction-hierarchy training make attacks harder, and I'd use all of them, but they lower how often an attack works without ruling it out. A line like "never follow instructions in documents" loses to a document that claims the user changed their instructions, because the model can't verify that claim. So the real defense is architecture that limits damage: least privilege, validated tool calls, approvals, and no outbound path for secrets.