I interview AI engineers and FDEs. I start with “Walk me through an AI project you built.” Then come the code, the failures, and the questions beyond memorized answers.

You tell me that RAG retrieves outside knowledge and fine-tuning changes how a model behaves. That’s correct. Then I ask what happens when retrieval brings back the wrong document, how you’d find out, and how you’d fix it.

I don’t ask for memorized answers or memorized code. Use an AI assistant to write it if you like. I do. What I go deep on is whether you understand the intuition, the foundation, and what the code does, and that comes from running things and watching what happens.

I’ve been hiring and managing engineers for over two decades, and these days I run AI and Forward Deployed (FDE) engineering teams. I’ve written before about how I evaluate engineers, and about the unfamiliar repository I hand FDE candidates. This piece is the AI part.

Where I start: “Walk me through an AI project you built.”

Every interview I run for an AI engineer or a forward deployed engineer (FDE) starts here. I want to know what problem you were solving, how you designed the solution, why you chose that design, what went wrong, and how you knew it worked. You need to have built something yourself before you sit down with me.

Then I move to the technical questions. I dive deep into the implementations. I focus on the intuition and the foundation, to see whether you understood why you selected that path. If you decided to add RAG to your system, I want to know that you understand what RAG is. Why do you need RAG? What is in-context learning? Therefore, I ask basic questions to find out if you really understand what you are talking about. If you do not know what in-context learning is, you will never know why we use RAG.

The nine questions below are only part of what I ask. They follow one path: a single call to a model, context in the prompt, retrieval to supply that context, what breaks, how you know it works, and how you work with an AI coding assistant.

It’s also the path of my chapters in Data Strategy for LLMs, the book I wrote with Eslam Kamal, and the code below comes from it. The book’s first five chapters go gradually through the intuition and the concepts until we reach the advanced parts, because that understanding is very important for designing AI systems. Each chapter starts from a challenge, builds the intuition, and moves to the solution in code, which is the foundation you need to build projects of your own. An article can hold the questions and the first lines of code. For the deeper answers, I’ll name the chapter under each question.

Data Strategy for LLMs: Build reliable AI applications with RAG, fine-tuning, evaluation, and context engineering

1. “What happens when you call an LLM?”

I was interviewing a candidate recently and they were explaining generic items that they learned from the internet. It is all good. However, I needed to see if they really understood what they were talking about or if they had spent the time implementing it themselves. They were talking about agents and orchestration, so I asked them about agents and how to build one. After getting more generic answers, I decided to ask them this question: “What happens when you call an LLM?” Surprisingly, they did not know. This is why it is very important to start with this. When you build AI systems, you will use APIs, not a ChatGPT UI.

Here is what it looks like:

import openai
client = openai.OpenAI(api_key=OpenAI_API_Key)
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "Tell me a short joke."}],
)
print(response.choices[0].message.content)

That’s all of it: a client, one call, and the line where the text comes back. OpenAI_API_Key is your key, and MODEL is whichever model your account can call. I leave the name out on purpose, because model names retire.

You don’t have to write this for me, and I don’t mind if an assistant wrote it for you. I want to know that you understand it. Call it twice. Does the second call know about the first? It doesn’t. Each request stands alone, so a conversation exists only because your code sends the earlier messages again in messages. Some newer APIs can keep that history on the server for you. Each request to the model is still independent. So what else goes in messages? What's missing from this before it could run in production? Then wrap the call in a function and ask the model something it has no way of knowing:

def ask_ai(prompt):
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": prompt}],
)
return response.choices[0].message.content
print(ask_ai("How long do we retain prompt logs at Rany.ai?"))

Rany.ai is my own site, and the retention policy is made up for the test and left out of the question. On one run of mine the model refused to give a number and then invented a support address, support@rany.ai, which doesn’t exist. A rerun in November 2026 refused and invented nothing. Why does the same question produce both? What is the model doing when it doesn’t have the fact?

Chapter 1 answers those. It starts from an empty account, and its notebook has this call ready to run on a model it picks and tests for you.

2. “Now make it answer from your own data.”

context = """Rany.ai retains the prompt logs from client engagements for 400 days. The retention
window covers the twelve-month client audit cycle plus a five-week reconciliation period, so the
logs stay available until the audit for that period closes."""
prompt = f"""Context: {context}
Question: How long do we retain prompt logs at Rany.ai?
Please provide a comprehensive answer based on the context provided."""
print(ask_ai(prompt))

With the policy in the prompt, the model answers 400 days and explains the audit cycle. It’s the same model that couldn’t answer a minute ago. The only thing that changed is what it was given to read.

That’s in-context learning, and I spend time on it in an interview because everything after it is the same move with more machinery. This is the snippet to run with different options. Shorten the context. Contradict it. Bury it under paragraphs that have nothing to do with the question. Watch what the model does each time. The harder questions start from there. What happens when the policy is one paragraph in ten thousand documents? What can you put in that prompt besides documents? What do you do when two paragraphs disagree about the same fact?

The second half of Chapter 1 is this experiment. The rest of the book is what it grows into.

3. “How does RAG get the right context into the prompt?”

def ask_rag(query):
# 1. Vector-search the index for the 2 closest chunks to the user's query
results = collection.query(query_texts=[query], n_results=2)
context = "\n\n".join(results["documents"][0])
    # 2. Build the prompt: role, retrieved context wrapped in tags, and the question
prompt = f"""
You are an expert assistant.
Use the following retrieved context to answer the user's question.
If the answer is not in the context, state that you cannot find the information.
Do not use any other information.
    <context>
{context}
</context>
    <question>
{query}
</question>
    Answer:
"""
    # 3. Hand the prompt to ask_ai() and return the grounded answer
return ask_ai(prompt)

What I like about retrieval is how short the code is. Search the index for the closest chunks, put them in the prompt between tags, and call the model. I don’t need you to write it. I need you to tell me what each step is for, and what happens to the answer when you change one.

collection is the index, and that's where I go deep, because the short function hides the long part of the work. How did the documents get in there? Why two results and not twenty? What does the model do when the answer isn't in what you retrieved?

Chapter 4 builds this function one step at a time, and its notebook is where you run it against real documents and watch it refuse a question the index can’t answer.

4. “Retrieval is returning bad results. What do you look at first?”

The answer you can prepare is a list of upgrades: a better embedding model, a reranker, maybe a different vector store. My question comes before any of that. What did the chunks look like?

I ask because I got this one wrong myself. On one of my own projects I tried three different embedding models to fix bad retrieval before someone looked at the chunks themselves. The problem was 2,000-character chunks with no overlap, which cut paragraphs in the middle of a sentence. Once the boundaries were fixed, the embedding model I’d started with worked fine. I’d been shopping for a better model, and the fault was in the data the whole time.

So I listen for the person who opens the data before the model catalog. An answer like “I printed twenty chunks and read them” tells me more than the names of five rerankers. Then the questions get specific. How big should a chunk be, and how much should two chunks overlap? What do you lose at each end? What do you store next to a chunk so that an old draft can’t pass for the published version? When I went looking for the right chunk size, the teams who had actually run the experiments disagreed with each other.

Chapter 3 is where the index gets built: extraction, chunking, overlap, embeddings, and the metadata you’ll wish you had later. It also goes through what those teams found, and the range I keep coming back to.

5. “A search for ‘server returns 502’ brings back the runbook for 503. Why?”

The function above runs. Now I want to know what you do when the retrieval under it is wrong. This is a failure I’ve seen repeatedly in production: someone asks for an exact identifier and the system returns its neighbor, with high confidence. Section 12.3 of one document comes back as section 14.1 of another.

I built a small test for it, with six runbooks, one per HTTP error code, and one search for “server returns 502”. This is what came back:

runbook 503  distance 0.3546
runbook 502 distance 0.4113
runbook 429 distance 0.5689
runbook 500 distance 0.5697

The right runbook is second. That’s a ranking error, and it’s all this output shows. An engineer who reads only the top result would check CPU and auto-scaling, which is what the 503 runbook says to do, and find nothing. A RAG function that passes two results to the model, like the one in question 3, would hand it both runbooks.

So why did the wrong document win? Did the ranking error turn into a wrong answer, or did the model recover, and how would you find out? Which distance are those numbers in, and what do you get if you never set one? What are your options for fixing it, and what does each one cost you? How would you have caught it before the on-call engineer did?

Chapter 5 takes this failure and three more that I ran into once real data arrived: old documents outranking new ones, one tenant’s data leaking into another’s results, and questions whose answer spans two documents. In its notebook each one is a problem cell you run, followed by the fix.

Questions 1 to 5, in that order, are also my system design round. When I ask you to design an LLM application end to end, I’m listening for the same path, and it’s the first five chapters.

6. “Prompt A averages 4.20 and prompt B averages 3.40. Do you ship A?”

A=5  B=2  (A wins)  What is the refund policy for digital products?
A=1 B=5 (B wins) Can I return a physical item after 30 days?
A=5 B=3 (A wins) What happens if my subscription renews and I want
A=5 B=3 (A wins) I bought a gift card and the recipient lost it. Ca
A=5 B=4 (A wins) My order arrived damaged. Who pays for return ship
Average: A=4.20 B=3.40

Those rows are from a five-question test I ran on a customer support assistant, and they’re printed in the book. The scores are an LLM judge’s, on a scale of 1 to 5, from that one run. A different model or another run can score differently. Prompt A carried the relevant policy text as context, and prompt B had only the question. Look at the second row. A scored 1 with the policy sitting right there in the prompt, and B scored 5 with nothing. An average can’t show you a row like that.

So the answer I’m hoping for is a question back: can I see the score for each question? Then I go deeper. What happened in that second row? Who scored these, and why would you trust the scorer? What would you check before you believed a 5?

Tian Pan argued in an April 2026 post that the scarce skill in this job is noticing that the system is wrong while the metric says it’s fine. I think he’s right. Chapter 9 is that skill in practice: the judge that produced these scores, the comparison step by step, and the checks that catch an answer the source never supported.

7. “When would you fine-tune instead of adding retrieval?”

On this one the lists give you a fair place to start. Retrieval supplies knowledge at the moment the model answers, and fine-tuning changes the model itself, usually to change how it behaves. I ask it anyway, because the textbook sentence doesn’t tell me whether you can make the call on a real system.

So I point back at the table in question 6. Two of those rows are failures. In one, the model answered without the policy. In the other, it had the policy and still failed. The scores don’t tell you why. What would you look at to find out: the answers, the reference, the judge’s reasons? What would you try before you trained anything? What data would the training need, and where would it come from?

In December 2024 my colleagues and I published a comparison of this kind on the AWS Machine Learning Blog: an open-source model before and after fine-tuning on domain content, and a commercial model with retrieval, all tested on the same question set. Chapter 9 reads those two rows for you. Chapters 7 and 8, which are Eslam’s, are the rest of the answer: when post-training is justified, what data each method needs, and where it sits among everything you should try first.

8. “What does your AI coding assistant know when a session starts, and who keeps that current?”

First I ask you to show me how you work with an AI coding assistant. Then I ask this. It’s question 2 at full size: a new session knows only what its tools load or retrieve about your project, so whatever it needs has to be put where it will be found, and somebody has to keep that true.

I’ll start with my own record. For a long time I kept the index of project knowledge for my coding assistant by hand, and it went stale twice, in ways I didn’t notice until the assistant confidently cited a file I had moved three weeks earlier. I’ve also had an agent rewrite a module to follow a design the team had rejected months earlier. That decision was written in a meeting note and never made it into the code, so the agent couldn’t have known.

The smallest version of an answer is three fields at the top of each knowledge file:

---
name: Target allocation (current)
description: The 60/40 target adopted January 2026. Supersedes the 2025 80/20 target.
last_verified: 2026-07-01
---

From there the questions get harder. Who writes decisions down, and where? What happens when a fact expires? Is the index generated, or does a person keep it by hand and hope? It’s the same thing I’m watching for when I hand an FDE candidate that unfamiliar repository: whether you ground your AI in the actual code, or let it guess and ship the hallucination.

Chapter 11 builds the whole pattern on a ten-file example and then moves it to an engineering repository: the map, the facts that expire, the rules the assistant follows, and the checks that tell you when the map has gone stale.

9. “What’s the last result that surprised you?”

There’s nothing to memorize for this one, and no code. It’s the question I’d keep if I could keep only one.

If you’ve run things, you have a surprise. Mine is the second row of that table: a model that refused a question with the answer in its prompt, and then got it right with no context at all. Without a run of your own, the best you can do is repeat somebody else’s.

I also learn from other managers I know in the industry, and I read the ones who write about how they hire. They have their own versions of this question. Jason Roell, Head of Engineering at Vurvey, published his hiring guide in January 2026. One of his signals is asking what a candidate tried recently that didn’t work, and his fastest filter is handing over a real agent trace with a failure in it and giving the candidate fifteen minutes. Eugene Yan and Jason Liu, in their guide to hiring ML and AI engineers, call data literacy “a crucial yet overlooked skill” and describe engineers who look at rows of data firsthand, checking for missing values and outliers. As I read them, we’re all after the same evidence: that you looked.

Common AI interview questions, and where the book develops them

AI engineer is first on LinkedIn’s 2026 list of the fastest-growing jobs in the U.S., and in July 2026 the job-data firm LinkUp counted listings for forward deployed engineers going from about 300 to more than 5,000 in a year, often under titles like Applied AI Engineer. Many teams now have several of these disciplines working on one system, so whatever your title says, expect more questions than my nine. Here are other common ones, each with the chapter that develops it. This is a map. The answers are in the chapters.

  • What is in-context learning, and how is it different from fine-tuning? Chapter 1.
  • What are embeddings, and why do the query and the documents need the same embedding model? Chapter 3.
  • How do you choose a chunking strategy, a chunk size, and an overlap? Chapter 3.
  • Which vector database would you pick, and when would you use a managed RAG service instead? Chapters 3 and 4.
  • Walk me through a RAG pipeline end to end. What happens when the retrieved text doesn’t fit in the context window? Chapter 4.
  • What is hybrid search, and when is vector search alone not enough? Chapter 5.
  • How do you answer a question whose evidence sits in more than one document? Chapter 5.
  • What are LoRA and parameter-efficient fine-tuning, and when is fine-tuning justified? Chapter 7.
  • Prompting, retrieval, tools, or fine-tuning: how do you decide? Chapter 8.
  • How do you evaluate a RAG system, and can you trust an LLM as the judge? Chapter 9.
  • How do you detect hallucinations and reduce them? Chapters 4 and 9.
  • How do you give an agent memory, and what is context engineering? Chapter 11.

I’d also prepare for questions about the data a working system depends on. They matter most for FDEs, because they’re what a customer’s data does to a demo.

  • Where does your data come from, and are you allowed to use it? Chapter 2.
  • How do you stop an old document from outranking the current one? Chapter 5.
  • How do you make sure a user only gets answers from documents they’re allowed to see? Chapter 5.
  • When would you generate synthetic data, and how do you know it’s any good? Chapters 6 and 9.
  • How do you keep data quality from slipping after launch, and where does a person review? Chapter 10.

Why I point you to the book

You have nine of my questions now, the three snippets I expect you to understand line by line, and the map. You don’t have the answers to the deeper questions, because each of those took a chapter.

The book is the foundation these interviews stand on. I ask about the call, the context, the retrieval, the failures, the evaluation, and the assistant because everything newer is built on them, and you have to know the foundation before the latest thing makes sense. That holds for the interview, and it holds for going deeper in any AI job after it.

I think it will serve you better than any list of questions and answers you could memorize. A list prepares you for the question as it’s written. I take detours, and you can follow a detour only if you understand the ground under it and have practiced it in code.

The practice is the point of the ten Jupyter notebooks that come with it. You run the call, then the failure, then the fix, and then you change something and run it again. The notebooks find and test a working model at runtime instead of naming one, so they depend much less on any single model staying available. When you’ve been through one, point the same code at documents of your own, such as your team’s runbooks or a product manual. That’s your first project, and it’s the one you’ll walk me through when the interview starts: something you built yourself, a failure you found in it, the fix, and a number from before and after. I don’t know a better source of confidence in an interview than having watched the thing happen.

I co-wrote it, so of course I’d say that. It’s also the foundation I interview on. If one of these interviews is ahead of you, get the book and open the Chapter 1 notebook.

Data Strategy for LLMs: Build reliable AI applications with RAG, fine-tuning, evaluation, and context engineering

Then change one thing and predict what the model will do before you run it: drop the context, ask the same question twice, add a paragraph that contradicts the first. Whatever surprises you is where to start.

Which result from your own project would you want an interviewer to challenge?

Rany ElHousieny is a Senior Generative AI Engineering Leader with 30+ years across Microsoft, startups, and enterprise AI platforms, and the co-author, with Eslam Kamal, of Data Strategy for LLMs (Packt, 2026). Connect with him on LinkedIn, where he writes the newsletter AI Solutions Architect. Follow him on Medium to get the next piece by email.

Sources

Further Reading

The book: Data Strategy for LLMs, which I wrote with Eslam Kamal (Packt, 2026).

More of my pieces on the same ground. The older ones use the libraries of their day, so read them for the ideas.


AI Engineer Interview Questions: What I Ask After the Right Answer was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.