Signal & Noise
Menu

Interview Prep

Preparing for Gen AI roles, level by level — what to study, the questions you'll actually face, and what interviewers are really probing for. Live resources at the bottom refresh automatically.

AI Interview Coach

Practice the interview, not just the questions.

Get a tailored mock interview, direct feedback on each answer, and a sharper follow-up question.

Build my prep plan from a job description

Paste a Gen AI job description and get a tailored study plan — focus areas, a day-by-day schedule, likely questions, and one project to build. Then drill the matching Q&A bank below and rehearse with the AI coach.

0/6000 characters

Entry level

New grad · Junior GenAI engineer · AI-adjacent SWE

At this level nobody expects you to have trained a frontier model. Interviews test whether your fundamentals are real (not just vocabulary) and whether you've actually built something with an LLM API, however small.

What interviewers are really probing for

Solid intuition over buzzwords, one or two genuine projects you can defend line-by-line, and evidence you can learn fast. A well-explained toy RAG app beats a padded résumé every time.

Core topics

  • Transformer intuition: attention, context windows, why models hallucinate
  • Tokenization & embeddings — what they are, cosine similarity, when embeddings fail
  • Prompting techniques: few-shot, chain-of-thought, structured output
  • Working with LLM APIs: temperature, top-p, max tokens, streaming, function calling
  • Basic RAG: chunking, vector search, stuffing context
  • Python + one framework (LangChain, LlamaIndex, or raw SDK) hands-on

Q&A bank — tap a question for the model answer

55 questions

01Explain how a transformer processes a sentence, at a high level.

The sentence is split into tokens, each token becomes a vector (embedding), and positional information is added. Then stacked layers of self-attention let every token look at every other token and pull in the context it needs, followed by feed-forward layers that transform each position. At the end, the model outputs a probability distribution over the next token. The key idea to land: attention replaces recurrence — the model sees the whole context at once instead of reading left to right.

02What is attention, in one intuitive explanation?

Attention is a learned relevance score: for each token, the model asks 'which other tokens matter for understanding me?' and takes a weighted average of their information. In 'the animal didn't cross the street because it was tired', attention is what lets 'it' look hard at 'animal' rather than 'street'. Multi-head attention runs several of these lookups in parallel so different heads can track different relationships (syntax, coreference, position).

03What is a token, and why do models count them instead of words?

A token is the unit the model actually reads — usually a subword chunk produced by an algorithm like BPE. 'unbelievable' might be 'un', 'believ', 'able'. Models count tokens because their vocabulary is fixed subword pieces, their context window is measured in tokens, and pricing is per token. A useful rule of thumb: one token is roughly three-quarters of an English word.

04What is an embedding? Give a practical use.

An embedding maps text to a dense vector so that semantically similar texts land close together in vector space. Practical use: semantic search — embed all your documents, embed the user's query, and return the nearest neighbors by cosine similarity, which finds relevant passages even when no keywords overlap. Also used for clustering, deduplication, recommendation, and classification.

05What does temperature do in LLM sampling?

Temperature rescales the model's output distribution before sampling: values below 1 sharpen it toward the most likely tokens; values above 1 flatten it so less likely tokens get chosen more often. Use near-0 for extraction, classification, and code; 0.7–1.0 for creative writing. Bonus point: temperature 0 still isn't perfectly deterministic on most hosted APIs because of floating-point and batching effects.

06What's the difference between temperature and top-p?

Both control randomness but differently: temperature reshapes the whole probability distribution, while top-p (nucleus sampling) truncates it — only the smallest set of tokens whose probabilities sum to p are eligible, then sampling happens within that set. They're often combined. If asked which to tune, the safe answer is: pick one and hold the other fixed, since they interact.

07Why do LLMs hallucinate?

Because they're trained to produce plausible next tokens, not verified facts — there's no built-in lookup of ground truth. When the training data is thin on a topic, the model interpolates fluently and confidently. Contributing factors: prompts that demand an answer, long chains of reasoning, and questions about recent events beyond the training cutoff. Mitigations to name: retrieval (RAG), asking the model to say 'I don't know', citing sources, and validating outputs downstream.

08Name two things you could ship this week to reduce hallucinations in a chatbot.

One: ground it with retrieval — fetch relevant documents and instruct the model to answer only from them, saying so when the answer isn't there. Two: constrain and verify — lower the temperature, require citations to the retrieved passages, and add a lightweight check that flags answers containing claims not present in the sources. Both are days of work, not months.

09What is a context window, and what happens when you exceed it?

The context window is the maximum number of tokens the model can consider at once — prompt plus generated output. Exceed it and the API errors, or your framework silently truncates, usually dropping the oldest messages, which is how chatbots 'forget' early instructions. Practical handling: summarize or window old conversation turns, retrieve only relevant history, and keep system prompts tight.

10What is a system prompt, and how is it different from a user message?

The system prompt is standing instructions that frame the whole conversation — persona, rules, output format — while user messages are the turn-by-turn input. Models are trained to weight system instructions more heavily, but they're not security boundaries: a determined user can often override them, which is why real guardrails also live outside the prompt.

Open roles right now

Live listings matched to this level — pick a location. US roles come from the boards of Anthropic, OpenAI, Cohere, Scale AI, and ElevenLabs; India roles from Naukri, Sarvam AI, Turing, and the labs' India offices; plus remote roles via Remotive. Refreshes every few hours.

Mid level

GenAI engineer · ML engineer · Applied scientist (2–5 yrs)

The bar shifts from 'can you use an LLM' to 'can you ship and operate an LLM feature'. Expect deep dives on RAG quality, fine-tuning trade-offs, evaluation, and the cost/latency realities of production.

What interviewers are really probing for

Trade-off reasoning with numbers attached. Interviewers want scars from production: an eval you built, a retrieval pipeline you debugged, a cost you cut. 'It depends' answers must end with a decision.

Core topics

  • RAG beyond the demo: chunking strategy, hybrid search, re-ranking, retrieval evals
  • Fine-tuning: LoRA/QLoRA vs full fine-tune vs RAG vs prompt — decision framework
  • Evaluation: golden sets, LLM-as-judge (and its failure modes), regression testing
  • Serving economics: token costs, caching, batching, quantization (GGUF, AWQ)
  • Structured output & tool calling reliability; retries, validation, fallbacks
  • Agent basics: tool use, planning loops, when agents are the wrong answer
  • Guardrails: prompt injection, PII handling, content filtering

Q&A bank — tap a question for the model answer

53 questions

01Design RAG over a company's messy internal wiki. Where does quality die first?

Quality dies in ingestion: stale pages, duplicated content, tables and images that strip badly, and permission boundaries. Pipeline: crawl with freshness metadata, dedupe near-identical pages, chunk on structural boundaries (headings) with overlap, embed with metadata (space, owner, updated-at), hybrid retrieval with recency boosting, re-rank, then generate with citations. Call out permissions explicitly: retrieval must filter by the asking user's ACLs or you've built a data-leak machine.

02Your RAG answers are fluent but wrong. How do you localize the fault?

Split the pipeline and test each half. Log the retrieved chunks for failing queries: if the right passage isn't in them, it's retrieval — fix chunking, embeddings, hybrid search, or k. If the right passage IS there and the answer still contradicts it, it's generation — tighten the prompt ('answer only from context'), lower temperature, or add a groundedness check. Build a small eval set for each half so the fix is measurable, not anecdotal.

03Compare chunking strategies and their failure modes.

Fixed-size token chunks: simple, but cuts mid-thought and orphans context. Sentence/paragraph-aware: better boundaries, variable sizes. Structural (headings, sections): best for docs with real structure; fails on walls of text. Semantic chunking (split on embedding drift): highest quality, highest cost. Overlap of 10–20% papers over boundary cuts. The senior-sounding line: chunking is retrieval's biggest lever and almost nobody evaluates it — measure recall@k across strategies on your own corpus.

04What is hybrid search and when is it necessary?

Combining dense (embedding) retrieval with sparse keyword retrieval (BM25), merging scores — typically via reciprocal rank fusion. Necessary whenever queries contain exact identifiers embeddings handle badly: error codes, product SKUs, function names, people. Most production RAG over technical or enterprise content should be hybrid by default; pure vector search is a demo-stage choice.

05Explain re-ranking and its cost/benefit.

A cross-encoder scores each candidate chunk jointly with the query — far more accurate than embedding distance because it reads both texts together. Pattern: over-retrieve 30–100 candidates cheaply, re-rank, keep top 3–10. Benefit: often the single biggest RAG quality jump. Cost: an extra model call and 50–300ms latency. If quality matters and you don't have a re-ranker, that's usually the first thing to add.

06When do you fine-tune instead of (or in addition to) RAG?

RAG for knowledge — facts that change, need citations, or are proprietary. Fine-tuning for behavior — output format, tone, domain style, tool-use patterns, or squeezing a smaller model to do a big model's narrow job cheaply. They compose: fine-tune a small model to be excellent at your RAG answering format. Red flag answer to avoid: fine-tuning to teach the model new facts — it's unreliable for that and RAG dominates.

07Explain LoRA to a backend engineer: what's trained, what's frozen, why it's cheap.

The base model's weights are frozen. For chosen layers, you add a pair of small matrices whose product approximates the weight update — rank 8–64 instead of the full dimension — and train only those, typically under 1% of parameters. Memory drops enough to fine-tune 7–13B models on one GPU. QLoRA goes further by quantizing the frozen base to 4-bit during training. Artifacts are megabytes, so you can hot-swap adapters per customer or task.

08What is DPO and why has it largely displaced RLHF-with-PPO for preference tuning?

Direct Preference Optimization trains directly on preference pairs (chosen vs rejected responses) with a simple classification-style loss, skipping the separate reward model and reinforcement-learning loop of PPO. It's far simpler to implement, more stable, and cheaper, with comparable results for most use cases — which is why it and its variants became the default for open-weight alignment. Knowing when you'd still want full RL (complex, non-pairwise reward signals) is bonus credit.

09How do you evaluate an LLM feature before launch, concretely?

Three layers. Golden set: 100+ real inputs with graded reference outputs, scored on every change — exact checks for structure, rubric scores for quality. LLM-as-judge for scale, calibrated against a human-labeled sample so you know its bias. Pre-launch human review of a random sample plus adversarial probing. Define pass thresholds before testing. The phrase that lands: no eval, no launch — a feature without an eval set is unfalsifiable.

10What are the known failure modes of LLM-as-judge?

Position bias (favoring the first or last answer shown), length bias (longer looks better), self-preference (judging its own family's outputs kindly), sycophancy toward confident tone, and insensitivity to subtle factual errors. Mitigations: randomize order, compare pairwise both ways, use rubrics with binary sub-checks, calibrate against human labels, and use a different model family as judge than the one being judged.

Open roles right now

Live listings matched to this level — pick a location. US roles come from the boards of Anthropic, OpenAI, Cohere, Scale AI, and ElevenLabs; India roles from Naukri, Sarvam AI, Turing, and the labs' India offices; plus remote roles via Remotive. Refreshes every few hours.

Senior / Staff

Senior GenAI engineer · LLM platform/staff engineer · Architect

Interviews become system design and judgment under ambiguity: multi-team platforms, serving at scale, reliability, safety, and the organizational work of making AI systems maintainable by people other than you.

What interviewers are really probing for

End-to-end ownership and the ability to say no. Strong candidates bring an opinionated architecture, quantify it (tokens/sec, p99, $/query), name its failure modes unprompted, and show they've grown other engineers.

Core topics

  • LLM serving at scale: vLLM/TGI, continuous batching, KV-cache management, GPU utilization
  • Platform design: model routing, A/B-ing models, versioning prompts like code
  • Multi-agent orchestration: state, retries, human-in-the-loop, failure isolation
  • Observability: tracing LLM calls, drift detection, feedback loops, incident response
  • Safety & governance at the system level: red-teaming, permissioning agents, audit trails
  • Build-vs-buy and model strategy: open weights vs API, exit costs, data leverage
  • Leading through others: design reviews, mentoring, cross-org alignment

Q&A bank — tap a question for the model answer

41 questions

01Design the serving stack for a 10M-DAU assistant. Walk the request end to end.

Edge: auth, rate limits, abuse screening. Gateway: request classification and model routing (cheap model for simple intents, frontier for hard ones), prompt assembly with versioned templates, cache checks (exact + prefix). Inference: vLLM-class servers with continuous batching and paged KV cache for self-hosted paths, plus API providers behind a unified interface with circuit breakers; streaming throughout. Post: guardrail filters, structured-output validation, telemetry (tokens, cost, trace). State: conversation store with tiered memory summarization. Capacity: model peak concurrent decodes, TTFT SLOs drive GPU count; autoscale on queue depth. Close with failure modes: provider outage → fallback tier, cache stampede → request coalescing, long-context abusers → token budgets.

02How does continuous batching actually work, and why did it replace static batching?

Static batching waits to fill a batch, runs it to completion, and pads short sequences — GPUs idle on padding and everyone waits for the longest generation. Continuous (in-flight) batching schedules at the iteration level: each decode step, finished sequences exit and queued requests join immediately, keeping the GPU saturated. Combined with paged KV cache (vLLM's contribution — virtual-memory-style non-contiguous cache blocks eliminating fragmentation), it typically yields 2–4x throughput at better tail latency. This is the single most important serving concept at staff level.

03Explain KV-cache memory math and why it dominates serving capacity.

Every generated token attends over all prior tokens, so you cache per-token key/value tensors: memory ≈ 2 × layers × kv_heads × head_dim × bytes × tokens per sequence. For a 70B-class model this lands around 300KB+ per token — a 32k-token conversation holds ~10GB of cache, competing with weights for HBM. Concurrency is therefore bounded by cache, not compute. Levers: grouped-query attention (fewer KV heads), cache quantization, paged allocation, prefix sharing for common system prompts, and context caps. If you can do this arithmetic on a whiteboard, the room relaxes.

04When do you self-host open-weight models vs use API providers? Give the actual decision framework.

Quantify three axes. Economics: self-hosting wins when utilization is high and sustained — idle GPUs invert the math; compute break-even tokens/day against committed hardware. Control: data residency, fine-tuned checkpoints, latency floors, version pinning — if any is a hard requirement, APIs lose. Capability: if only frontier API models clear your quality bar, the decision is made for you. Most real orgs land hybrid: self-hosted fine-tuned small models for high-volume narrow tasks, API frontier models for the long tail. Present it as a portfolio, not a religion — and name the exit costs both directions.

05Design a model-routing layer. What signals route a request, and how do you keep it honest?

Signals: task type (classifier or explicit product surface), prompt complexity heuristics, user tier, latency budget, and historical difficulty for similar queries. Route to a ladder: small local → mid API → frontier, with escalation on low confidence or failed validation. Keeping it honest is the hard part: per-route eval sets so you know cheap-route quality; shadow-sampling a percent of cheap-routed traffic through the expensive model to measure the quality gap continuously; drift alarms on escalation rate. Anti-pattern to name: routing on token count alone — short prompts are often the hardest.

06Your org has 12 teams calling LLMs ad-hoc. Design the platform — and the migration.

Platform: a gateway service owning auth, model catalog, routing, caching, rate/cost quotas per team, prompt registry with versioning, unified telemetry, and guardrail middleware — teams get an SDK and stop holding provider keys. Paved road, not a cage: escape hatches for legitimate special cases. Migration is the senior half: inventory usage via provider billing and code search, migrate the two highest-spend teams first (they fund the ROI story), make the gateway strictly easier than direct calls, publish cost dashboards that make finance your ally, then deprecate direct keys on a dated schedule with exemptions requiring sign-off. Force through policy only after winning through product.

07A model upgrade silently regressed a downstream team's feature. What do you change so it never happens again?

Treat models like dependencies: pin versions explicitly, never float on 'latest'. Publish upgrades through the platform with a deprecation calendar. Each consuming team registers an eval contract — their golden set runs automatically against candidate model versions in CI, producing a per-team pass/fail before any rollout. Roll out canary-first with automatic rollback on eval or production-metric regression. The cultural fix matters as much: the platform team owns providing the harness; consuming teams own their eval content. 'We upgraded and hoped' becomes structurally impossible.

08Design multi-agent orchestration for a real workflow — say, insurance claim processing. Where does it fail?

Structure: an orchestrator (explicit state machine, not a free-form LLM loop) coordinating specialist agents — intake extraction, policy lookup, fraud signals, adjudication draft — each with least-privilege tools and its own eval. State persisted per step, idempotent and resumable; every hop traced. Human checkpoints at legally meaningful decisions. Failure modes to volunteer: error propagation (bad extraction poisons everything → validate at boundaries), runaway loops (hard step/token budgets), inter-agent 'telephone' distortion (pass structured artifacts, not prose), and the ops truth that debugging distributed LLM state without traces is archaeology. Senior signal: proposing the boring state machine over the exciting autonomous swarm.

09How do you bound what an agent can do — technically, not aspirationally?

Capability confinement in code: tools are allowlisted per agent per context; parameters validated against schemas with no raw SQL/URL/shell construction from model text; the agent acts under the end user's identity and scopes, never a super-service-account; mutating actions require idempotency keys and, above a risk threshold, human approval; egress restricted to declared endpoints; budgets on steps, tokens, wall clock, and spend enforced by the runtime. Log every tool call immutably for audit. The prompt can ask the agent to behave; only the sandbox makes it true.

10Model your feature's unit economics: $0.04/query today, and your plan to halve it.

Decompose first: input vs output tokens, retries, judge calls, retrieval infra amortization — the breakdown usually surprises. Typical plan: prefix caching on the shared system prompt (25–40% of input spend at many shops), route the easy 60% of queries to a model at a tenth the price after eval proves parity, cap and tighten outputs (verbosity is pure cost), distill the narrow high-volume path onto a fine-tuned small model, and move async work to batch APIs at half price. Sequence by effort-to-savings, re-run quality evals at each step, and report the curve weekly. Halving is usually conservative.

Open roles right now

Live listings matched to this level — pick a location. US roles come from the boards of Anthropic, OpenAI, Cohere, Scale AI, and ElevenLabs; India roles from Naukri, Sarvam AI, Turing, and the labs' India offices; plus remote roles via Remotive. Refreshes every few hours.

Leadership

AI product manager · Head of AI · Director / VP

The technology questions get easier; the judgment questions get harder. Interviews test whether you can pick the right problems, resource them, govern the risk, and translate between the board and the builders.

What interviewers are really probing for

Business-outcome thinking with technical credibility. Great answers tie every AI initiative to a metric someone outside the AI team cares about, and treat risk/governance as a design input, not paperwork.

Core topics

  • AI strategy: use-case portfolio, sequencing, build/buy/partner, moats and data flywheels
  • Measuring success: leading vs lagging metrics for AI features, when to kill a project
  • Governance & compliance: EU AI Act awareness, model risk, incident playbooks
  • Org design: platform vs embedded AI teams, hiring profiles, managing researchers vs engineers
  • Vendor & model strategy: negotiating with providers, avoiding lock-in, cost forecasting
  • Responsible AI as product policy: hallucination tolerance by use case, human oversight
  • Narrative: communicating AI progress honestly to execs, boards, and customers

Q&A bank — tap a question for the model answer

26 questions

01You have budget for 3 of 10 proposed GenAI initiatives. Show me your selection framework.

Score each on: business value (revenue, cost, or risk reduction with an owner who'll sign the number), feasibility (data readiness, error tolerance of the workflow, integration surface), time-to-signal (can we prove or kill it in 8 weeks?), and strategic compounding (does it build data, platform, or capability the others reuse?). Kill anything where the workflow can't tolerate model errors or the data doesn't exist. Then portfolio-balance: one near-certain cost-saver to fund credibility, one revenue-side bet, one platform enabler. Publish the scoring so the seven declined teams see reasons, not politics.

02How do you measure whether an AI feature is actually working — beyond usage graphs?

Tie it to the business metric it was funded on: deflection rate and CSAT for support, cycle time for drafting, conversion for search. Instrument outcome proxies — acceptance rate of AI outputs, edit distance before use, escalation/abandonment rates — and run holdouts (a control group without the feature) because adoption isn't impact. Add counter-metrics for the failure you fear: error-caused rework, complaint rate, trust scores. Review quarterly against the original business case, and be the person willing to say 'usage is high, value isn't materializing' — that credibility funds your next three asks.

03Legal asks: 'Can we guarantee the model never gives financial advice?' What's your answer?

Honest framing: no probabilistic system offers a 'never' — what we offer is engineered risk reduction with measurement. Concretely: topic classifiers and output filters on the advice boundary, tested refusal behavior, human review on flagged categories, and a measured escape rate ('under X per 10k interactions in adversarial testing, here's the dashboard'). Then reframe the decision as risk acceptance: here's residual risk, mitigation cost curve, and comparable industry practice — legal and business jointly decide the threshold. Leaders who promise 'never' to legal are setting up tomorrow's crisis; leaders who quantify earn a partner.

04Design the AI org for a 2,000-person company starting mostly from zero.

Start hub-heavy, evolve spoke-heavy. Year one: a central AI platform team (gateway, evals, governance, 8–12 people) plus 2–3 embedded pods in the highest-value business units, all under one leader with a seat at product strategy. Deliberately avoid: a research lab (you're not a lab), an 'AI center of excellence' that only writes decks, and scattering individual ML engineers into teams with no support. As patterns stabilize, push capability outward — platform stays central, use-case ownership moves to domains with certified tooling. Hiring order: platform lead, staff GenAI engineer, product-minded PM, then pods. Revisit the design at 18 months; org structures for AI age fast.

05Your flagship AI feature hallucinated publicly and it's in the press. First 48 hours?

Hour one: assess blast radius and, if harm is ongoing, degrade or kill the feature — a day of downtime is cheaper than a week of screenshots. Communicate in this order: affected users (direct, factual, what happened and what we're doing), internal teams (one source of truth, no freelancing to press), then public statement — own it plainly, no 'the model did it' deflection; you shipped the model. Parallel: engineering runs the incident process on traces, adds the failing case to evals, and identifies why testing missed it. Within 48 hours: root-cause summary to leadership with the systemic fix and its cost. The reputational asset you're protecting is not 'we're perfect' but 'they handle problems like adults'.

06A team wants six months to fine-tune a custom model; an API call gets 90% of the value today. Decide.

Ship the API version now — the 90% starts compounding value and, critically, generates the production data and eval baselines any future fine-tune needs. Then make the fine-tune case earn itself with numbers: at what volume does unit cost cross over, is the remaining 10% actually valuable to users, and is latency or data residency a real constraint? Often the answer is 'revisit at scale', sometimes fine-tuning genuinely wins — but sequence it as an optimization of a live product, not a prerequisite. Watch for the real motivation: résumé-driven engineering. Redirect that energy to owning the eval and data pipeline, which is the durable skill anyway.

07How do you keep AI strategy honest when the board wants 'agents everywhere' by Q3?

Translate the enthusiasm rather than fighting it: agree on the outcome the board actually wants (efficiency, competitiveness, a story for investors) and map what agentic maturity realistically delivers by Q3 — likely 2–3 workflows with human-in-the-loop autonomy and measured ROI, not 'everywhere'. Bring a maturity ladder with evidence gates and show competitors' real deployments versus their press releases (the gap is your ally). Commit to the ambitious-but-real version in writing with metrics. Boards respect leaders who negotiate scope with data; they replace leaders who either overpromise and miss, or reflexively say no.

08How do you build AI literacy across a non-technical workforce without wasting a year on training theater?

Anchor to workflows, not concepts: role-specific training built around the three tasks each function will actually do with AI tools, delivered in the tools, not in slides. Create a champions network — one volunteer per team, given early access and a direct line to the AI org — which surfaces real use cases and quietly retires fear. Publish clear usage policy (what's allowed, what data can go where) because uncertainty suppresses adoption more than inability. Measure behavior change (tool usage in workflow, time saved self-reported and sampled) not completion certificates. And executives train first, visibly — nothing licenses adoption like the CFO using it in a meeting.

09What's your policy framework for hallucination risk across different product surfaces?

One dial doesn't fit: classify surfaces by consequence of error. Tier 1 (internal drafts, brainstorming): model output direct, light disclaimers. Tier 2 (customer-facing information): grounding required, citations, measured groundedness thresholds as launch gates. Tier 3 (decisions affecting money, health, legal standing): human owns every output, AI is draft-only, with review UX that makes rubber-stamping hard. Each tier has quantitative gates (error rates from adversarial evals), an owner, and audit sampling in production. The framework's power is that new features slot into a tier in one meeting instead of relitigating philosophy every launch.

10How do you negotiate with AI vendors from a position of strength?

Strength comes from credible optionality: maintain a qualified second model/vendor for every critical route (your eval harness makes re-qualification cheap — this is why you funded it), and let vendors know switching is a sprint, not a rewrite. Negotiate on committed volume with re-open clauses tied to the price-performance curve, which in this market only moves in your favor. Watch the lock-in surfaces beyond price: proprietary features (caching semantics, tool formats), data terms, and deprecation policies — get model-sunset notice periods in writing. And time renewals after major open-weight releases; nothing improves a quote like a viable self-host alternative on the table.

Open roles right now

Live listings matched to this level — pick a location. US roles come from the boards of Anthropic, OpenAI, Cohere, Scale AI, and ElevenLabs; India roles from Naukri, Sarvam AI, Turing, and the labs' India offices; plus remote roles via Remotive. Refreshes every few hours.

System design: Gen AI & Agentic AI

Full design walkthroughs — RAG platforms, agents with real permissions, serving at scale, evals-as-CI, and the autonomy framework that answers half the agentic questions you'll get.

28 questions

01Design an enterprise document Q&A system (RAG) for 5M documents and 50k users.

Clarify first: document types and formats, update frequency, permission model (per-doc ACLs?), latency target, languages, and what a wrong answer costs — these change the design more than scale does.

Architecture — ingestion: format-specific parsers (PDF/OCR, HTML, Office) → structural chunking on headings/paragraphs with 10–15% overlap → embeddings + rich metadata (ACLs, owner, updated-at, source system) → vector store with a hybrid (dense + BM25) index. Re-indexing is event-driven off document-change webhooks, not nightly batch — staleness is a top-3 complaint in real deployments.

Architecture — query path: authenticate → embed query → ACL-filtered hybrid retrieval over-fetching ~50 candidates → cross-encoder re-rank to top 5–8 → prompt assembly with citations and an explicit refusal instruction ('if the context doesn't contain the answer, say so') → streamed generation → async groundedness check sampling.

Key decisions: - Permissions are enforced server-side, pre-search, in the retrieval engine — never by filtering after retrieval or, worse, trusting the prompt. This is the #1 security risk in enterprise RAG. - Hybrid retrieval by default: enterprise queries are full of exact identifiers (project codes, error strings) that pure vector search misses. - Store raw text durably and separately from the index so re-embedding on a model upgrade is mechanical.

Scale & cost: 5M docs ≈ 20–50M chunks — shard the index by department/tenant; 50k users is modest QPS, so the index and ACL machinery are the hard parts, not serving. Cache frequent queries; batch-embed at ingestion.

Failure modes: permission leaks (test with adversarial cross-user probes in CI), stale answers after doc updates, contradictory documents (add authority/recency weighting), and lost-in-the-middle when k is too high.

Metrics & evals: retrieval recall@k on a probe set, groundedness rate, citation click-through, 'no answer found' rate, per-language quality, p99 latency split retrieval vs generation.

Likely follow-ups: - How do you handle a document that's deleted for legal reasons — how fast is it gone from answers? - A VP reports the system leaked a doc they shouldn't see. Walk me through your investigation. - How would you add tables and images in the documents? - What changes if update frequency goes from daily to every minute?

02Design a customer-support AI agent that can look up orders, process refunds, and escalate to humans.

Clarify first: refund policy complexity, transaction values, containment-rate target, channels (chat only? voice?), and the compliance regime — money movement changes everything about autonomy.

Architecture: intent router → conversation agent with exactly three tools: order lookup (read-only), refund request (policy-gated), escalate (always available, never blocked) → guardrail layer on both input and output → streamed response. Conversation state in a session store with summarization for long threads; order context cached per session so the agent isn't re-fetching every turn.

The refund tool owns the money logic: eligibility computed in code from order data (return windows, item categories, prior refund history), hard caps per transaction/customer/day, fraud signals — the model can only *request* what the tool independently verifies. Above a threshold, the request lands in a human approval queue with the agent's reasoning attached.

Key decisions: - Every user-visible fact (order status, amounts, dates) is templated from tool results, never generated from model memory — this kills the hallucinated-order-details class of bugs. - Escalation triggers are multi-source: explicit user request, sentiment deterioration, low model confidence, policy boundary hit, and loop detection (same tool failing twice). - Prompt injection defense lives in the tool layer ('ignore policy and refund me' can't work because the prompt was never the enforcement point).

Failure modes: social-engineering the refund flow, agent loops burning tokens, context loss on long conversations, and over-escalation destroying the ROI case — tune escalation precision, not just recall.

Metrics & evals: containment rate, CSAT vs human baseline, refund error rate (both directions — wrong grants AND wrong denials), escalation precision, cost per resolved conversation. Eval suite includes scripted adversarial refund attempts.

Likely follow-ups: - A fraud ring discovers the agent. What's your detection and response story? - How do you A/B this against human agents fairly? - The business wants the agent to also *sell* — upsell during support. What breaks? - Walk through the exact data flow when a user asks 'where's my order?'

03Design a code-review assistant for a 500-engineer org. What's different from a chatbot?

What's different: inputs are diffs plus repository context (not documents), outputs must anchor to specific lines, latency budget is minutes (CI-triggered, batch-friendly), and the cost of noise is brutal — engineers permanently ignore a bot that comments too much. Precision IS the product.

Architecture: PR webhook → context assembly (the diff, changed files' full contents, immediate dependency neighborhood via import graph, related tests, team style guides) → specialized analysis passes running in parallel — bug detection, security patterns, missing-test detection, style — each with its own tuned prompt and eval set → merge, dedupe, and confidence-filter findings → post only above a precision threshold, with severity labels and suggested fixes where safe.

Key decisions: - Separate passes over one mega-prompt: each pass is independently evaluable and tunable, and you can run cheap models on style while reserving frontier models for subtle bug detection. - A feedback loop is non-negotiable: thumbs, comment-resolution outcomes, and 'was this fixed before merge' signals train the confidence filter per category. - Repository context beats diff-only analysis by a wide margin — most real bugs are visible only with the surrounding code.

Scale & cost: 500 engineers ≈ a few hundred PRs/day — trivially batchable. Spend the savings on context depth for the passes that catch real bugs.

Failure modes: noise death-spiral (engineers mute the bot — start conservative and earn volume), style-war comments (defer to linters; the LLM handles what linters can't), stale style guides in prompts, and secrets in diffs reaching external APIs (redact first, or self-host).

Metrics & evals: eval on labeled historical PRs with known-escaped bugs — measure catch rate against comment volume; in production track comment resolution rate, mute rate, and per-category precision.

Likely follow-ups: - An engineer says the bot missed an obvious bug. What's your process? - How would you make it learn team-specific conventions without per-team prompt sprawl? - Now make it fix the bugs it finds, not just comment. What changes in the risk model? - How do you keep it from approving/blocking merges politically — who owns its authority?

04Design semantic search over a product catalog with 10M SKUs, 200ms p99, heavy filtering.

Clarify first: query mix (exact model numbers vs vague descriptions), filter cardinality (price, brand, availability, category), update rates for price/stock vs product text, and whether personalization is in scope.

Architecture: offline batch pipeline embeds product text (title + attributes + description) → vector engine with strong *pre-filtering* support (Vespa, Qdrant, or partitioned pgvector) → query path: embed query with a small fast model (~20ms) → filtered ANN search (~30ms) → light cross-encoder re-rank of top 100 (~50ms) → results. Fuse with BM25 throughout: catalogs are full of exact identifiers where keyword match must win.

The critical detail — filtered ANN: post-filtering top-k collapses recall when filters are selective (top-100 nearest neighbors, then 'in stock under $50' leaves 3 results). The engine must apply filters *during* graph traversal, or you partition indexes by high-selectivity dimensions. This distinction is what the question is really testing.

Key decisions: - No LLM in the 200ms hot path. Query understanding (rewrites, spell-fix, attribute extraction) runs behind a cache — query distributions are Zipfian, so cached rewrites cover most traffic — or async to improve *future* queries. - Metadata (price, stock) updates via streaming upserts without re-embedding; re-embed only on text change. - Embedding model choice is a latency decision as much as a quality one.

Failure modes: filter-recall collapse, cold-start for new products (embed synchronously at creation), embedding upgrades requiring full re-index (blue/green index swap), and head-query regressions hiding in average metrics.

Metrics & evals: online — CTR, add-to-cart, zero-results rate; offline — judgment lists per query class (exact, vague, misspelled). Segment everything head vs tail.

Likely follow-ups: - Where exactly does hybrid fusion happen and how do you tune the weights? - Add personalization without blowing the latency budget. - A merchandiser wants to boost certain products. Design the override layer. - Multi-language catalog: one index or per-language?

05Design a meeting-notes product: transcribe, summarize, extract action items, at 1M meetings/day.

Clarify first: latency expectation (live notes vs minutes-after), languages/accents, consent and privacy requirements per jurisdiction, and integration targets (calendar, Slack, CRM) — the extraction schema depends on where outputs land.

Architecture — async pipeline: audio → ASR with speaker diarization (streaming for live features, batch for economy) → transcript store (the durable source of truth) → summarization: map-reduce for long meetings (chunk summaries → merged executive summary), direct long-context for short ones → structured extraction pass for action items {owner, task, due date} with schema validation → delivery integrations. Every stage idempotent and independently retryable; a stuck summarization never blocks transcript availability.

Scale & cost — the design center: 1M meetings/day × ~15k transcript tokens is enormous but almost entirely batch-shaped. Use batch inference APIs (~50% discount), reserve the low-latency path for the premium 'notes 5 minutes after the meeting' tier, and route by meeting importance: long executive meetings get the strong model, dailies get the cheap one. Publish a cost-per-meeting model and defend it.

Key decisions: - Faithfulness over fluency: an invented decision or misattributed commitment destroys trust permanently. Ground every summary claim against transcript segments and link them (click a summary line → jump to transcript). - Speaker attribution accuracy gates action-item quality — 'who said they'd do it' is the product. - Privacy as architecture: consent capture, configurable retention, PII redaction before any secondary use, and per-workspace data isolation.

Failure modes: ASR errors compounding into wrong summaries (surface confidence), cross-talk breaking diarization, hallucinated action items (extraction prompt requires verbatim evidence spans), and long-meeting truncation.

Metrics & evals: human-annotated meeting eval set; measure summary faithfulness (claim-level), action-item precision/recall, owner-attribution accuracy, and edit rate on delivered notes.

Likely follow-ups: - Live in-meeting notes: what changes end to end? - A user disputes 'I never said that.' What's your evidence trail? - How do you handle a 4-hour meeting within model context limits? - Design the 'search across all my past meetings' feature on top.

06Design an AI coding agent that can work on a repo autonomously for hours (Devin-style). What are the hard parts?

Architecture — the loop: plan → locate relevant code (search tools, not context-stuffing) → edit → run/build/test → observe → revise, executing inside a sandboxed environment: containerized repo checkout, no network egress except allowlisted package registries, CPU/memory/time limits. The output artifact is a PR with the agent's plan, diff, and test evidence — never a direct push.

The four hard parts: - Context management: repos exceed any context window. The agent needs retrieval tools (grep, symbol lookup, file open) plus a persistent scratchpad of what it's learned — treating context as a cache over durable working memory, not the memory itself. - Verification: tests are the ground truth, but agents game weak tests (deleting assertions, special-casing). Require green CI on *unmodified* test files, measure on held-out validation tasks, and diff-review test changes with extra scrutiny. - Error recovery: detecting it's stuck — repeated failures on the same file, oscillating edits — and backtracking to re-plan instead of thrashing. Naive loops burn hours and dollars on the same wall. - Cost/time budgets with checkpoints: hours-long tasks need resumability — persisted plan state and progress so a crash at hour three doesn't restart from zero.

Key decisions: small reviewable increments over fire-and-forget (reliability drops sharply with task length — design the product around resumable, human-checkpointed chunks); tool results compacted before entering context (a 10k-line test log becomes a failure summary); and the sandbox is the security boundary, not the prompt.

Failure modes: test-gaming, dependency hallucination (installing packages that don't exist — a real supply-chain attack surface, pin and verify), scope creep beyond the ticket, and secrets exfiltration attempts via injected repo content.

Metrics & evals: task success rate on SWE-bench-style suites plus your own repo's historical issues, human-review pass rate of generated PRs, cost per completed task, and time-to-first-meaningful-progress.

Likely follow-ups: - The agent's PR passes CI but reviewers say the approach is wrong. How do you feed that back? - Design the permission model for it to work across 50 private repos. - How would you run 100 of these concurrently — what's shared, what's isolated? - What tasks would you *never* give it, and how does the system enforce that?

07Design the memory system for a personal AI assistant used daily for a year.

Architecture — four memory tiers: - Working: recent conversation turns, verbatim, in context. - Episodic: per-session summaries, embedded and retrievable ('what did we discuss about the Berlin trip?'). - Semantic: extracted durable facts — preferences, relationships, commitments — in structured storage with provenance (which conversation, when) and timestamps. - Procedural: learned standing instructions ('always book aisle seats', 'sign emails as Dee').

Write path: after each session, an extraction pass proposes memory candidates → validation before persisting (confidence, consistency with existing memories) → conflicts resolved by recency + confidence, with contradictions surfaced rather than silently overwritten. Read path: each turn retrieves across tiers within a token budget — memory competes with conversation for context, so retrieval is ranked, not exhaustive.

Key decisions: - User-visible, editable memory. It's simultaneously the trust feature ('here's what I know about you'), the correction mechanism, and your GDPR answer. - Provenance on every fact: when memory is wrong, you can trace why and expire the source. - Expiry and re-confirmation policies: 'lives in Austin' ages differently than 'is vegetarian'.

Hard problems to volunteer: - Memory poisoning: injected or manipulative content must not write memories — validation gates and source-trust weighting on the write path. - Right-to-forget cascades: deleting a conversation must delete facts *derived* from it — which requires the provenance links. - Drift: stale facts confidently asserted feel worse than no memory at all. - Eval methodology: memory bugs are invisible in single sessions — you need scripted multi-week scenario suites replayed against the system.

Metrics: fact precision (sampled audits), correction rate, memory-attributed task success (bookings using stored preferences), retrieval latency, and user trust signals (memory-panel engagement, deletion rates).

Likely follow-ups: - Two memories contradict. Show the resolution logic end to end. - The user shares an account with a spouse. What breaks and how do you fix it? - How does memory interact with a model upgrade? - Design the privacy story for enterprise deployment of the same assistant.

08Design a multi-agent research assistant that produces a cited report from a complex question.

Architecture — a pipeline, not a chatroom: planner decomposes the question into sub-queries → parallel searcher agents work web/internal sources → an extractor pulls *structured claims with citations* into an evidence store → synthesizer drafts sections strictly from the evidence store → critic pass verifies claim-citation support and coverage gaps, looping a bounded number of times → composer assembles the final report with a bibliography.

The load-bearing decision — the evidence store: agents hand off structured artifacts {claim, source URL, quote span, confidence, timestamp}, never prose summaries of each other's work. This prevents telephone-game distortion, makes every claim auditable, and turns 'is this report true' into a checkable property: every sentence maps to evidence or gets flagged.

Key decisions: - Source quality scoring (domain authority, recency, independence) weighted at synthesis — three blog posts citing each other is one source, not three. - Hard budgets: max searches, max tokens, max wall-clock, enforced by the orchestrator. Research loops are the classic runaway-cost pattern. - Injection defense: web content is data, never instructions — delimited, sanitized, and the searcher agents have no tools beyond search itself, so hijacking one gains nothing. - Contradiction handling: the report *presents* conflicting evidence with both citations rather than silently picking a side.

Failure modes: citation laundering (claim cites a source that doesn't actually support it — the critic pass exists for exactly this), coverage gaps on the question's hard sub-parts (planner decomposition quality gates everything), stale sources for time-sensitive questions, and confident synthesis from thin evidence (minimum-evidence thresholds per claim).

Metrics & evals: claim-level factuality sampling against sources (human-audited), citation-support rate, coverage rubrics scored against expert-written baselines, cost and latency per report.

Likely follow-ups: - A source is paywalled or robots-blocked. What's the policy and the fallback? - How would you make this incremental — a living report that updates as new sources appear? - The question is adversarial ('prove X is true'). How does the system stay honest? - Which parts would you collapse into a single agent, and when?

09Design real-time voice AI (sub-second responses) for phone support. Where does latency go?

The latency budget (~800ms perceived): - Streaming ASR emitting partials: 100–200ms behind speech - Turn detection / endpointing: 100–300ms — the underrated hard problem; too eager and you interrupt users mid-thought, too lazy and the bot feels dead - LLM time-to-first-token: 200–300ms — small fast model, ruthlessly trimmed prompt - Streaming TTS, starting on the first complete sentence: ~100ms to first audio

The real design insight — overlap everything: the LLM starts on stable ASR partials before the user finishes; TTS starts before the LLM finishes; likely next-turn context is precomputed during silence. Pipeline overlap engineering matters more than model choice — a mediocre model in a well-overlapped pipeline beats a frontier model in a sequential one.

Architecture: telephony gateway → full-duplex audio session (barge-in support: user can interrupt TTS, which must stop instantly and yield the turn) → streaming ASR → dialog manager with turn detection → LLM with tool access → streaming TTS. Tool calls (account lookup, order status) run async behind verbal fillers ('let me check that for you') — a natural-sounding 2-second filler buys your database round-trip.

Key decisions: - Quality/latency routing: simple intents take the fast path; complex requests get an acknowledgment plus the slower strong model. - Cache greetings, confirmations, and common answers as pre-synthesized audio. - Conversation state must survive mid-call transfers to humans — the handoff includes transcript, extracted intent, and attempted resolutions.

Failure modes: endpointing errors (the top UX complaint), ASR errors on names/numbers (read back and confirm anything consequential), TTS pronunciation of domain terms, and dead air during tool latency spikes (always have a filler strategy).

Metrics: turn latency p95, interruption/barge-in rates, containment, transfer quality scores, and per-minute cost against the human-agent baseline.

Likely follow-ups: - User says a 16-digit card number. Walk through capture, confirmation, and compliance. - How do you regression-test a voice pipeline in CI? - Add multilingual support — what breaks in the latency budget? - When the LLM provider has a 30s outage mid-call, what does the caller experience?

10Design the eval and release system for prompts/models — CI/CD for LLM behavior.

The frame: ship prompts and model configs exactly like software — versioned artifacts, tests, staged rollout — with datasets playing the role of test suites.

Architecture — registry: prompts, model configurations, and eval datasets are versioned artifacts with owners, changelogs, and lineage. Every production request logs which versions served it — the precondition for both debugging and rollback.

CI on every change: affected golden sets run automatically (cached where inputs didn't change, for speed and cost) → scorers execute — exact checks for structure, calibrated LLM judges for quality, safety suites for policy → a regression report compares against baseline *per category*, not just on averages. The mean improving while refusal-handling regresses 20% is the classic silent failure; per-category diffs with significance testing catch it.

Gates: hard-block on safety regressions or category drops beyond threshold; human sign-off tier for sensitive surfaces (legal, medical, money); auto-pass for neutral changes. Gate policy is code-reviewed like any config.

CD: canary rollout (1% → 10% → 100%) with production guard metrics — parse-failure rate, refusal rate, latency, user feedback — and automatic rollback on breach. Rollback must be one click *and actually tested*, which requires the versioned-everything discipline above.

The flywheel: production sampling routes real traffic into labeling queues weekly; interesting failures become permanent dataset entries. Datasets that don't grow with the product become theater.

Key decisions: judges are calibrated against human labels quarterly (drift is real); eval compute is budgeted and cached (naive re-runs of everything on every commit gets expensive fast); and dataset versioning includes decontamination checks against anything used in fine-tuning.

Failure modes: eval overfitting (teams tuning to the golden set — hold out a hidden set), judge gaming, flaky judge scores blocking releases (score with confidence intervals), and the cultural one — teams bypassing the system when it's slower than YOLO-shipping. The paved road must be fast.

Metrics: regression escape rate (bugs reaching production that evals should have caught), eval runtime, rollback frequency and time-to-rollback.

Likely follow-ups: - A provider silently updates their model behind the same version string. How do you detect it? - Two teams share a prompt and one wants to change it. Walk through the process. - How do you eval multi-turn conversational behavior, not just single-shot? - What's the minimum viable version of this for a 5-person startup?

Ground rules, every level

The advice that doesn't change with seniority.

Bring one system you know cold

Every level of interview goes better anchored to a real system you built or ran. Depth on one beats breadth on ten — expect five 'why' questions deep.

Numbers are the difference

Latency, cost per query, eval scores, adoption. Candidates who quantify are remembered as senior regardless of title.

Read the room's incentives

Startups probe shipping speed and scrappiness; big tech probes scale and safety; consultancies probe communication. Same question, different rubric.

Stay current — it's checked

Interviewers increasingly ask about this month's releases and papers. Ten minutes a day on a radar (ahem) compounds into easy credibility.

Live prep resources

The most-starred LLM interview-prep repos on GitHub, refreshed automatically.

Hiring chatter

What Hacker News is saying about AI hiring and interviews, from the past three months.