Design an evaluation system for self-driving models. The system must support multiple evaluation modalities: (1) human evaluation (human-in-the-loop annotation and review, similar to Scale AI / Mercor business model), and (2) LLM-based automated evaluation. The discussion is fully open-ended. Interviewers probe deeply into simulation-specific concerns: how to simulate unexpected real-world events (e.g., road hazards), how to detect when simulated objects don't look realistic (sim-to-real gap), how to run thousands of simulations under compute constraints, how to accelerate simulations, and how to handle bugs discovered during live demos.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Cover happy path. Clarify scope. Identify the obvious bottleneck. Pick a reasonable storage and reasonable scaling approach.
All of the above plus: explicit failure handling, durability vs latency trade-offs, choose the right batching/caching strategy, articulate why.
All of the above plus: organizational concerns (rollout, migration, on-call), quantitative analysis, multi-region considerations, what could go wrong with the proposed solution at 10x scale.
Common mistakes: Applying a generic/classic system design template without adapting to self-driving domain specifics; Not drawing any diagrams or visual architecture — making the discussion hard to follow; Ignoring sim-to-real gap and compute constraints entirely; Not connecting the design to real-world analogues (Scale AI, Mercor) which anchors the scope
What passers do: Framing the problem with a concrete real-world analogy (Scale AI / Mercor) to anchor scope quickly; Proactively addressing both human eval and LLM eval tracks as distinct subsystems; Demonstrating familiarity with simulation-specific challenges (sim-to-real, compute budgets)
Why people fail: Mechanically applying generic SD framework without engaging with domain-specific probing questions; Failing to draw any diagrams, leading to an unstructured open-ended conversation; Inability to answer unusual follow-up questions about simulation realism and compute constraints; Not adapting when interviewer pivoted away from standard system design topics
Edge cases probed: Simulating rare/unexpected on-road events (e.g., potholes, sudden obstacles); Detecting sim-to-real artifacts — when simulated objects don't look realistic; Running thousands of simulations under tight compute constraints; Simulation acceleration strategies; Handling bugs discovered during live showcase/demo
Alternative approaches: Generic distributed evaluation platform (no simulation focus) (Covers scalability and human/LLM eval well but misses domain-specific concerns like sim-to-real gap and compute-constrained simulation scheduling — likely to trigger heavy follow-up probing from the interviewer.); ML-system-design framing (model registry + offline batch eval) (Addresses model versioning and offline metrics well, but does not naturally surface human-in-the-loop workflows or simulation acceleration; interviewer may redirect toward operational/simulation concerns.)