AO
AceOffer
·
Back
Waymo Common Problems

Evaluation System Design

System DesigneasyLast reported April 2026
By AceOffer · Updated April 2026 · Reported 2× across 190+ candidate reports

Understanding the Problem

Design an evaluation system for self-driving models. The system must support multiple evaluation modalities: (1) human evaluation (human-in-the-loop annotation and review, similar to Scale AI / Mercor business model), and (2) LLM-based automated evaluation. The discussion is fully open-ended. Interviewers probe deeply into simulation-specific concerns: how to simulate unexpected real-world events (e.g., road hazards), how to detect when simulated objects don't look realistic (sim-to-real gap), how to run thousands of simulations under compute constraints, how to accelerate simulations, and how to handle bugs discovered during live demos.

Functional Requirements

Structured requirements coming soon. For now, see the full problem statement above and the deep-dive prompts below.

Non-Functional Requirements

Latency, throughput, availability, consistency targets — being authored.

The Set Up

Defining the Core Entities

Core entities (Request, Batch, Worker, Cache, etc.) — being authored.

The API

POST /endpoint → describe request shape GET /endpoint → describe response shape (API spec being authored)

High-Level Design

Component diagram + walkthrough mapping each functional requirement to a system flow — being authored.

Potential Deep Dives

These are the directions the interviewer is likely to push you. Each one has multiple valid solutions at different quality tiers.

1)How would you simulate unexpected events on the road, like a sudden pothole or an obstacle? (when: Candidate proposes simulation as part of evaluation)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

2)How would you detect that objects in your simulation don't look like the real world? (when: Candidate describes a simulation platform)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

3)You have no compute budget — how do you still run thousands of simulations? (when: Candidate proposes running large-scale simulations)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

4)How would you accelerate simulations? (when: Candidate proposes simulation acceleration)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

5)What if there's a bug discovered during a live demo? (when: Candidate describes a live demo / showcase environment)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

What is Expected at Each Level?

L4 / Mid-level
Cover happy path. Clarify scope. Identify the obvious bottleneck. Pick a reasonable storage and reasonable scaling approach.
L5 / SeniorTarget
All of the above plus: explicit failure handling, durability vs latency trade-offs, choose the right batching/caching strategy, articulate why.
L6 / Staff+
All of the above plus: organizational concerns (rollout, migration, on-call), quantitative analysis, multi-region considerations, what could go wrong with the proposed solution at 10x scale.

Insider Notes

**Common mistakes:** Applying a generic/classic system design template without adapting to self-driving domain specifics; Not drawing any diagrams or visual architecture — making the discussion hard to follow; Ignoring sim-to-real gap and compute constraints entirely; Not connecting the design to real-world analogues (Scale AI, Mercor) which anchors the scope **What passers do:** Framing the problem with a concrete real-world analogy (Scale AI / Mercor) to anchor scope quickly; Proactively addressing both human eval and LLM eval tracks as distinct subsystems; Demonstrating familiarity with simulation-specific challenges (sim-to-real, compute budgets) **Why people fail:** Mechanically applying generic SD framework without engaging with domain-specific probing questions; Failing to draw any diagrams, leading to an unstructured open-ended conversation; Inability to answer unusual follow-up questions about simulation realism and compute constraints; Not adapting when interviewer pivoted away from standard system design topics **Edge cases probed:** Simulating rare/unexpected on-road events (e.g., potholes, sudden obstacles); Detecting sim-to-real artifacts — when simulated objects don't look realistic; Running thousands of simulations under tight compute constraints; Simulation acceleration strategies; Handling bugs discovered during live showcase/demo **Alternative approaches:** Generic distributed evaluation platform (no simulation focus) (Covers scalability and human/LLM eval well but misses domain-specific concerns like sim-to-real gap and compute-constrained simulation scheduling — likely to trigger heavy follow-up probing from the interviewer.); ML-system-design framing (model registry + offline batch eval) (Addresses model versioning and offline metrics well, but does not naturally surface human-in-the-loop workflows or simulation acceleration; interviewer may redirect toward operational/simulation concerns.)
Waymo · System Design · Last reported April 2026