AO
AceOffer
·
Back
Waymo Common Problems

Inference System Design at Scale (100M DAU)

System DesigneasyLast reported May 2026
By AceOffer · Updated May 2026 · Reported 1× across 190+ candidate reports

Understanding the Problem

Design an ML inference system serving 100M daily active users. Perform a back-of-envelope analysis covering: (1) memory requirements — model weights, KV cache, activations per request; (2) bandwidth requirements — memory bandwidth for inference throughput; (3) latency optimization strategies; (4) OOM (out-of-memory) mitigation scenarios. The system should be production-grade, addressing real scaling constraints at the hardware and software levels.

Functional Requirements

Structured requirements coming soon. For now, see the full problem statement above and the deep-dive prompts below.

Non-Functional Requirements

Latency, throughput, availability, consistency targets — being authored.

The Set Up

Defining the Core Entities

Core entities (Request, Batch, Worker, Cache, etc.) — being authored.

The API

POST /endpoint → describe request shape GET /endpoint → describe response shape (API spec being authored)

High-Level Design

Component diagram + walkthrough mapping each functional requirement to a system flow — being authored.

Potential Deep Dives

These are the directions the interviewer is likely to push you. Each one has multiple valid solutions at different quality tiers.

1)How would you handle OOM situations in production at this scale? (when: Candidate completes basic capacity estimation)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

2)How do you optimize kernel-level performance for inference — memory layout, kernel fusion? (when: Candidate mentions latency requirements)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

3)Walk through QAT vs post-training quantization trade-offs and when you'd choose each. (when: After system architecture is laid out)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

4)How would you evaluate a distilled or quantized model in production? What metrics and eval frameworks? (when: Discussion of model quality)

Bad
Naive approach with serious trade-off — being authored.
Good
Solid baseline with reasonable trade-offs — being authored.
Great
Production-grade approach with explicit trade-off rationale — being authored.

What is Expected at Each Level?

L4 / Mid-level
Cover happy path. Clarify scope. Identify the obvious bottleneck. Pick a reasonable storage and reasonable scaling approach.
L5 / SeniorTarget
All of the above plus: explicit failure handling, durability vs latency trade-offs, choose the right batching/caching strategy, articulate why.
L6 / Staff+
All of the above plus: organizational concerns (rollout, migration, on-call), quantitative analysis, multi-region considerations, what could go wrong with the proposed solution at 10x scale.

Insider Notes

**Common mistakes:** Failing to convert DAU to peak QPS correctly (forgetting peak multiplier); Ignoring KV cache memory in total memory estimation; Not distinguishing memory-bound vs compute-bound inference regimes; Proposing only software solutions without considering hardware (accelerator) constraints **What passers do:** Structured back-of-envelope calculation presented clearly before jumping to design; Explicitly addressing both memory and bandwidth bottlenecks; Knowing specific techniques: FlashAttention, continuous batching, QAT, distillation; Connecting kernel-level optimizations (fusion, memory layout) to system-level impact **Why people fail:** Jumping straight to system components without doing the capacity math; Unable to reason about GPU memory hierarchy and HBM bandwidth; Shallow treatment of OOM — only suggesting 'add more GPUs' without algorithmic solutions **Edge cases probed:** OOM during peak traffic spikes; Latency tail (p99) under bursty load; Memory bandwidth bottleneck vs compute bottleneck crossover point; Handling mixed-precision (FP16/BF16/INT8) in the same serving stack **Alternative approaches:** Quantization-Aware Training (QAT) for memory reduction (Reduces model memory footprint and improves throughput on quantized hardware, but requires retraining and may degrade accuracy vs post-training quantization.); Knowledge Distillation (Produces a smaller, faster student model suited for serving; trades some accuracy for significant latency/memory gains; requires training pipeline overhead.); Model/Pipeline Parallelism (Enables serving models too large for a single GPU; introduces inter-GPU communication overhead and load-balancing complexity.); Speculative Decoding (Reduces end-to-end latency for autoregressive models by using a smaller draft model; adds complexity and a secondary model to maintain.)
Waymo · System Design · Last reported May 2026