Design an ML inference system serving 100M daily active users. Perform a back-of-envelope analysis covering: (1) memory requirements — model weights, KV cache, activations per request; (2) bandwidth requirements — memory bandwidth for inference throughput; (3) latency optimization strategies; (4) OOM (out-of-memory) mitigation scenarios. The system should be production-grade, addressing real scaling constraints at the hardware and software levels.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Cover happy path. Clarify scope. Identify the obvious bottleneck. Pick a reasonable storage and reasonable scaling approach.
All of the above plus: explicit failure handling, durability vs latency trade-offs, choose the right batching/caching strategy, articulate why.
All of the above plus: organizational concerns (rollout, migration, on-call), quantitative analysis, multi-region considerations, what could go wrong with the proposed solution at 10x scale.
Common mistakes: Failing to convert DAU to peak QPS correctly (forgetting peak multiplier); Ignoring KV cache memory in total memory estimation; Not distinguishing memory-bound vs compute-bound inference regimes; Proposing only software solutions without considering hardware (accelerator) constraints
What passers do: Structured back-of-envelope calculation presented clearly before jumping to design; Explicitly addressing both memory and bandwidth bottlenecks; Knowing specific techniques: FlashAttention, continuous batching, QAT, distillation; Connecting kernel-level optimizations (fusion, memory layout) to system-level impact
Why people fail: Jumping straight to system components without doing the capacity math; Unable to reason about GPU memory hierarchy and HBM bandwidth; Shallow treatment of OOM — only suggesting 'add more GPUs' without algorithmic solutions
Edge cases probed: OOM during peak traffic spikes; Latency tail (p99) under bursty load; Memory bandwidth bottleneck vs compute bottleneck crossover point; Handling mixed-precision (FP16/BF16/INT8) in the same serving stack
Alternative approaches: Quantization-Aware Training (QAT) for memory reduction (Reduces model memory footprint and improves throughput on quantized hardware, but requires retraining and may degrade accuracy vs post-training quantization.); Knowledge Distillation (Produces a smaller, faster student model suited for serving; trades some accuracy for significant latency/memory gains; requires training pipeline overhead.); Model/Pipeline Parallelism (Enables serving models too large for a single GPU; introduces inter-GPU communication overhead and load-balancing complexity.); Speculative Decoding (Reduces end-to-end latency for autoregressive models by using a smaller draft model; adds complexity and a secondary model to maintain.)