Batch Inference System Design

Problem Overview

Design a batch inference system for an LLM serving backend. The core API is: batch_infer(list<string> input) -> list<string> output. Multiple incoming single API requests must be aggregated server-side into batches before being dispatched to GPU workers, because one GPU can only process one batch at a time. Key constraints/goals: minimize…

• 6 candidate-reported follow-ups — the exact probes interviewers asked, with the trigger for each
• The rest of the problem statement — full requirements, constraints, and edge cases
• Approach and trade-offs — what passing candidates did, and the mistakes that sink people