Inference API System Design

Problem Overview

Design an HTTP API service that exposes an LLM (large language model) inference/sampling function to users. Core function given: A black-box model sampling function that takes a list of strings (batch size 1–100) and returns a list of strings (same length). For a batch of any size from 1 to…

• 18 candidate-reported follow-ups — the exact probes interviewers asked, with the trigger for each
• The rest of the problem statement — full requirements, constraints, and edge cases
• Approach and trade-offs — what passing candidates did, and the mistakes that sink people