GPU Allocation for Mixed Large/Small Model Inference

Problem Overview

You have a shared pool of 8 GPUs serving two model types in an inference system: a large model that requires all 8 GPUs per batch request, and a small model that requires only 1 GPU per batch request. Both models have the same per-batch inference latency. Design the dispatching,…

• 5 candidate-reported follow-ups — the exact probes interviewers asked, with the trigger for each
• The rest of the problem statement — full requirements, constraints, and edge cases
• Approach and trade-offs — what passing candidates did, and the mistakes that sink people