Prompt Playground System Design

**Common mistakes:** Treating it like a traditional whiteboard SD; neglecting UX, user flows, and client-side concerns in favor of pure backend architecture; Not proactively raising large prompt handling; only mentioning it when the interviewer explicitly asked; Putting all user tiers in a single message queue; missing tier-based priority separation; Proposing only a vector DB for prompt caching without considering the simpler and usually-sufficient KV-store exact-match alternative; Not discussing S3 offload threshold or the decision logic for when to use blob storage vs. inline DB column; Designing a context-aware chat system instead of the explicitly stateless prompt playground; Failing to mention streaming (SSE/WebSocket) for real-time token delivery from LLM **Interviewer hints:** Interviewer actively posed targeted sub-questions rather than letting candidate fully drive; did not wait for candidate to reach topics organically; One interviewer typed notes in the shared Google Doc while candidate talked — explicitly said 'focus on thinking, I'll take notes'; When candidate put all events in one queue, interviewer asked 'are all events really in the same queue?' as a leading prompt toward tier-based queue separation; Interviewer repeatedly steered back to large prompt handling ('what do you do when the prompt is very large?') until candidate addressed S3 offload and client rendering explicitly; Interviewer confirmed that the system design prompt from Anthropic explicitly calls out: scalability, reliability, UX and user flows, data design, alternative solutions, and trade-offs as evaluation axes **What passers do:** Opened with targeted requirements clarification: stateless constraint, share feature, expected prompt sizes, user types; Proactively addressed large prompt handling (S3 chunked upload, URI in DB, lazy load, virtual scrolling) without waiting to be asked; Covered both backend scalability (async queue, rate limiting, tier separation) and frontend/UX concerns (client rendering, multiple windows); Articulated clear trade-offs for every major design decision (KV vs. vector cache, S3 threshold, sync vs. async, SSE vs. WebSocket); Remained flexible and engaged when interviewer steered toward specific sub-topics rather than rigidly following a pre-planned script **Why people fail:** Spent majority of time on LLM inference backend plumbing while barely touching UX and client-side performance; Did not address the large prompt edge case until the interviewer brought it up, then gave a shallow answer; Gave a generic distributed-system design not tailored to prompt-playground-specific requirements (e.g., stateless model, large text blobs, developer UX); Became inflexible when interviewer probed a specific area, unable to pivot away from a pre-memorized structure; Missed the no-context stateless constraint and designed a context-carrying chat system **Edge cases probed:** Prompt text exceeding 10MB — upload, storage, and DOM rendering implications; Many browser windows open simultaneously, each with a large prompt — client memory and CPU pressure; GPU cluster partial failure requiring real-time dynamic rate limit reduction; Semantic similarity-based prompt cache vs. exact-match hash cache; User tier differentiation (paid vs. free) in queue and rate-limit design; Stateless / no-context constraint — must not bleed context across sessions; Choosing the right S3 offload size threshold and its effect on read latency **Alternative approaches:** Synchronous REST API (no queue) (Simpler implementation; fine when LLM latency is bounded and traffic is low. Blocks connection threads during long inference; poor scalability under spikes; tight coupling between API tier and inference tier.); KV store (exact-match hash) for prompt caching instead of vector DB (Much simpler operationally; low latency. Only cache-hits on byte-identical prompts; misses semantically equivalent but lexically different prompts. Usually sufficient unless semantic deduplication is a product goal.); Inline DB storage for all prompt text (no S3 offload) (Simpler data-access pattern with no external blob dependency. Fails badly for 10MB+ prompts: bloats row sizes, degrades query performance, increases backup costs, and may exceed RDBMS column limits.); WebSocket instead of SSE for streaming responses (WebSocket is bidirectional and supports client-initiated mid-stream messages. SSE is simpler (plain HTTP, auto-reconnect, works through HTTP/2 multiplexing) and sufficient for unidirectional token-by-token LLM output streaming.)