Design a distributed, multi-tenant CI/CD system that schedules and executes user-defined workflows in response to git push events. The system receives push information via API calls from an internal service containing the repository ID and current commit hash. Workflows are defined as a sequence of sequential jobs in a single YAML file stored at a static location per repository. Users should be able to view job output and status in real-time as jobs run. The system must be fault-tolerant, horizontally scalable, and support exactly-once execution semantics. Key areas to address: event ingestion, workflow config parsing, job scheduling, worker execution model, real-time log streaming, failure recovery, and multi-tenant isolation.
Canonical prompt (verbatim, observed most frequently): 'Design a multi-tenant CI/CD system which schedules and executes user-defined workflows in response to git pushes. The system receives information about pushes via API calls from an internal service which contain the repository id and the current state of the repository (commit hash). Workflows are a sequence of jobs which are defined within a single YAML file in a static location for each repository. Users should be able to view the output and status of jobs as they are running.'
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Naive approach with serious trade-off — being authored.
Solid baseline with reasonable trade-offs — being authored.
Production-grade approach with explicit trade-off rationale — being authored.
Cover happy path. Clarify scope. Identify the obvious bottleneck. Pick a reasonable storage and reasonable scaling approach.
All of the above plus: explicit failure handling, durability vs latency trade-offs, choose the right batching/caching strategy, articulate why.
All of the above plus: organizational concerns (rollout, migration, on-call), quantitative analysis, multi-region considerations, what could go wrong with the proposed solution at 10x scale.
Common mistakes: Designing a stateful scheduler that holds all workflow state in memory — fails to scale horizontally; Skipping scope clarification (containers vs shell scripts, linear vs DAG) and designing for wrong scope; Omitting real-time log streaming component entirely — frequently a key deep-dive area; Not addressing exactly-once execution or idempotency at all; Not handling worker failure / job reconciliation / stuck RUNNING state; Spending too long on non-core components (frontend, auth) at expense of scheduler/worker depth; Describing broad overview of all CI/CD concepts without depth on any single component; Conflating CI (build pipeline) with CD (deployment pipeline) without clarifying which is in scope; Using k8s-native scheduling when interviewer scoped to simple shell script execution; Not addressing multi-tenant isolation (fairness, quota enforcement)
Interviewer hints: 'Think of this as a job scheduler problem' — given when candidates focus too broadly on DevOps tooling; 'Assume jobs are just shell scripts, no need to worry about k8s or container orchestration' — given when candidate over-engineers container layer; 'How would you keep your scheduler stateless?' — hint toward CDC/queue-driven design; 'What happens if the worker crashes after it starts executing but before it finishes?' — prompting fault tolerance discussion; 'Focus on how the next job gets triggered after the previous one completes' — nudging toward CDC/event-driven scheduler; 'Think about how you'd handle the stuck running state' — explicit hint about reconciler pattern; Interviewer corrected scope when candidate discussed unneeded components, redirecting to core scheduler design
What passers do: Immediately reframing the problem as a 'job scheduler' and driving the design around that mental model; Proactively clarifying scope: sequential vs DAG, containers vs scripts, scale targets — before designing; Proposing stateless, CDC-driven scheduler that enables horizontal scaling of all components; Explicitly addressing exactly-once execution with idempotency keys; Designing real-time log streaming (WebSocket/SSE from worker to user) without being prompted; Describing stuck-job reconciler with TTL-based detection; Structuring answer as: event ingestion → config parsing → job state machine → worker execution → log streaming → failure recovery → multi-tenancy; Candidate's work background directly overlapped with CI/CD infrastructure, enabling deep technical conversation
Why people fail: Broad high-level overview without depth on any layer; interviewer has to prompt for every detail; Missing log streaming component — revealed as gap during deep-dive; Designing stateful scheduler with in-memory state, unable to explain horizontal scaling; Not clarifying scope → designing for wrong assumptions (DAG when linear is expected, or vice versa); Giving generic 'add a queue here, add a cache there' without explaining data flow or failure modes; Running out of time before reaching critical components (fault tolerance, multi-tenancy); Ignoring interviewer redirections and continuing to over-index on non-essential components
Edge cases probed: Jobs running in minimal Docker containers with no HTTP server or RPC capability — how to retrieve logs and detect completion; Concurrent git pushes to the same repository triggering multiple workflow runs simultaneously; Job stuck indefinitely in RUNNING state (worker dies after starting but before completing); Two sequential jobs require different compilers/dependencies — how are their Docker image caches handled independently; Multi-part artifact upload to S3 fails mid-upload — how to clean up and retry; Idempotency: same push event delivered twice to Push Handler API; YAML workflow file is missing or malformed at the static location; Very large number of jobs in a single workflow (deep sequential chain); Tenant submits burst of git pushes — fair scheduling under load
Alternative approaches: DAG-based scheduler with parallel job execution (Handles non-linear workflows with parallel branches; required if workflow is a DAG rather than linear sequence. Significantly more complex dependency tracking. Most interviewers scope problem to linear/sequential only — confirm before going here.); Stateful in-memory scheduler (Simpler to implement initially — scheduler loads full workflow state into memory and steps through jobs. Cannot horizontally scale easily; single point of failure; not recommended unless explicitly scoped to small scale.); Worker self-chaining (worker queries next job) (After completing a job, worker queries DB for next job in sequence and enqueues it directly, removing need for CDC. Simpler but couples worker to scheduler logic; less clean separation of concerns.); Polling-based status vs. CDC/push (Scheduler polls DB periodically for completed jobs instead of using CDC. Simpler to implement; adds latency between job completion and next job start; less efficient at scale.); Kubernetes-native job scheduling (Leverage k8s Jobs and CronJobs for orchestration. Appropriate for container-based workloads; significant operational complexity; interviewer often says to scope away from k8s unless it's a GitHub Actions variant.)
What you just read — canonical solution, follow-up arc, what passing candidates actually did — exists for all 100 OpenAI questions, refreshed monthly from new candidate reports.