class GPT:
Before the round, the recruiter's prompt names it ML Debug: a transformer model written in PyTorch whose bugs you find and fix.
In the round you get a small GPT-style causal language model, a training loop that overfits one sentence, and a sample() function; comments mark the regions that contain bugs. When the fixes are right, the loss converges and the sample produces text. Sixty minutes, and a follow-up comes once the model works.
This editor has no PyTorch, so the model here is a numpy port of the forward pass, and the tests check each fix directly instead of training.
Part 1 — Fix the bugs
Find and fix the four planted bugs so the model is a correct next-token language model: the loss targets, the positional-embedding initialization, the causal mask, and a typo in the attention output. The tests call the static GPT.next_token_loss(logits, idx) on hand-built logits, and compare attention(x, p) with a hand computation that uses the given softmax(x, axis=-1). Fix each inside its marked region and keep the interface as given. The positional table is meant to stay learnable and the bug is its initial value; the test accepts any reasonable init (a small random one like the token embedding's, a sin/cos table, or zeros) and fails only the given unit-variance one.
Reported variations
nn.Linear dimensions, in one), and a missing gradient step (loss.backward(), in one). One counted five bugs, and two more said four or five.generate() and how to improve it, then asked for the cache.The model's constructor. GPT(vocab_size, block_size, n_embd=16, n_head=2, n_layer=1, seed=0) is given; the tests build small models with these keyword settings (n_embd, n_head, n_layer, seed), so keep the signature as it is.
Standard explanation — no candidate report records what the interviewer accepted here.
Without a cache every step re-runs attention over the whole prefix: O(t²·d) at length t, so generating n tokens from a prompt of length P costs O(n·(P+n)²·d). With a KV cache each step is one query against the cached keys, O(t·d), so the total drops to O((P+n)²·d); the price is O(layers·length·d) memory for the cache. In the one report that describes this order, the interviewer then asked for the cache.
Attention appends the new keys and values to its layer's cache and attends over everything cached. Positions start at the cache length. A query at absolute position p may see keys 0..p; when every step adds a single token that means no mask at all, which is why one report's rule was to apply the causal mask only while the cache is empty (checked before appending). Generate prefills the prompt once, then feeds one token per step. Verify by comparing the output with the uncached generate; reports say that comparison is the whole check.
Replace the vocabulary head with a classification head. One interviewer preferred taking the mean over the token representations before the logits. The prediction and the loss both change: one logit vector per sequence, and a cross-entropy over the class labels. Run the provided test.
Standard explanation — no candidate report records what the interviewer accepted here.
One report had this longer form: many ways to change the model, to be discussed with the interviewer. Name the pooling options (mean over tokens, the last token, a dedicated CLS token), say which one you chose and why, then train and report an evaluation metric.
Shape of the round. Sixty minutes, one question, PyTorch in CoderPad, reported by 35 candidates from January 2025 to September 2026, in phone screens and onsites, mostly for research-engineer and MLE roles. The recruiter names the round in advance ("ML Debug"), and the interviewer knows where the bugs are and will say where and how many. The bugs are small errors, not unusual architecture: one candidate estimated fifteen minutes for someone who knows the code. Then comes the follow-up: a KV cache (fourteen reports) or a classifier (eight).
The bar. Candidates who passed describe knowing a minimal GPT cold: one read every post, hand-wrote the main transformer structure, watched Karpathy's free course, and had GPT plant fake bugs to practise on; another says knowing nanoGPT is basically enough, and a third that one practice run of the simplest GPT plus KV cache plus decoding covers it. Two candidates who finished were rejected and pointed to time: one had no time to test the cache, and one who finished both parts right at the end guessed that the pace cost them.
Recency. Still reported in August and September 2026.
About this page. The editor has no PyTorch, so the model is a numpy port. The bugs, their symptoms and the cache interface follow the reports; the exact file in your interview will differ, and there the train and sample harness, not these tests, decides.
Before the round, the recruiter's prompt names it ML Debug: a transformer model written in PyTorch whose bugs you find and fix.
In the round you get a small GPT-style causal language model, a training loop that overfits one sentence, and a sample() function; comments mark the regions that contain bugs. When the fixes are right, the loss converges and the sample produces text. Sixty minutes, and a follow-up comes once the model works.
This editor has no PyTorch, so the model here is a numpy port of the forward pass, and the tests check each fix directly instead of training.
Part 1 — Fix the bugs
Find and fix the four planted bugs so the model is a correct next-token language model: the loss targets, the positional-embedding initialization, the causal mask, and a typo in the attention output. The tests call the static GPT.next_token_loss(logits, idx) on hand-built logits, and compare attention(x, p) with a hand computation that uses the given softmax(x, axis=-1). Fix each inside its marked region and keep the interface as given. The positional table is meant to stay learnable and the bug is its initial value; the test accepts any reasonable init (a small random one like the token embedding's, a sin/cos table, or zeros) and fails only the given unit-variance one.
Part 2 — KV cache
Make generation cheaper with a KV cache. The KV and Cache classes are given. Change your model so that forward(idx, cache=None) takes an optional cache: attention appends each layer's new keys and values and attends over everything cached, and positions start at len(cache). Then write generate_cached(model, idx, max_new_tokens: int), which prefills the prompt once with model.forward(prompt, cache) and then feeds one token per step, and must return exactly what the given generate(idx, max_new_tokens: int) method returns. The prompt plus the new tokens must fit in block_size. (The function name and signature are this page's; sequences in a batch share positions.)
Reported variations
nn.Linear dimensions, in one), and a missing gradient step (loss.backward(), in one). One counted five bugs, and two more said four or five.generate() and how to improve it, then asked for the cache.What is the time complexity of generate(), and how would you make it faster?
Implement a KV cache: the KV and Cache classes are given; update attention and the positional offset and write your own generate.
Turn the model into a classifier by changing the last layer; make the given test pass.
| Approach | Notes |
|---|---|
| Fix inside the marked regions | The interviewer knows where each bug is and can follow a fix made there; the provided train and sample functions then show whether the loss converges. |
| Compare against a memorised nanoGPT | Fast when the architecture is familiar; one candidate says knowing nanoGPT is basically enough. |
| Masking rule for the cache | One report's rule: apply the causal mask only while the cache is empty, checked before appending. |
Common mistakes: Leaving the loss unshifted because you are thinking masked-LM rather than next-token prediction.; Replacing the learnable positional embedding with sin/cos instead of fixing its initial value.; Reading the driver code as well as the marked regions; the candidate who did said it did not help.; Passing over the typo in the attention output: one candidate said it took extra time to spot.; Running the code too often: CoderPad ran PyTorch very slowly for one candidate.; Finishing the cache with no time left to run it.
Interviewer hints: The interviewer knows where the bugs are and will say where and how many, but each bug has several valid fixes, so if you fix one a different way they may not be sure how to help. For the cache, the check is that its output matches the version without it. For the classifier, one interviewer preferred mean pooling before the logits.
What passers do: One candidate who passed the technical rounds read every post about the round, hand-wrote the main transformer structure, practised on bugs planted by an LLM, and then found the four bugs quickly. Another says knowing nanoGPT is basically enough, and a third that one practice run of the simplest GPT with a KV cache and decoding covers the follow-up.
Send them this page. It is free to read, no account needed.
What you just read — canonical solution, follow-up arc, what passing candidates actually did — exists for all 91 OpenAI questions, refreshed monthly from new candidate reports.