Not stated in the source reports. Both reports describe a single dedicated round; candidates refer to it by name as “RL Fundamentals.”
Not stated in the source reports — the script is PyTorch, so assume a shared editor or notebook, but treat this as an assumption rather than a reported fact.
Candidates call this round “RL Fundamentals” — that is the name the recruiter gives it and the term every report uses; “technical deep dive” is our catalog label, not theirs. One report frames the bar as needing to be fluent in the whole GRPO training pipeline, not just the loss math. A complete-but-buggy GRPO training script (~100–200 lines): rollout sampling, group-relative advantage normalization, PPO-style ratio + clip loss, and a small RL environment loop. Whether it runs at all on the first try depends on the data: sampling raises a RuntimeError as soon as a sampled row contains a negative logit, and only survives if every row happens to be non-negative with a positive finite sum — in which case it runs but silently samples the wrong distribution. That conditional is the trap. Two bugs are numerical (you find them by running the script), the third is algorithmic (you find it by reading the ratio formula). Reports indicate the rough shape is: a Policy module → a rollout(env, n_steps) function that calls multinomial → a compute_advantage(returns) helper → a PPO/GRPO clip-objective loss → a training loop with mini-epochs over each batch. (The component breakdown is inferred from the canonical bug list — not verbatim from a single report.)
All three bugs identified and fixed
Code that runs without NaN errors and produces a non-trivial training signal
Verbal explanation of each bug's root cause + fix
Answers to RL-theory follow-ups about ratio clipping and on-policy/off-policy drift
A diagnostic — typically print(ratio.mean()) — that surfaces the on-policy follow-up cleanly
Asked in the round Anthropic and candidates both call "RL Fundamentals". Given a simplified but complete GRPO (Group Relative Policy Optimization) training script, identify and fix all bugs in the implementation. Reported bugs span at least three issues: (1) raw logits are passed directly to multinomial sampling without first applying softmax — which raises a RuntimeError on any negative logit and otherwise samples the wrong distribution silently (the reported NaN comes from the next bug, not this one); (2) the normalized-advantage calculation divides by the standard deviation without adding a numerical-stability epsilon, risking division-by-zero NaNs; (3) a third bug discoverable by anyone familiar with the algorithm — generic to PPO-style importance ratios rather than GRPO-specific (exact nature not disclosed in reports, but likely relates to incorrect ratio computation, e.g., computing ratio as model_logprob − old_logprob instead of exp(model_logprob − old_logprob)). After the bugs are fixed, the interviewer asks a series of follow-up questions about training dynamics and RL theory.
Read the code top-to-bottom and trace the data flow: (1) locate the multinomial call and prepend a softmax over logits; (2) find the advantage normalization and add an epsilon — std(correction=0) + eps, since a plain std + eps still returns NaN for a singleton group under PyTorch's default unbiased correction, so either use correction=0 or guarantee group size > 1; (3) verify the importance-sampling ratio is computed as exp(new_logprob − old_logprob) (not as a raw log-difference). Then reason about when the ratio can drift from 1 even in nominally on-policy GRPO: if multiple gradient steps are taken on the same rollout batch (PPO-style mini-epochs), the current policy diverges from the sampling policy within the batch, making the ratio ≠ 1.
ratio.mean() is the diagnostic that surfaces the on-policy follow-up: on the first mini-epoch step over a rollout batch it should print ≈1.0, and on subsequent mini-epoch steps the empirical mean and the ratio distribution shift even though E_old[π_new/π_old] is still 1 in expectation. (Empirical KL and clipped-fraction are the more sensitive diagnostics; ratio.mean() is what the interviewer's prompt explicitly references.)ratio.mean() still isn't exactly 1 on a nominally on-policy step. The expected answer is the mini-epoch loop: the policy updates between gradient steps within the same rollout batch, so by step k > 1 the sampling policy ≠ the current policy. Having this answer staged turns a stretch follow-up into a planted close.# Before: action = torch.multinomial(logits, 1) # After: probs = F.softmax(logits, dim=-1) action = torch.multinomial(probs, 1) # Cleaner: action = torch.distributions.Categorical(logits=logits).sample()
correction=0.# Before: advantage = (returns - returns.mean()) / returns.std() # After: advantage = (returns - returns.mean()) / (returns.std(correction=0) + 1e-8)
min() is active — but in aggregate the update direction is wrong. PROVENANCE: the two reports behind this question name the softmax and epsilon bugs explicitly; the second report states there are three bugs but does not name the third. This entry is therefore RECONSTRUCTED, not verbatim — we present it because the interviewer's own follow-up in the first report interrogates exactly this formula (‘if ratio = model_logprob − old_logprob, can it still train?’), which is strong circumstantial evidence. Treat bugs #1 and #2 as confirmed and #3 as the most likely third.exp of the log-prob difference. Verify by printing ratio.mean() — on the first mini-epoch step it should print exactly 1.0 (since π_new == π_old at that point). On later mini-epoch steps the empirical mean may drift, but E_old[ratio] = 1 still holds in expectation; the more sensitive drift diagnostic is the empirical KL or clipped-fraction.# Before: ratio = new_logprob - old_logprob # WRONG — this is the log-ratio, not the ratio # After: ratio = torch.exp(new_logprob - old_logprob) # Sanity check: on the first mini-epoch step π_new == π_old, so ratio == 1 exactly: # assert torch.allclose(ratio, torch.ones_like(ratio)) when step == 0
If the ratio were computed as model_logprob − old_logprob (an additive log-difference rather than the exponentiated ratio), could the model still train? Why or why not?
Why do we clip the importance-sampling ratio in PPO/GRPO? What does clipping accomplish, and when is a ratio actually clipped?
In the provided code the ratio should theoretically always be 1 (on-policy). But when you print it at runtime it is not 1. Debug why.
What are the practical training-stability consequences when the ratio is frequently clipped versus rarely clipped?
| Approach | Trade-offs |
|---|---|
| Unit-test each component independently | Faster isolation of NaN sources via shape/value assertions on logits, probabilities, advantages, and ratios, but requires writing additional test harness code during the interview. |
| Run and inspect printed values first | Quickly reveals symptoms (e.g., ratio printed ≠ 1) but may miss root-cause logic errors without careful code reading. |
Base: debug NaN errors in GRPO code (softmax + epsilon fixes)
With three explicit bugs to find
With deep follow-up on ratio clipping theory
With follow-up: why ratio is not strictly 1 at runtime (not strictly on-policy)
Forgetting to apply softmax before multinomial sampling — torch.multinomial raises a RuntimeError on any negative logit, and silently samples the WRONG distribution if logits are all non-negative (it does NOT NaN-explode). Easy to miss if your test env happens to produce non-negative logits early.
Computing advantage as (returns - baseline) / std without an epsilon — division by zero on early rollouts where all returns are similar (sparse-reward envs, early training). Note: in GRPO, the group must have size > 1 — PyTorch's default unbiased=True std of a size-1 group is itself NaN, which epsilon won't save; use correction=0 or guarantee group size > 1.
Computing the importance-sampling ratio as a log-difference instead of exp(log-difference) — the model still 'trains' but the surrogate objective is wrong, the clip becomes meaningless, and updates are biased. The interviewer's first follow-up question is specifically about this: 'if ratio = model_logprob − old_logprob, could the model still train? Why not?'
Assuming the script is on-policy and never printing the ratio — misses the entire mini-epoch follow-up. The whole point of the third follow-up is that 'on-policy' GRPO with multiple gradient steps per rollout batch ISN'T strictly on-policy after step 1.
Spending too much time reading top-to-bottom before running — the first run reveals 2/3 bugs immediately. Read for ~5 minutes for context, then run.
Answering 'why do we clip?' with only 'to prevent large updates' — the great answer goes further: clipping is a trust-region HEURISTIC that removes the policy-improvement incentive when the per-action ratio leaves [1−ε, 1+ε]. It does NOT analytically bound KL(new || old) — that's why PPO has a separate adaptive-KL-penalty variant and why GRPO/DeepSeek-Math add an explicit KL term to a reference policy on top of the clip. Whether the gradient drops to zero depends on the advantage sign and which branch of the min() is active per sample, not just on whether the ratio is in the clip band — so 'clipping zeros the gradient' is too strong a generalization.
Finds the two numerical bugs (softmax, epsilon) within 30 minutes. Can explain that softmax converts logits to probabilities and that division-by-zero produces NaN, but doesn't yet connect to the surrogate objective. Misses the ratio bug or finds it only after a strong hint. Answers the clipping follow-up correctly at the textbook level (ε=0.2, bounds policy change) but doesn't ground it in the trust-region intuition.
Finds all three bugs, including the ratio formula, within the time budget. Articulates the importance-sampling ratio formula correctly the first time. Answers ratio-clipping and on-policy/off-policy follow-ups with the right intuition — knows that the ratio is clipped when new_logprob − old_logprob falls outside [log(1−ε), log(1+ε)] (asymmetric bounds in log-space), and that clipping removes the incentive only in the sign-dependent direction — flat for A>0 when r>1+ε and for A<0 when r<1−ε, while the unclipped branch stays active in the opposite direction — rather than imposing a hard constraint. Reasons correctly about training-stability consequences. Gets the on-policy-drift follow-up via the mini-epoch path. Instruments the code with print(ratio.mean()) proactively while noting that under the old policy E[ratio] = 1 holds in expectation regardless of mini-epoch updates — what actually shifts is the empirical KL and clipped-fraction.
All of IC5 plus: derives why the policy drifts during mini-epochs without being asked, discusses the bias-variance trade-off introduced by clipping, and is precise that clipping is a trust-region HEURISTIC — it does not analytically bound KL(new || old). Notes that PPO ships a separate adaptive-KL-penalty variant precisely because clipping alone does not constrain KL, and that GRPO/DeepSeek-Math add an explicit KL term to a reference policy on top of the clip. Connects to advanced topics: why GRPO uses group-relative baselines instead of a learned value function (eliminates the critic; the group baseline reduces variance versus NO baseline — it is not guaranteed to beat a well-fit learned critic with GAE), the bias-variance trade-off vs. GAE. Suggests concrete diagnostics for production training runs: monitor clipped-fraction per batch, watch the empirical KL — noting samples drawn from π_old directly estimate KL(old ‖ new); KL(new ‖ old) needs full distributions or importance weighting — and KL to the reference policy, alert on either above a threshold.
Candidates call this round “RL Fundamentals.” It is the term used in every thread that mentions it, and the name recruiters match candidates into — if you were told you have an RL Fundamentals round, this is the question. Two independent reports describe it: October 2025 and March 2026. The first reporter noted it had never been written up before, which is why so many 2026 threads are people actively trading for it.
The structure of the bug list is intentional: the first two bugs are “do you know PyTorch,” the third is “do you understand GRPO,” and the on-policy mini-epoch follow-up is “do you understand training systems.” The debugging is not what decides the round — both reports agree the bugs are quick. The March 2026 reporter called all three straightforward — spotted at a glance by anyone who knows GRPO — and the follow-ups basic, with only one hard part: working out why the run is not strictly on-policy, which took them a long time.
Read that “basic” carefully — it is conditional on RL background, and the two reports diverge exactly there. The October reporter, who expected to fail, described the same follow-ups as a barrage of interrogation in their blind spots. The corpus makes the stakes explicit: one candidate was matched to this round by a recruiter despite having told them they had no RL background, and an experienced commenter's advice was blunt: even if you know the question, you are unlikely to survive the follow-up discussion starting from scratch — sincerely, ask your recruiter to switch topics. So: if you have RL experience, this round is very winnable and the follow-ups really are basics. If you do not, memorizing the three bugs will not save you, and asking the recruiter for a different topic is a legitimate move that candidates do make.
Candidates consistently report that printing ratio.mean() during training is what unblocked the on-policy follow-up; practice that instinct. In the same loops, this round is reported alongside coding & design (a weighted data batcher with checkpointing), general coding (distributed worker find-mode/median), NN fundamentals, LLM prompting & engineering (55 min), and an ML configuration system round that candidates repeatedly note has almost no public information.
PROVENANCE OF THE ANSWERS: the QUESTIONS on this page are reported — every follow-up above appears in a candidate's own write-up. The model answers ("how a great answer goes") are NOT: this round took the pipeline's light extraction path, so no passing_patterns or expected_approach were ever scraped for it. The theory worked out below — the REINFORCE degeneration, the asymmetric log-space clip band, the KL caveats — is derived from standard PPO/GRPO results and independently reviewed, not transcribed from an interview. Trust it as well-checked reasoning, not as an interviewer's rubric.
What you just read — canonical solution, follow-up arc, what passing candidates actually did — exists for all 51 Anthropic questions, refreshed monthly from new candidate reports.