Not stated in the reports.
Not stated: no report names the framework or the editor. The reports mention multinomial sampling on logits and model log-probs; the code on this page uses PyTorch as our illustration.
A simple GRPO training script (one report's words) with bugs in it. What the reports show of it: logits passed to multinomial sampling, a normalized advantage divided by a standard deviation, a ratio built from model_logprob and old_logprob, and ratio clipping. No report gives its length or structure. Candidates call the round "RL Fundamentals"; "technical deep dive" is our catalog label.
The bugs fixed: one report names two (a softmax before sampling, an epsilon on the advantage std); another says there are three
Answers to the RL follow-ups: the log-difference ratio, why and when the ratio is clipped, and what clipping does
An answer to whether the ratio in this code should in theory be 1, and why; then the ratio printed (in one report it was not 1) and the cause debugged
Asked in the round candidates call "RL Fundamentals". You get a simple GRPO (Group Relative Policy Optimization) training script, debug it, then answer follow-up questions on RL basics. One candidate (October 2025) first hit a nan error and found two bugs: the code passed raw logits to multinomial sampling without a softmax, and the normalized advantage divided by the standard deviation without an epsilon. Another candidate (March 2026) says there were three bugs, all easy to spot for anyone who knows GRPO, but does not name them; no report names the third.
After the fixes the first candidate was asked whether the model still trains if the ratio is computed as model_logprob - old_logprob, why the ratio is clipped, when it gets clipped and what clipping does, and then whether the ratio in this code should in theory be 1, and why; printed, it was not 1, and they had to debug why. The second candidate called the follow-ups basic except one: finding why the run is not strictly on-policy, that is, why the ratio is not always 1, which took them a long time.
The version may differ: an August 2026 poster says the GRPO debugging question they got was different from, and more complicated than, the one posted on the forum.
No report describes a model answer; this is our approach. (1) Find the multinomial call and sample from softmax(logits), or from Categorical(logits=logits). (2) Find the advantage normalization and add an epsilon to the std; also make sure every group has more than one sample, because torch's default std is the unbiased one (correction=1), which is NaN for a single value whatever the epsilon (or use correction=0). (3) No report names the third bug, so check the rest of the loss against the GRPO objective: the ratio should be exp(new_logprob − old_logprob), the clip should act on that ratio, and any KL term should compare with the reference policy. Then be ready to explain why the printed ratio is not 1 in nominally on-policy code (see the follow-ups).
# Before: action = torch.multinomial(logits, 1) # After: probs = F.softmax(logits, dim=-1) action = torch.multinomial(probs, 1) # Cleaner: action = torch.distributions.Categorical(logits=logits).sample()
# Before: advantage = (returns - returns.mean()) / returns.std() # After: advantage = (returns - returns.mean()) / (returns.std(correction=0) + 1e-8)
# Before: ratio = new_logprob - old_logprob # the log-ratio, not the ratio # After: ratio = torch.exp(new_logprob - old_logprob) # At the first update on a fresh batch the ratio should be exactly 1: # assert torch.allclose(ratio, torch.ones_like(ratio))
If the ratio were computed as model_logprob − old_logprob (the log-difference rather than its exponential), could the model still train? Why or why not?
Why do we clip the ratio in PPO/GRPO, and when does a ratio actually get clipped?
What happens to training when the ratio gets clipped?
In this code, should the ratio in theory always be 1? Why? Print it: it is not 1. Debug why.
October 2025: a nan error first, then two named bugs (no softmax before multinomial sampling; no epsilon on the advantage std), then RL follow-ups
March 2026: three bugs, not named; the hard follow-up was why the ratio is not always 1
Passing logits straight to sampling. In PyTorch (our illustration), torch.multinomial takes weights that need not sum to 1: a negative logit raises an error (its message mentions inf, nan or element < 0). It runs only when every weight in the row is finite and non-negative and the row sum is above zero, and then it samples in proportion to the raw logits, not the softmax policy.
Normalizing advantages without an epsilon. A group whose rewards are exactly equal gives 0/0 = NaN. Torch's default std is the unbiased one (correction=1), so a one-sample group is NaN too, epsilon or not: keep groups larger than one or use correction=0.
Thinking near-equal rewards blow up the advantages. They do not: (r − mean)/std is scale-invariant, so rewards [1, 1, 1 + 1e-6] normalize to about [−0.71, −0.71, 1.41] with the population std, or [−0.58, −0.58, 1.15] with torch's default unbiased std, and no entry can exceed √(n−1) in size (population std). The real effect is that tiny differences, even float noise, become unit-size advantages; an epsilon damps only differences much smaller than itself (with eps = 1e-8, rewards [5, 5, 5 + 1e-12] give advantages below 1e-4).
Computing the ratio as the log-difference instead of its exponential: training still improves, but near d = 0 only above-average samples get a gradient (below-average ones only once d passes 1 − ε) and the clip no longer acts as a trust region (see the first follow-up).
Assuming the ratio is 1 without printing it. At the first update on a fresh batch it is exactly 1 only if the old and new log-probs come from the same computation; the on-policy follow-up is about exactly this.
Answering 'why do we clip?' with only 'to prevent large updates'. Clipping is a trust-region heuristic that removes the incentive to move further once the ratio leaves [1−ε, 1+ε] in the direction the advantage favours. It does not bound KL(new ‖ old), which is why PPO has an adaptive-KL-penalty variant and the standard GRPO objective adds a separate KL term to a reference policy (some implementations drop it, so check whether this code has one). Whether a sample's gradient is zero depends on the advantage sign and which branch of the min() is active, not only on whether the ratio is outside the band.
Candidates call this round "RL Fundamentals", and recruiters match candidates to it under that name. Two first-hand reports describe it, from October 2025 and March 2026. The October candidate had not seen it written up before; several later threads ask to trade for it.
The two candidates described the difficulty differently. The March 2026 candidate called the three bugs straightforward, spotted at a glance by anyone who knows GRPO, and the follow-ups basic, except one: working out why the run is not strictly on-policy, which took them a long time. The October candidate described the same kind of follow-ups as a run of questions in their blind spots and expected to fail. Another candidate was matched to this round although they had told the recruiter they had no RL background; a commenter's advice was that without that background, even knowing the question, you are unlikely to get through the follow-up discussion, and to ask the recruiter to switch topics.
In the October candidate's loop this round sat alongside a coding round (find the mode across distributed workers, with a median follow-up) and a coding-and-design round (a weighted data batcher with checkpointing).
What on this page is ours: no report names the third bug. The third entry in the bug catalogue (the ratio computed as the raw log-difference) is our guess, based only on the interviewer asking about that formula; since it was asked after the fixes, as a what-if, the code probably did not contain it. The cause of the ratio drift is not reported either. The follow-up answers, the playbook and the prerequisites are our analysis from standard PPO and GRPO results, not an interviewer's rubric.
Send them this page. It is free to read, no account needed.
What you just read — canonical solution, follow-up arc, what passing candidates actually did — exists for all 49 Anthropic questions, refreshed monthly from new candidate reports.