Transformer Architecture and Attention Optimization

Problem Overview

Explain the transformer architecture in detail — describe each layer (encoder/decoder) and its role, the Query/Key/Value (Q/K/V) self-attention mechanism and how attention scores are computed (scaled dot-product attention), computational bottlenecks (when is attention the bottleneck vs. feed-forward layers?), and techniques to…

• 5 candidate-reported follow-ups — the exact probes interviewers asked, with the trigger for each
• The rest of the problem statement — full requirements, constraints, and edge cases
• Approach and trade-offs — what passing candidates did, and the mistakes that sink people