def crawl_single(startUrl: str, htmlParser: HtmlParser) -> List[str]:
The phone-screen coding question of 2025, reported in forty-eight threads from January 2025 to September 2026, and the one most often paired with the recruiter's "be familiar with handling concurrency" line. You are given a starting URL and a helper that fetches a page and returns the URLs on it: candidates name it getURLs, getLinksFrom, get_url, getLinks, htmlParser, or "a jsoup wrapper", and several confirm it is provided, not written by you. The task, as two candidates state it: write a web crawler that only visits pages on the same hostname as the start URL; strip the #fragment from a URL before deduplicating, so http://example.com/page#a and http://example.com/page#b are one page; visit each page once; no other URL normalization is required. One interviewer refused to read or explain the prompt and let the candidate work it out; another steered a candidate who wanted to start concurrent back to a single-threaded version first.
Part 1 — single-threaded crawler: a plain BFS (one candidate: "BFS, done in seconds"). The code is run against a real small website; one report says the result is correct when it has fewer than 100 URLs, another counted 55. The tests on this page call crawl_single(startUrl, htmlParser) with a mock parser that records how many pages it fetched, and accept any iterable of the visited URLs.
The spoken follow-ups, from the reports that describe them: thread pool versus asyncio versus hand-rolled threads and their trade-offs (one candidate felt the interviewer was waiting for the word "threadpool"; another found the asyncio direction welcome); coroutine questions; how to spread the crawl over a cluster and assign URLs to machines, or scale to a million URLs; and, in a London system-design variant of the question, politeness toward the crawled servers and near-duplicate pages.
Notes: Replit was the platform in early 2025 (slow, import errors on run, five minutes lost for one candidate), CodeSignal since mid-2025. Python dominates the reports; one non-Python candidate wrote threads directly, and two 2026 replies say Python candidates now get the image-processing question as Q1 while other languages still get the crawler. One interviewer told a candidate who ran out of time before the threaded part that "most people don't get to the multi-threaded part"; that candidate was rejected. One candidate who rewrote the provided synchronous parser as an async one hit a page their parser could not decode and had no time to fix it.
The interviewer expects ThreadPoolExecutor (thread pool) as the answer. Candidates who mentioned native threads, locks, or other primitives first were steered toward thread pool — one report says the interviewer showed no reaction until 'threadpool' was mentioned. The key insight to articulate: the helper function (getURLs/getLinksFrom) is a blocking/synchronous I/O call, and Python's GIL is released during I/O, so threading genuinely helps here despite the GIL. The expected implementation is to submit each URL fetch to a ThreadPoolExecutor rather than fetching serially. Candidates who jumped straight to thread pool were rewarded; candidates who listed multiple primitives (native threads, locks, semaphores) first and arrived at thread pool late were penalized. Having code running is expected — not just describing the idea.
One report explicitly says the interviewer asked for the async version to be implemented and run, not just described. The key trap: the provided helper function (getLinksFrom/getURLs) is synchronous/blocking. If you use asyncio naively, the blocking call will block the event loop and provide no benefit. The correct approach is asyncio.to_thread() to run the blocking helper in a thread pool from within the async context, or rewriting the parser using aiohttp. One candidate used asyncio.Queue with worker coroutines and asyncio.to_thread(htmlParser.getUrls, url). Another candidate failed because they forgot a single 'await' keyword. A third failed because their async HTML parser could not decode a particular codec (0x89), and the recommendation from that candidate is to use ThreadPoolExecutor wrapped in asyncio.to_thread rather than writing a custom async parser. At least one interviewer indicated asyncio is preferable to raw multithreading for this problem; another interviewer seemed to expect thread pool more than asyncio.
Reports describe this as a verbal/design discussion, not a coding task. The expected answer involves a distributed queue (e.g., Redis) as central URL storage, multiple worker machines pulling from the queue, a host server coordinating assignment, and a persistent store (DB) for visited URLs and results. One candidate described 'Redis as central queue storage with 1 host server accessing the queue and assigning slave servers to batch process.' Another mentioned queue + Redis + DB. The interviewer in one case also asked about this in the context of production follow-up after the single-machine solution was complete.
Standard explanation — no candidate report records what the interviewer accepted here.
Rate limiting per domain (e.g., minimum delay between requests to same host); use a per-domain queue with a configurable crawl rate; respect robots.txt.
Standard explanation — no candidate report records what the interviewer accepted here.
Compute a content hash (e.g., MD5/SHA of page text) and store in a seen-hashes set; skip or deduplicate pages with matching hashes; simhash/locality-sensitive hashing for near-duplicate detection.
This was asked as a follow-up to the multi-threaded implementation. The report mentions it but gives no detail on the expected answer beyond the framing — the question is about ensuring that when a new URL is discovered (a task is ready), a worker picks it up immediately rather than waiting. This points toward using a queue (e.g., Python's queue.Queue or asyncio.Queue) so workers block on get() and process tasks as soon as they arrive, rather than batching or polling.
Forty-eight Anthropic threads, phone screens in the great majority, from January 2025 to September 2026 (two more threads in the pipeline's list are a Mercor and a Lyft round and are not counted). Through 2025 this was Coding Q1 and the usual phone screen; from early 2026, replies say Python candidates get the image-processing question as Q1 and candidates in other languages still get the crawler, and several 2026 reports met file deduplication or the LRU cache under the same "concurrency" prompt instead. Prepare all of them.
What is reported about grading: the code is run against a real small site (fewer than 100 URLs; one count of 55); the concurrent version is written and run in most reports, and one interviewer told a candidate who did not reach it that most people do not. One candidate who wrote everything correctly and compared runtimes still heard no positive comment and was not advanced; another passed and heard the next day. Platforms: Replit in early 2025 (slow, import errors), CodeSignal since mid-2025; one candidate was given Replit by the recruiter and CodeSignal by the interviewer. One interviewer would not read or explain the prompt. The provided helper downloads and parses the page and returns the links; most of them carry #fragments that must be stripped.
The phone-screen coding question of 2025, reported in forty-eight threads from January 2025 to September 2026, and the one most often paired with the recruiter's "be familiar with handling concurrency" line. You are given a starting URL and a helper that fetches a page and returns the URLs on it: candidates name it getURLs, getLinksFrom, get_url, getLinks, htmlParser, or "a jsoup wrapper", and several confirm it is provided, not written by you. The task, as two candidates state it: write a web crawler that only visits pages on the same hostname as the start URL; strip the #fragment from a URL before deduplicating, so http://example.com/page#a and http://example.com/page#b are one page; visit each page once; no other URL normalization is required. One interviewer refused to read or explain the prompt and let the candidate work it out; another steered a candidate who wanted to start concurrent back to a single-threaded version first.
Part 1 — single-threaded crawler: a plain BFS (one candidate: "BFS, done in seconds"). The code is run against a real small website; one report says the result is correct when it has fewer than 100 URLs, another counted 55. The tests on this page call crawl_single(startUrl, htmlParser) with a mock parser that records how many pages it fetched, and accept any iterable of the visited URLs.
Part 2 — concurrent crawler: the follow-up in nearly every report: make it faster. Most candidates wrote a multi-threaded version (thread pool, ThreadPoolExecutor, "multi-thread"); several wrote asyncio, two compared or benchmarked the runtimes, and one interviewer asked how a task gets processed the moment it is ready. One candidate: the concurrent version "is mainly about making visited thread-safe so the same URL is never fetched twice". The tests call crawl_threaded(startUrl, htmlParser, max_workers=16) and check that the result matches Part 1 and that every page is fetched exactly once. Browsers cannot start threads, so the tests swap in a synchronous stand-in for ThreadPoolExecutor: each submit runs its job to completion before returning. That means a lock-ordering race or a drain loop that waits only once cannot be caught here; those two bugs only show with real threads, so run it locally with a real pool as well.
The spoken follow-ups, from the reports that describe them: thread pool versus asyncio versus hand-rolled threads and their trade-offs (one candidate felt the interviewer was waiting for the word "threadpool"; another found the asyncio direction welcome); coroutine questions; how to spread the crawl over a cluster and assign URLs to machines, or scale to a million URLs; and, in a London system-design variant of the question, politeness toward the crawled servers and near-duplicate pages.
Notes: Replit was the platform in early 2025 (slow, import errors on run, five minutes lost for one candidate), CodeSignal since mid-2025. Python dominates the reports; one non-Python candidate wrote threads directly, and two 2026 replies say Python candidates now get the image-processing question as Q1 while other languages still get the crawler. One interviewer told a candidate who ran out of time before the threaded part that "most people don't get to the multi-threaded part"; that candidate was rejected. One candidate who rewrote the provided synchronous parser as an async one hit a page their parser could not decode and had no time to fix it.
Thread pool, asyncio or hand-rolled threads, and why?
How would you crawl on a cluster and assign URLs to machines, or scale to a million URLs?
Our crawler is too aggressive; how would you be polite to the servers? Many URLs point to the same or similar content; how would you handle that?
How would you make it faster?
How do you make sure a task is processed the moment it is ready?
How would you compare the two versions' performance?
Coroutine questions
| Approach | Notes |
|---|---|
| ThreadPoolExecutor | What most reports describe; wraps the provided blocking helper without rewriting it; the GIL is released during I/O. |
| asyncio with the helper in asyncio.to_thread or run_in_executor | Several candidates' route; one passed steering toward it; the helper must not be awaited directly. |
| asyncio with aiohttp and your own parser | One candidate tried it and their parser failed on a page's encoding; high risk under time pressure. |
| Cluster of crawlers | Spoken follow-up only: partition URLs by host across machines; one candidate proposed a central Redis queue handing batches to workers. |
Common mistakes: Deduplicating before stripping the fragment, or explaining the steps in the wrong order: one candidate knew the rule and was still asked repeatedly where the sanitising happens.; Running out of time before the concurrent version: one interviewer told such a candidate that most people do not get there, and that candidate was rejected.; Rewriting the provided synchronous parser as an async one: one candidate's parser could not decode a page (0x89) and the round ended there; their advice was a thread pool.; Awaiting the synchronous helper directly, so the async version is no faster: a replier's diagnosis for a candidate who saw no speedup.; Listing many concurrency primitives before the one you will use: one candidate mentioned gevent and felt nothing landed until 'threadpool'.; Not running the code, or losing time to the environment: the result is checked against a real site (fewer than 100 URLs), and one candidate spent five minutes on Replit import errors.
Interviewer hints: The recruiter's prompt in many reports: be familiar with handling concurrency using primitives or libraries of your choice; in 2026 the same prompt also led to file deduplication, the LRU cache and image processing.; One interviewer insisted on the single-threaded crawler before any concurrent version.; One interviewer would not read or explain the prompt and left it to the candidate.; One interviewer asked how a task gets processed as soon as it is ready; another asked for the trade-offs between manual threads, a thread pool and asyncio.; One candidate felt the interviewer was waiting for the word 'threadpool'; another prepared asyncio and found the conversation went that way.
What passers do: Write the single-threaded BFS, run it against the site, then make it concurrent and run that too; one candidate who did both, with a runtime comparison, still got no positive comment and was not advanced, so expect no feedback either way.; Strip the fragment before the visited check and say so in that order.; Use urlparse for hostnames and URL handling (one candidate's advice) and a thread pool or asyncio.to_thread around the provided helper rather than a rewritten parser.; One candidate who passed the phone screen and heard the next day had prepared the asyncio version deliberately and steered the follow-up toward it.
Why people fail: Not reaching the concurrent version in time (one confirmed rejection, with the interviewer's remark that most people do not).; An async rewrite of the parser that could not decode a page, with no time left.; Coding slowly or getting stuck on a detail of the BFS: one candidate's improved version was never run before time ran out.
Edge cases probed: URLs with #fragments pointing at the same page.; Links to other hostnames that must be skipped.; A URL discovered by two pages at once in the concurrent version (fetch it once).; A real site with fewer than 100 reachable pages (one count: 55).
Send them this page. It is free to read, no account needed.
What you just read — canonical solution, follow-up arc, what passing candidates actually did — exists for all 49 Anthropic questions, refreshed monthly from new candidate reports.