Given a starting URL and a provided helper function (variously named getURLs, getLinksFrom, getPageLinks, or a jsoup wrapper) that takes a URL and returns a list of URLs found on that page, implement a web crawler that visits all reachable pages within the same hostname as the starting URL. Requirements: (1) Only crawl pages whose hostname matches the starting URL's hostname. (2) Handle URL fragments: strip the '#...' portion from URLs before deduplication (e.g., 'http://example.com/page#section' → 'http://example.com/page'). (3) Avoid visiting the same URL twice. (4) URL normalization beyond fragment removal is NOT required. The solution must be run against a real small website (e.g., andyljones.com); the expected result is fewer than 100 unique URLs. The interviewer confirms correctness by running the code. Python is strongly recommended; the platform used is Replit or CodeSignal.
What is the difference between multi-threading and multi-processing? When would you use each?
When a task is ready, how do you ensure it gets processed immediately (task scheduling in multi-threaded version)?
Would you use ThreadPoolExecutor or asyncio? What are the tradeoffs?
How would you improve concurrency performance and throughput beyond simple locking?
What if we need to scale this to crawl millions of URLs across multiple machines?
The crawler is too aggressive and overloading target servers. How would you implement a politeness policy?
Many different URLs point to the same or very similar content. How would you detect and handle duplicates?
How would you handle relative URLs and redirect loops?
How would you implement rate limiting and handle request timeouts?
How would you optimize this to use multiple threads / improve performance?
How would you implement this with async I/O (coroutines) instead of threads?
| Approach | Notes |
|---|---|
| asyncio with aiohttp | True async I/O; avoids Python GIL limitations for I/O-bound work; requires rewriting the provided synchronous HTML parser as an async version (a noted pitfall—candidates must handle codec/encoding errors); more complex to implement correctly under time pressure. |
| ThreadPoolExecutor (thread pool) | Simpler to implement; works well for I/O-bound tasks despite GIL since GIL is released during I/O; interviewer-expected answer according to multiple reports; can wrap the existing blocking helper without rewriting it. |
| Manual threads with locks | Demonstrates understanding of concurrency primitives but is more verbose and error-prone; interviewers consistently redirected candidates toward thread pool instead. |
| Distributed crawling (multi-server follow-up) | For the scale-out follow-up only (not required to implement): use a centralized queue (e.g., Redis) with one coordinator assigning batches to worker servers; requires no code, just verbal design. |
Common mistakes: Forgetting a single 'await' keyword in the async version caused one candidate's code to not run correctly during the interview, leading to a failed round despite otherwise correct logic.; Writing a custom async HTML parser that failed to handle a specific encoding (0x89 codec), with no time left to fix it. The recommendation from that candidate: use ThreadPoolExecutor/asyncio.to_thread instead of writing a custom async parser.; Listing multiple concurrency approaches (native threads, locks, semaphores, gevent) before arriving at thread pool — the interviewer showed no positive signal until 'threadpool' was mentioned, but time was wasted.; Not sanitizing (stripping #fragment) before the visited-set check, or describing the sanitization step in the wrong order when explaining the solution verbally.; Treating the provided helper function as if it were already async when it is actually blocking/synchronous — using asyncio naively without wrapping the blocking call causes the event loop to block and provides no concurrency benefit.; Not having the code actually run and produce a result — the interviewer runs the code against a real website and expects fewer than 100 URLs as output. Candidates who ran out of time before running the concurrent version were penalized.
Interviewer hints: The recruiter email explicitly mentions concurrency: 'The interviewer will be checking if you're coding productively, reviewing concurrency primitives, etc' — candidates who noticed this prepared both sync and async/threaded versions.; Interviewers said 'most people don't get to the multi-threaded part' — implying that completing just the single-threaded version that runs correctly is table stakes, and the concurrent version is the differentiator.; Interviewer repeatedly asked 'where does sanitize happen?' when a candidate described URL normalization steps in the wrong order — the interviewer wanted to hear that fragment stripping happens before the visited-set deduplication check.; The platform is Replit or CodeSignal; Replit was described as slow and buggy (import errors on run). Multiple candidates recommended using Replit anyway because the interviewer can review code afterward and interact in real time.; At least one interviewer said asyncio/coroutines are preferable to raw multithreading for this problem, framing multithreading as 'not the optimal solution' — though other interviewers appeared to expect thread pool specifically.
What passers do: Writing the single-threaded version first, getting it running and producing <100 URLs, then extending to concurrent version — candidates who completed both working versions with code that actually ran were rewarded.; Stripping the URL fragment (#...) before deduplication, and articulating clearly that sanitization happens before the visited-set check — one candidate was repeatedly asked 'where does sanitize happen' because they described it in the wrong order.; Using Python with urlparse for URL handling and ThreadPoolExecutor (or asyncio.to_thread wrapping the blocking helper) for concurrency — Python was strongly recommended by multiple sources and interviewers are daily Python writers.; One candidate who passed phone screen quickly (strong hire, notified next day) described deliberately over-preparing: writing async version proactively and steering the follow-up conversation toward asyncio rather than multithreading.; Benchmarking/timing both the sync and async versions and showing the async version is faster — at least one candidate included a timer comparison and this was positively noted.
What you just read — canonical solution, follow-up arc, what passing candidates actually did — exists for all 51 Anthropic questions, refreshed monthly from new candidate reports.