Applied AI Tech Screen: Debug an Async Refund, Then Rapid-Fire LLM Questions

Format. A technical screen for Mercor's Applied AI role, from one report posted in September 2026: a debugging exercise, and a list of rapid-fire questions on LLM applications and idempotency that ends with two questions about a project you lead. The report doesn't give the length, whether you run the code, or the outcome.

1. Debug an async refund function. The code, as the candidate posted it:

async def refund_money(request_id: str):
    refund = await get_db_record(request_id)
    if refund.status == "done":
        return
    for i in range(3):
        try:
            await stripe_api_call(
                refund.request_id,
                refund.amount
            )
            break
        except Error:
            continue
    await mark_db_as_done(request_id)

The report sums it up as debugging an async refund/payment function with retries and database state updates.

What's wrong with it (ours, not from the report). We ran it against a fake database and a fake Stripe; the first four of these reproduce:

• except Error: names a class Python doesn't have (unless the codebase defines one). The first time the Stripe call raises, evaluating Error raises NameError, so nothing is ever retried. Catch the client library's real exception types.
• It marks the refund done even when every attempt fails. After three failures the loop ends and mark_db_as_done runs anyway, so the customer is never refunded but the record says they were.
• A retry can refund twice (unless stripe_api_call already sends an idempotency key, which the snippet doesn't show). If Stripe processes the refund but the response is lost, the retry sends a second refund. Send an idempotency key derived from the request, the same on every attempt.
• Two concurrent calls can both refund. Both read a status that isn't done, then both call Stripe. Claim the work with an atomic state change (pending → processing, as a compare-and-set or a row lock) before calling out.
• A crash between the Stripe call and mark_db_as_done leaves a refunded record that isn't marked; the next run refunds again unless the idempotency key is there. A processing claim also needs a lease or a reconciler, so a worker that dies doesn't hold it forever.
• Every error is retried, with no pause and no record. Retry only what can succeed later (timeouts, 5xx, rate limits), not an invalid request; back off exponentially with jitter; log each failure.
• A missing record (refund is None) raises an unclear AttributeError.

How Stripe's idempotency keys behave (from Stripe's API v1 docs): Stripe saves the first response for a key, errors included, and returns it for any repeat within about 24 hours; reusing a key with different parameters is an error, and so is a repeat that arrives while the first request is still running. So a retry after a timeout is safe, but after a saved 5xx the same key just replays the error: check the refund's status at Stripe instead of resending.

A fixed version (ours). Tested against the fakes: every attempt failing, success on a later retry, a lost reply, every reply lost, two concurrent calls, an invalid request, a charge already refunded, an unexpected exception, retrying a failed refund, a repeat call and a missing record.

import asyncio
import logging
import random

log = logging.getLogger("refunds")


class RetryableError(Exception):          # timeout, 5xx, rate limit: may succeed later
    pass


class PermanentError(Exception):          # invalid request: retrying won't help
    pass


class AlreadyRefundedError(Exception):    # the provider says this charge is already refunded
    pass


async def refund_money(request_id: str, max_attempts: int = 3) -> str:
    refund = await get_db_record(request_id)
    if refund is None:
        raise KeyError(f"no refund record for {request_id}")
    if refund.status == "done":
        return "done"
    # Claim the work with an atomic update (pending or failed -> processing), so a
    # second concurrent call does not also send a refund. A worker that dies here
    # leaves the claim behind, so the claim needs a lease or a reconciler.
    if not await claim_for_processing(request_id):
        return "in_progress"
    try:
        for attempt in range(max_attempts):
            try:
                await stripe_api_call(
                    refund.request_id,
                    refund.amount,
                    idempotency_key=f"refund-{request_id}",   # same key on every retry
                )
                await mark_db_as_done(request_id)
                return "done"
            except AlreadyRefundedError:
                await mark_db_as_done(request_id)
                return "done"
            except PermanentError:
                log.warning("refund %s rejected", request_id, exc_info=True)
                await mark_db_as_failed(request_id)
                return "failed"
            except RetryableError:
                log.info("refund %s attempt %d failed", request_id, attempt + 1, exc_info=True)
                if attempt < max_attempts - 1:
                    await asyncio.sleep(min(2 ** attempt, 30) * random.uniform(0.5, 1.0))
        # Every attempt timed out: Stripe may or may not have refunded. Don't guess;
        # flag it so a reconciler checks the refund at Stripe before retrying or failing it.
        await mark_db_as_needs_check(request_id)
        return "unknown"
    except BaseException:
        await mark_db_as_needs_check(request_id)  # never leave the claim stuck in processing
        raise

When every attempt times out, the refund may or may not have gone through, so the record is flagged for a reconciler to check at Stripe rather than marked failed.

2. Rapid-fire questions (word for word from the report), each with a short answer (ours):

• What is idempotency, and why do we need idempotent operations? An operation is idempotent if doing it twice has the same effect as doing it once. Networks force retries (a timeout doesn't tell you whether the request ran) and message queues deliver at least once, so any operation that may be repeated must be safe to repeat: use idempotency keys stored with the first result, naturally idempotent writes (set a value rather than add to it), or a dedupe table.
• What is RAG, and how is it related to fine-tuning? Retrieval-augmented generation fetches relevant documents at query time and puts them in the prompt, so knowledge stays outside the model: easy to update, citable, and filtered per user. Fine-tuning changes the model's weights, which suits teaching a format, style, tone or task behaviour, but is a costly way to keep facts current and can't cite sources. They combine well: retrieval for knowledge that changes, fine-tuning for behaviour, including fine-tuning a model to use retrieved context better.
• How would you handle context limit exceeded in an LLM application? Count tokens before the call rather than waiting for the error. Then shrink what you send: drop or summarize older turns, retrieve only the relevant chunks instead of whole documents, and split long inputs into pieces processed separately and combined (map-reduce). Use a longer-context model if the task needs it, and handle the error gracefully if it still happens.
• LLM calls are usually stateless. How would you make an LLM application stateful? Keep the state outside the model: store the conversation and task state per session (a database or cache), and send the relevant part with each call. Summarize older history to stay within the context window, and keep long-term memory as stored facts retrieved when relevant. Some provider APIs can also keep the conversation on their side.
• How would you handle prompt injection in an AI agent system? Treat everything the agent reads (web pages, documents, tool output) as untrusted data, not instructions; delimiting it helps but is not enough on its own. Limit the damage instead: give tools the least privilege they need, validate tool calls against an allowlist, require human confirmation for sensitive actions, never let model output trigger privileged operations directly, and log and monitor what the agent does. Prompting alone doesn't solve it.

3. Ownership (word for word, listed with the rapid-fire questions):

• What project are you currently leading?
• Why would you consider yourself the owner of that project rather than just a participant?

Our advice: pick a project where you set the goal, made the trade-offs, unblocked others and answered for the result, and say which decisions were yours, with a number for the outcome.