Perplexity's byte-tokenizer OA, taken on CoderPad, from two reports. One is a test-taker's detailed account of the OA prompt, from December 2025 (the method-signature block is missing from the post); it links to a post with a screenshot of the provided slow_tokenize. The other is a first-hand account posted in January 2026, with a reply from a second candidate. Four hours were given, but one candidate's recruiter said most people finish in about an hour. Both candidates were rejected. The first spent too long on Part 1 and guessed that was the reason. The second had read earlier reports, wrote the solution straight off, passed every test and was still rejected; no reason is given.

The OA has three steps: Part 0 (describe the given algorithm in English), Part 1 (a fast tokenize) and Part 2 (estimate a token count). Only code in byte_tokenizer.py is graded, and only standard-library imports are allowed.

Given — slow_tokenize, as posted

The prompt says to treat the reference implementation's behavior as authoritative, even if you think some other behavior is preferable, and not to assume any input constraint unless it follows from the problem or from the reference's code and comments. Even if you know LLM tokenization, expect subtle differences from the implementations you have seen. It recommends at least ten minutes studying the code before coding.

The screenshot shows lines 207–278 of the provided file: the method below, comments included. It ends at token_ids = new_token_ids; the final return token_ids is not shown and is added here. The constructor is not shown either: ours only sets the two attributes the method reads.

class ByteTokenizer:
    def __init__(self, token_alphabet: list[bytes]):   # ours: not in the screenshot
        self.token_alphabet = token_alphabet
        self.token_to_id = {t: i for i, t in enumerate(token_alphabet)}

    # NOTE: DO NOT CHANGE ANY CODE IN slow_tokenize.
    def slow_tokenize(self, text: bytes) -> list[int]:
        """
        Tokenize some text slowly using a naive algorithm.

        Returns a list of ints. Each int represents the zero-based
        index of a token within self.token_alphabet.
        """
        # STEP 1: INITIALIZATION
        #
        # We initialize the list of tokens (token_ids) by treating
        # each byte in the text as its own single-byte token.
        #
        # Thanks to our guarantees on self.token_alphabet, we can
        # perform this conversion simply by translating each byte
        # in the text to its corresponding ASCII integer value.
        # This is especially easy to do in Python via list(text).
        #
        # Example: the text b"hello" becomes the following list of
        # tokens:
        # [
        #   104,   (corresponding to b"h")
        #   101,   (corresponding to b"e")
        #   108,   (corresponding to b"l")
        #   108,   (corresponding to b"l")
        #   111    (corresponding to b"o")
        # ]
        token_ids: list[int] = list(text)

        # STEP 2: MERGING INTO MULTI-BYTE TOKENS
        #
        # We iterate over each multi-byte token in ascending order
        # of appearance in the token alphabet. For each such
        # token, we scan over token_ids and try to merge
        # consecutive pairs of tokens into the current token.
        #
        # Carefully read the code below to understand the precise
        # logic of how this is done.
        for cur_token_id in range(256, len(self.token_alphabet)):
            cur_token_bytes = self.token_alphabet[cur_token_id]

            # Find which pairs of smaller tokens can be merged into
            # the current token.
            pairs_to_merge: set[tuple[int, int]] = set()
            for split_point in range(1, len(cur_token_bytes)):
                left_bytes = cur_token_bytes[:split_point]
                right_bytes = cur_token_bytes[split_point:]

                left_id = self.token_to_id.get(left_bytes)
                right_id = self.token_to_id.get(right_bytes)

                # If both parts exist as tokens, this is a valid merge
                if left_id is not None and right_id is not None:
                    pairs_to_merge.add((left_id, right_id))

            # Perform all valid merges for the current token,
            # from left to right.
            new_token_ids: list[int] = []
            i = 0
            while i < len(token_ids):
                # Check if tokens at positions (i, i + 1) can be
                # merged into the current token.
                if (i < len(token_ids) - 1 and
                        (token_ids[i], token_ids[i + 1]) in pairs_to_merge):
                    new_token_ids.append(cur_token_id)
                    i += 2

                else:  # No merge is possible, copy the original token
                    new_token_ids.append(token_ids[i])
                    i += 1

            token_ids = new_token_ids

        # Added: the screenshot of the provided file ends at the line above.
        return token_ids

The comments refer to guarantees on self.token_alphabet that the screenshot does not show. What they imply is that the first 256 entries are the single bytes, so byte value b starts as token id b.

What the tests call (our setup — ask your interviewer)

t = ByteTokenizer(token_alphabet)                    # token_alphabet: list[bytes]
t.slow_tokenize(text: bytes) -> list[int]            # given
t.tokenize(text: bytes) -> list[int]                 # Part 1
t.estimate_token_count(text: bytes, sample_size: int, rng: random.Random) -> int   # Part 2

The test alphabets: entry i for i < 256 is the single byte i; every later entry is two earlier entries joined, and no entry repeats. The screenshot does not show whether the OA guarantees that second property, and the tests never use an alphabet without it.

Before coding — the OA's Part 0 (not graded here)

Write a plain-English description of slow_tokenize in part0_description.md, precise and detailed enough that a smart computer science student could reimplement the algorithm from scratch. The prompt lets you go back and edit it during Part 1.

Part 1 — Fast tokenize, identical output

• Implement tokenize(self, text: bytes) -> list[int]. It must return identical values to slow_tokenize; any difference is incorrect.
• Speed (prompt): strive to have the Part 1 tests run in 10 seconds or less in total. They tokenize about 500 KB of text with a token alphabet of 10,000+ distinct tokens. That takes algorithmic improvements and reasonably efficient code, but no hyperoptimization. Only tokenize() is timed: any preprocessing in the constructor is free, within reason.
• The prompt says the private test set is more comprehensive than the tests you are given.
• One candidate tried several simple changes to the reference, and each one timed out. That candidate says the fast version needs a priority queue and a linked list.
• Our tests (our setup): identical output on hand-built and 300 random alphabets, and a speed check: on 20 KB of text over a 2,000-token alphabet, tokenize must be at least 10 times faster than slow_tokenize. That is our scaled-down stand-in for the 10-second target.

Part 2 — Estimate the token count from samples

• Suggested time (prompt): 20–40 minutes.
• Add estimate_token_count(self, text: bytes, sample_size: int, rng: random.Random) -> int, which estimates the number of tokens self.tokenize(text) returns. The signature block is missing from the posted prompt; this one uses the parameter names the prompt uses, and returning an int is our choice.
• You may call self.tokenize any number of times, as long as the bytes passed to it, summed across all calls, do not exceed sample_size. You may assume sample_size is at least 1000.
• Any randomness goes through rng, a random.Random object.
• Other than calls to self.tokenize, your code should not implement anything tokenizer-like.
• If sample_size >= len(text), return the exact token count.
• Accuracy (prompt): for text larger than the sample, within 20% of the true count; for sample sizes above 10,000, within 5%. Meeting these suffices to pass, and simple approaches are fine. The OA's own Part 2 tests pass whatever you return and print a table of your accuracy; the private test set is more comprehensive.
• One candidate notes that Part 2 runs on your Part 1 tokenize, so if Part 1 does not pass, Part 2 cannot either. That candidate thought Part 2 seemed to have hidden test cases.
• Our tests (our setup): exact counts on a 3,000-byte text at sample sizes 3,000 and 5,000; the byte budget at sample sizes 1,000, 1,500 and 4,321; within 20% at sample size 2,000 on 30 KB of text, and within 5% at sample size 12,000 on 40 KB, each over five rng seeds.