Every natural language pipeline starts with tokenization: turning raw text into the discrete units a model can actually consume. In on-device systems the tokenizer quietly determines your vocabulary size, your sequence lengths, and your memory footprint — often more than the model architecture itself. Byte Pair Encoding (BPE) is the most widely used subword algorithm, and understanding its mechanics explains how a model that runs entirely in a browser tab can handle natural language without shipping a vocabulary that would otherwise be impractical.

Why Word- and Character-Level Tokenizers Fall Short

A word-level tokenizer keeps whole words as units. It is intuitive, but it has two serious problems for on-device work. First, the vocabulary must cover every word you might ever see; typos, new terms, and proper names such as "Shapley" or "MiniLM" all need entries, or they collapse into an unknown token. Second, the embedding matrix grows with the vocabulary: every added word means another row of numbers that must be downloaded and held in memory.

A character-level tokenizer fixes the vocabulary problem — 26 letters plus punctuation is tiny — but creates a new one: sequence length. A typical sentence becomes dozens of characters, and transformer attention costs grow quadratically with sequence length. On a phone CPU, that cost is real, and the model must also learn that "c", "a", "t" and "c", "a", "t", "s" are related — information a word-level model gets for free.

Subword tokenizers sit in the middle. Frequent words stay whole ("the", "and", "equilibrium"), while rare words are split into reusable pieces ("unhappiness" becomes "un" + "happiness"). The vocabulary stays compact, sequences stay short, and there is no unknown token at all.

How BPE Works

BPE was introduced for machine translation by Sennrich et al. (2016) and is a purely frequency-driven merge algorithm. You start with a corpus split into characters, count adjacent pairs, merge the most frequent pair, and repeat until you reach a target vocabulary size. The result is a list of merge rules plus a final vocabulary that ships with the model.

The training loop is compact:

from collections import Counter

def get_stats(words):
    pairs = Counter()
    for word, freq in words.items():
        symbols = word.split()
        for i in range(len(symbols) - 1):
            pairs[symbols[i], symbols[i + 1]] += freq
    return pairs

def merge(symbols, pair):
    out, i = [], 0
    while i < len(symbols):
        if i < len(symbols) - 1 and (symbols[i], symbols[i + 1]) == pair:
            out.append(symbols[i] + symbols[i + 1])
            i += 2
        else:
            out.append(symbols[i])
            i += 1
    return out

The classic worked example uses the corpus "low low low low lower newest newest". The first merge is "l" + "o" into "lo", because that pair is the most frequent. Next, "lo" + "w" becomes "low". Then "e" + "s" becomes "es", "es" + "t" becomes "est", and so on. After enough merges, "lower" and "newest" are each a single token, and any unseen word containing those pieces can still be tokenized. That last property is the whole point: BPE never produces an unknown token, because any unseen word can be decomposed into known subword units.

Applying Merges at Inference Time

Training discovers the merges; inference applies them in order. Given the learned merge list, tokenizing a new word is deterministic: split it into characters, then scan left to right applying each merge rule in the order it was learned, longest pieces first where they overlap. In JavaScript this is a few loops over a small merge table:

function tokenize(word, merges) {
  let symbols = word.split("");
  for (const [left, right] of merges) {
    const next = [];
    for (let i = 0; i < symbols.length; i++) {
      if (i < symbols.length - 1 &&
          symbols[i] === left && symbols[i + 1] === right) {
        next.push(left + right);
        i++;
      } else {
        next.push(symbols[i]);
      }
    }
    symbols = next;
  }
  return symbols;
}

Because the merge table is static, the same bytes always produce the same tokens — reproducibility matters when you index a knowledge base once and query it thousands of times. In the browser, the vocabulary and merge list ship as compact JSON alongside the model weights; the tokenizer runs as pure local code with no network round trip.

WordPiece and Byte-Level BPE

BPE has two close relatives worth knowing. WordPiece, used by BERT-family models, uses the same merge framework but scores candidate pairs by how much they increase the likelihood of the training corpus rather than by raw frequency. Byte-level BPE, used by GPT-style models, applies the merges to raw bytes instead of characters, so any Unicode text tokenizes without a special unknown path.

The all-MiniLM-L6-v2 sentence transformer that powers retrieval on ReLU.chat uses a WordPiece tokenizer with a 30,522-token vocabulary. That tokenizer, like the model itself, runs locally: ReLU.chat executes the quantized ONNX model in the browser through ONNX Runtime and transformers.js, so tokenization happens on the device and the raw text never leaves it.

Why Tokenization Matters On-Device

Three concrete effects matter when the model runs in a browser:

  • Embedding table size. The input embedding matrix is vocab x hidden dimension. For 30,522 tokens at 384 dimensions in FP32 that is roughly 46 MB of weights; at INT8 it is about 11.5 MB. A smaller vocabulary is directly smaller download and memory.
  • Sequence length. Shorter token sequences mean cheaper attention. Subword tokenization keeps average token counts closer to word-level than to character-level, which keeps transformer inference fast on CPUs without GPUs.
  • Determinism and privacy. The merge rules are static, so the same bytes always produce the same tokens, and because tokenization is local, nothing needs to be sent to a server to be preprocessed.

ReLU.chat's own policy network does not tokenize text at all — it consumes 25 numeric features — but the retrieval pipeline depends on the MiniLM tokenizer producing consistent embeddings for every query and every knowledge-base entry. Tokenizer consistency is what makes the 384-dimensional vectors comparable in the first place: if tokenization drifted between index time and query time, cosine similarity would rank noise.

Key Takeaway

BPE builds a compact subword vocabulary by repeatedly merging the most frequent adjacent pairs, giving small on-device models full coverage of natural language without huge embedding tables. Because tokenization runs locally, it is also one more layer of the privacy story: the text is processed where it is typed.