A 22MB transformer model takes time to load. Users shouldn't stare at a loading bar. ReLU.chat uses a three-stage progressive loading strategy that gives usable answers instantly.

Stage 1: BOW + Heuristic (Immediate)

Before any model loads, the system builds a Bag-of-Words vocabulary from the knowledge base:

const voc = new Set();
for (const e of KB) for (const t of tokens(entryText(e))) voc.add(t);
bowVocab = new Map();
[...voc].forEach((w, i) => bowVocab.set(w, i));

BOW vectors are sparse binary representations — each dimension is 1 if the word appears, 0 otherwise. Combined with the heuristic policy fallback, the system can answer basic queries within milliseconds of page load.

Stage 2: MLP Policy (Fast)

The MLP weights (363 KB JSON) load in parallel with the transformer. Once loaded, the system switches from heuristic to policy-driven responses. This happens within ~100ms.

The MLP provides:

  • Intent classification (definition, example, formal, application, comparison)
  • Mode detection (normal, greeting, help, off-topic, comparison)
  • Topic and fragment count decisions
  • Tone and creativity calibration

Stage 3: Dense Transformer (Medium)

The MiniLM model (~22 MB, quantized ONNX) streams in via the service worker's cache. Once loaded:

  1. All knowledge base entries are re-encoded with dense embeddings
  2. BOW vectors are hot-swapped with 384-dimensional dense vectors
  3. BM25 index is rebuilt with proper tokenization
  4. The signal layer switches to dense-sparse ensemble ranking

The hot-swap is invisible to the user — they keep chatting while the system improves in the background.

Bot-Pack Fast Path

When a bot-pack is available (precomputed BM25 index + entry vectors), stages 1-3 are faster:

  • BM25 index loads from JSON instead of being computed client-side
  • Entry vectors load from JSON instead of being encoded at runtime

Key Takeaway

Progressive loading turns a blocking 22 MB download into a non-blocking upgrade. Users get instant answers that improve as the model loads. The key insight: BOW is good enough for first turns, and the hot-swap is invisible.