When a model runs in the browser, the tempting implementation is the obvious one: load the ONNX session, call session.run(), and put the embedding on screen. It works — until the tab stops responding to clicks while the matrix multiplications run. The browser's main thread is a serialized pipeline that parses input events, runs JavaScript callbacks, computes style and layout, and paints. A transformer forward pass is exactly the kind of work that turns that pipeline into a queue: every frame, keystroke, and pointer event behind the inference call waits its turn. Web Workers exist to move this compute off the UI thread entirely.

What Happens When Inference Runs on the Main Thread

The browser treats any task that runs longer than about 50 milliseconds as a long task, and long tasks are what make a page feel unresponsive. A single embedding computation for a sentence transformer takes a few tens of milliseconds on a desktop, but on a mid-range phone — especially one that is already warm — it can stretch to hundreds of milliseconds. Retrieval makes it worse: a chat turn may embed the query, score thousands of indexed entries, and re-rank the top candidates, which is several sequential main-thread tasks, each one long enough to drop frames.

There is a structural cost too. ReLU.chat renders answers progressively, revealing responses in roughly 40-character chunks scheduled through requestAnimationFrame. That callback only fires when the main thread is free. If a 300 ms inference pass runs on the main thread, every requestAnimationFrame callback scheduled during that window is delayed, and the streaming animation freezes exactly when the user is waiting for the next chunk.

It is worth being precise about what does not fix this. Wrapping the inference call in await or Promise does not help: the browser executes JavaScript on a single thread, and an async function merely reorders tasks on that same thread. The tab stays responsive only if the expensive work physically runs elsewhere. The setTimeout-yield pattern — chopping a forward pass into slices and yielding between them — keeps the event loop alive but does not remove the long task; it only breaks it into several slightly shorter ones. A worker is the difference between borrowing time from the UI thread and not using it at all.

Moving Inference into a Dedicated Worker

A dedicated worker is a separate JavaScript execution context with its own event loop and global scope. The main thread and the worker communicate only through postMessage, so the heavy math never competes with rendering for the same thread.

// main.js
const worker = new Worker(new URL('./embed.worker.js', import.meta.url), { type: 'module' });

worker.onmessage = (event) => {
  if (event.data.type === 'embedding') {
    renderEmbedding(event.data.values);
  }
};

worker.postMessage({ type: 'load', modelPath: '/models/all-MiniLM-L6-v2.onnx' });
worker.postMessage({ type: 'embed', text: 'mixed-strategy equilibrium' });

// embed.worker.js
import * as ort from 'onnxruntime-web';

let session = null;

self.onmessage = async (event) => {
  const { type } = event.data;

  if (type === 'load') {
    session = await ort.InferenceSession.create(event.data.modelPath, {
      executionProviders: ['wasm'],
    });
    self.postMessage({ type: 'ready' });
  } else if (type === 'embed') {
    const inputs = tokenize(event.data.text);
    const results = await session.run(inputs);
    self.postMessage({ type: 'embedding', values: results.output.data });
  }
};

Two details matter. First, the worker owns tokenization too, so the vocabulary lookup never touches the main thread. Second, the InferenceSession is created once and reused — session creation is expensive and should never happen on the hot path.

Transferables: Move Memory Instead of Copying It

postMessage has two modes. By default, structured clone copies the payload, which for a 22 MB weight buffer means copying 22 MB. The transfer list avoids the copy: ownership of the ArrayBuffer moves to the worker and the main thread's copy is detached.

const response = await fetch(modelUrl);
const buffer = await response.arrayBuffer();
worker.postMessage({ type: 'weights', buffer }, [buffer]);
// after this call, `buffer` is detached on the main thread

Transferring is effectively free and is the right choice for model weights, embedding indexes, and any large typed array the main thread will not use again. For small messages — a query string, a handful of numbers — copying is cheaper than the transfer overhead. Reserve transfer for payloads in the hundreds of kilobytes and up.

Worker Pools and Shared State

One worker handles one inference request at a time. A chat interface often wants overlap: embed the query while the policy network picks a response style, or re-rank candidates while the next message is being typed. A small pool — capped at navigator.hardwareConcurrency - 1, leaving one core for the UI thread — allows that parallelism without saturating the device.

SharedArrayBuffer lets multiple workers read the same weight buffer without copies, but it requires cross-origin isolation (COOP/COEP headers) and is not available in every embedding context. For most on-device setups, one shared worker with a request queue is simpler, and fast enough.

What Stays on the Main Thread

The division of labor is: the main thread owns the DOM, input, layout, and paint; the worker owns math. Anything that mutates the page must come back through a message and be applied in a requestAnimationFrame callback or a microtask. Keep the streaming renderer on the main thread, and let the worker post embeddings, logits, and retrieval results as plain typed arrays. The main thread stays under the long-task threshold, the streaming animation stays smooth, and inference feels like a background service rather than a freeze.

Key Takeaway

Inference is compute, and compute does not belong on the UI thread. A dedicated worker moves the model forward pass off the render path, transferable buffers move large weight arrays with zero copies, and the main thread is left free to do what it does best: rendering streaming responses at 60 frames per second.