Conversation has a rhythm. In human dialogue, a pause of a few hundred milliseconds reads as thinking; a pause of a second reads as a problem. Chat interfaces inherit that rhythm, and users judge an AI chatbot by it: the moment between pressing send and seeing the first token is the single most important latency in the product. The 400ms rule is a budget for that moment — the first visible token should appear within 400 ms of the user's input, and everything after that should stream.
The Perception Thresholds That Matter
Interaction latency research gives three landmarks:
- Under 100 ms: perceived as instant.
- 100–300 ms: perceived as a small, acceptable delay.
- 300–1000 ms: the user notices the wait and starts to doubt the system.
- Over 1000 ms: the flow of the conversation breaks; users repeat themselves or abandon the turn.
For conversational UI, the operative number is time to first token, not time to completion. Users tolerate a long streamed answer — the stream itself signals progress — but they do not tolerate a silent wait. Streaming turns a 5-second total response into a 400 ms first-token experience.
Where the Budget Goes
A browser-based chat turn spends time in several stages, and each one takes a slice of the budget:
- Input handling and validation: single-digit milliseconds.
- Intent classification: cosine similarity against prototype vectors — fast, but it runs before retrieval.
- Retrieval: BM25 scoring plus dense embedding and cosine similarity over the index.
- Policy inference: picking response mode, topic count, fragment count, and tone.
- Rendering: scheduling the first chunk through
requestAnimationFrame.
The budget equation is simple arithmetic: the sum of these stages must stay under 400 ms on the slowest device you support. On a warm phone, embedding the query alone with a 22 MB quantized model can take 100–300 ms, which leaves almost nothing for the other stages — so the architecture has to cheat the equation, not just optimize it.
Progressive Loading: Answer Before the Model Is Ready
The most effective way to meet a 400 ms budget on a slow device is to not wait for the big model at all. ReLU.chat loads in three progressive stages: a heuristic/BOW fallback answers the first turns instantly with no model weights loaded; an MLP policy (~13,079 parameters) takes over as soon as its weights arrive; and the full dense MiniLM embedder hot-swaps in behind the scenes. The service worker pre-caches the model and policy weights, so the hot-swap never depends on the network.
The result is that first-turn latency is bounded by the fastest stage, not the slowest one. The user gets an answer while the 22 MB model downloads, and by the time the conversation needs dense retrieval, the weights are already warm.
Micro-optimizations matter inside the budget too: memoizing query embeddings so repeated questions skip the forward pass, bounding top-k selection instead of sorting the full index, LRU caches for hot fragments, and pre-built BM25 IDF tables so retrieval never recomputes statistics. Each one shaves tens of milliseconds — the difference between 380 ms and 450 ms on the device that matters.
Streaming Is the Second Half of the Rule
The 400ms rule has a second clause: after the first token, keep tokens coming. ReLU.chat reveals responses in roughly 40-character chunks scheduled via requestAnimationFrame, so the answer appears progressively and the frame rate stays stable. Rendering a long answer all at once converts total latency into perceived latency — the user stares at a blank screen for the full generation time, then gets everything at once.
If a stage genuinely cannot meet the budget — the model is still downloading, or the device is throttling — show a typing indicator and keep the pipeline warm. A visible, honest "working" state is far cheaper than a silent freeze.
Perceived latency is not the same as measured latency, and conversational UI gets to exploit the gap twice. First, chunked streaming makes total latency invisible: the user stops timing at the first token, so a 4-second generation reads as a 400 ms response. Second, progressive rendering keeps the frame budget intact — each ~40-character chunk is cheap to layout and paint, so the animation itself never adds jank that would feel like additional delay. The failure mode to avoid is the opposite: buffering an entire answer and rendering it at once, which turns a fast pipeline into a slow-feeling one.
Measuring the Budget
Budgets are only useful if you measure them. Instrument the pipeline:
const t0 = performance.now();
const result = await answer(userText);
const firstTokenMs = performance.now() - t0;
const totalMs = performance.now() - tStart;
if (firstTokenMs > 400) {
report('budget-overrun', { firstTokenMs, stage: slowestStage() });
}
Track time to first token separately from total time, record it per device class, and watch the Long Tasks API for main-thread jank that steals budget from the renderer. Set a per-stage budget (for example, intent 50 ms, retrieval 150 ms, policy 30 ms, render 20 ms) so regressions are attributable, and re-measure after every change to the model or the index.
Key Takeaway
Users experience latency as time to first token, not time to completion. Budget 400 ms for that first token on your slowest device, stream everything after it, and design the pipeline so the heavy model is never on the critical path of the first answer — progressive loading, warm caches, and memoization turn a hard latency target into a routine one.