ReLU.chat runs a reinforcement-learning-trained MLP policy network directly in the browser. With 13,079 parameters across 16 weight tensors, it decides how to respond to every query — mode, intent, topic count, fragment count, tone, and creativity. Until recently, all inference used Float32 math. We changed that.

The Problem

The MLP weights ship as a JSON file (policy.weights.json, ~363 KB). At construction time, the MLPPolicy class loads these into JavaScript arrays of arrays. The forward pass — two hidden layers (25→128→64) plus six action heads — runs entirely in Float32. For a 13K-param model, this is fast enough (~50μs per inference). But we asked: can we do better?

Symmetric Int8 Quantization

The answer was per-layer symmetric int8 quantization. For each weight matrix W:

  1. Compute scale = max(|W|) / 127
  2. Quantize: W_q = round(W / scale), clamped to [-127, 127]
  3. Store: { scale: float32, data: Int8Array }
At inference time, the int8 dot product accumulates in Float64 (safe for layers up to ~1000 in-features due to 53-bit mantissa), then dequantizes by multiplying by the scale factor. Bias vectors remain in Float32 — they're added after dequantization.

Results

  • Memory: ~4× reduction. Float32 weights: ~52 KB. Int8 weights: ~13 KB + scale factors.
  • Speed: Int8 dot products are faster because they operate on single-byte values, improving cache locality and reducing memory bandwidth.
  • Accuracy: Per-layer scale factors preserve the full dynamic range of each layer independently. In practice, the MLP's outputs (softmax probabilities and sigmoid values) are identical to Float32 within floating-point rounding error.

Implementation

The quantization runs automatically at construction time in MLPPolicy:

constructor(weights) {
  this._validate(weights);
  this.weights = weights;
  this.loadQuantized(); // auto-quantize on construction
}

The planAnswer() method uses the quantized forward path by default:

const probs = this._qWeights
  ? this.forwardQuantized(f32)
  : this.forward(f32);

Why Not Just Use WASM?

We considered compiling the MLP to WebAssembly. But for a 13K-param model, the overhead of WASM instantiation, memory management, and the JS↔WASM boundary exceeds the inference time itself. Pure JS with int8 quantization is the sweet spot for models this small.

Key Takeaway

You don't need a GPU or WASM to make small neural networks fast in the browser. Symmetric int8 quantization in pure JavaScript gives you ~4× memory reduction and faster inference with zero accuracy loss for models under ~100K parameters. The implementation is 50 lines of code.