The default assumption in applied ML is that bigger is better: more parameters, more knowledge, higher quality. On a server, that trade is often worth paying. On an edge device — a phone, a tablet, a browser tab — the same assumption quietly fails, because quality is not a property of the model alone. It is a property of the model, the task, the data it is evaluated on, and the constraints it runs under. A small distilled model, fine-tuned for a narrow task and quantized to fit in memory, can beat a model ten times its size in exactly the situations where on-device AI is useful.

What Distillation Actually Removes

Knowledge distillation trains a small "student" model to reproduce the outputs of a large "teacher" — not just the hard labels, but the teacher's probability distribution over answers. The student learns where the teacher was confident and where it hedged, which compresses the teacher's behavior into far fewer parameters. What distillation removes is redundant capacity, not task knowledge: a student distilled on a specific domain can match the teacher on that domain while keeping a fraction of the parameters.

The interesting consequence is that a distilled model fine-tuned on a narrow task can beat a larger general model on that task. The general model spends capacity on everything; the distilled model spends all of its capacity on one thing. For a closed-domain chatbot backed by a curated knowledge base, that narrow focus is precisely what you want.

Why Small Models Win on Edge Devices

The quality gap between a small and large model is measured in accuracy points; the gap in operational cost is measured in orders of magnitude.

  • Latency: a forward pass is roughly proportional to parameter count. Halve the parameters and you roughly halve the compute, before considering cache effects.
  • Memory: a quantized small model fits in the tens of megabytes. ReLU.chat's all-MiniLM-L6-v2 embedder is quantized ONNX at roughly 22 MB with 384-dimensional embeddings; a large encoder in fp32 would consume an order of magnitude more memory, competing with the page itself on low-end devices and triggering tab eviction in mobile browsers.
  • Cold start: smaller weights parse faster, so the first inference starts sooner. In a progressive-loading architecture this determines how quickly the full pipeline can hot-swap in.
  • Thermal throttling: sustained compute on phones throttles CPU and GPU clocks. A small model finishes its pass before throttling sets in; a large model runs most of its work at reduced clocks.
  • Cache locality: weights that fit in the processor's cache avoid repeated memory traffic. The difference between an in-cache and out-of-cache matrix multiply is often larger than the difference between model sizes.

Quantization compounds all of this. ReLU.chat's policy network — an MLP with 25 inputs, hidden layers of 128 and 64, and 6 action heads, roughly 13,079 parameters — is auto-quantized to symmetric int8 at load, cutting memory about 4x. When a model is that small, quantization is nearly free and the memory win is effectively permanent.

When a Small Model Matches — or Beats — a Big One

Quality is task-relative. In a retrieval pipeline, the embedder does not answer questions; it ranks candidates. ReLU.chat combines dense cosine similarity with field-weighted BM25 (k1=1.5, b=0.75) and fuses the rankings at 70% dense / 30% sparse. The retrieval layer carries most of the quality burden: if BM25 and the dense index agree on the right fragment, a modest 384-dimensional embedder is plenty. A giant embedder adds little when the knowledge base is curated and the queries are short.

The same budget argument applies in reverse. A small embedder leaves compute budget for the parts of the pipeline that improve quality more than parameters do: exact-match scoring, bigram phrase matching, re-ranking, and ensemble fusion. Spending the same milliseconds on retrieval engineering instead of a bigger model usually moves end-to-end quality more.

This holds only when the task is bounded. If the knowledge is closed — a game-theory glossary, a historical dataset, a data-science reference — a distilled model over a curated index will beat a general model doing zero-shot search, and do it privately in the browser with no server round trip.

When Bigger Is Actually Necessary

There are tasks where size genuinely wins: open-ended generation, multi-step reasoning, long-context comprehension, and anything requiring world knowledge beyond the knowledge base. No amount of retrieval engineering substitutes for a model that has seen the answer during training. But those tasks also carry the full edge cost — hundreds of megabytes of weights, seconds of latency, and usually a server. The honest framing is a decision rule: if the task is narrow and the knowledge is bounded, buy quality with retrieval and fine-tuning, not parameters. If the task is open, size wins, and you should measure whether the edge device can afford it.

The Decision Rule, Measured

Whatever you choose, measure on the target device with the real constraints: throttled CPU, warmed cache, memory pressure from the rest of the page. A model that scores higher on a GPU benchmark but takes 800 ms to produce a first token on a mid-range phone is not higher quality — it is broken. Pick the smallest model that clears your quality bar on your own evaluation set, then spend the saved budget on retrieval, quantization, and latency.

Key Takeaway

Model size is a means, not a goal. On edge devices, a small distilled model fine-tuned for a narrow task — quantized, cache-local, and paired with strong retrieval — routinely beats a much larger general model, because quality is measured end to end, under real constraints, on the actual task.