IndexedDB for On-Device AI: Embeddings and Session Storage

On-device AI applications accumulate state: embeddings, session history, model metadata. None of it should go to a server, so the browser needs a real local persistence layer. localStorage is synchronous and tiny, the Cache API is designed for static resources, and IndexedDB is the browser's actual database — asynchronous, transactional, indexable, and capable of storing binary blobs. For browser-based AI, it is the right home for embeddings and session state.

Why Not localStorage or the Cache API

localStorage has two deal-breakers for AI workloads. It is synchronous, so reads and writes block the main thread, and it stores strings only, with browsers capping each origin around 5 MB. A 384-dimensional embedding as a JSON array alone costs several kilobytes; a few thousand of them blow the quota.

The Cache API is the right tool for a different job: immutable, versioned resources. ReLU.chat's service worker pre-caches the model and policy weights with it, which is exactly what the API is for. But caching is awkward for data you write, update, query, and evict. IndexedDB handles that: object stores, indexes, transactions, and raw binary values. The natural split is Cache API for model files, IndexedDB for the data the app generates.

Storing Embeddings as Binary

A MiniLM embedding is 384 floats. As a Float32Array that is 1,536 bytes; as a JSON array of numbers it balloons several times larger and costs a parse on every load. IndexedDB can store the raw ArrayBuffer directly:

function openDB(name, version, upgrade) {
  return new Promise((resolve, reject) => {
    const req = indexedDB.open(name, version);
    req.onupgradeneeded = () => upgrade(req.result);
    req.onsuccess = () => resolve(req.result);
    req.onerror = () => reject(req.error);
  });
}

const db = await openDB("relu-chat", 1, (db) => {
  db.createObjectStore("embeddings", { keyPath: "id" });
  db.createObjectStore("sessions", { keyPath: "id" });
});

async function saveEmbedding(id, vector) {
  await db.put("embeddings", { id, buffer: vector.buffer });
}

async function loadEmbedding(id) {
  const record = await db.get("embeddings", id);
  return record ? new Float32Array(record.buffer) : null;
}

Because the buffer is stored without serialization, loading a stored embedding is a copy of 1.5 KB instead of a JSON parse — the difference between a sub-millisecond operation and something that visibly stutters on a slow phone.

Designing the Object Store Schema

Two object stores cover most AI app needs. An embeddings store keyed by id holds one record per indexed document, with the vector as a raw buffer and metadata (title, tags, source) as plain fields. A sessions store holds conversation state, keyed by conversation id, with a timestamp field that you can index for "recent conversations" queries:

db.createObjectStore("sessions", { keyPath: "id" })
  .createIndex("updated_at", "updated_at");

Indexes make metadata queries fast without scanning every record, and they are the reason to choose IndexedDB over a flat file of JSON. Keep the vectors themselves out of indexed fields — index keys have size limits and should stay small — and treat the binary buffers as opaque payloads.

Searching Stored Vectors Locally

Once embeddings are in IndexedDB, retrieval is straightforward: load the buffers into a typed array in memory at startup, then run brute-force cosine search. For a few thousand 384-dimensional vectors this is a few million multiply-adds, comfortably interactive on mobile CPUs. If the corpus grows large enough that loading everything is wasteful, a cursor over the embeddings store processes records in chunks without holding them all at once:

async function scanAll(storeName, onRecord) {
  const tx = db.transaction(storeName, "readonly");
  const cursor = await tx.objectStore(storeName).openCursor();
  while (cursor) {
    onRecord(cursor.value);
    cursor.continue();
  }
}

Two optimizations keep this fast in practice, both of which ReLU.chat uses: bounded top-k selection (a fixed-size heap instead of sorting the whole score array) and query embedding memoization (cache the query vector so repeated questions skip the model). The heavy work is embedding inference; the search over stored vectors is nearly free.

Persisting Session State

Conversations are stateful, and on-device memory is limited by design. ReLU.chat keeps up to 30 turns of session memory with importance-based eviction, always protecting the most recent 5 turns. Older responses are compressed into 120-character summaries after 5 turns, a running summary vector is maintained with EMA decay at alpha 0.75, and entity importance decays with a half-life of 5 turns. That is a lot of structured state — and IndexedDB is the natural place to persist it.

Persisting session state has a user-visible payoff: reload the page, close the tab, or let the OS evict the background tab, and the conversation resumes instead of resetting. And because everything is written to the origin's local database rather than sent anywhere, the privacy property holds end to end: no transcript, summary, or embedding ever leaves the device. ReLU.chat's privacy-first model — no servers, no LLMs, no tracking — depends on exactly this kind of local persistence.

Transactions, Quotas, and Alternatives

IndexedDB's guarantees come from its transaction model. Group related writes in a single readwrite transaction so a partial failure rolls back cleanly, and always handle QuotaExceededError — for AI apps the fix is eviction, dropping the lowest-importance session records, which importance-based eviction already orders for you. Bump the database version and run migrations in the onupgradeneeded handler, which fires once per version change.

For very large binary payloads — model files, multi-megabyte vectors — the Origin Private File System (OPFS) offers a file API with better streaming behavior, while the Cache API remains the right home for immutable model weights. The pragmatic split: Cache API for weights, IndexedDB for structured app data, OPFS for anything that behaves like a file. IndexedDB is the workhorse in the middle, and it is the one designed for the query patterns AI apps actually have.

Key Takeaway

IndexedDB stores embeddings as raw binary and session state as structured records, with transactions and indexes where AI apps need them. Combined with the Cache API for model weights, it gives browser-based AI a complete local persistence story — and keeps the privacy promise of on-device processing intact.

IndexedDB for On-Device AI: Storing Embeddings and Session State Locally