How I built offline semantic memory for my desktop AI companion: one env var, no Cloudflare account

How I built offline semantic memory for my desktop AI companion: one env var, no Cloudflare account Ankita AI is a desktop AI assistant by Krish Verma — a little companion that lives on your desktop and remembers what you tell it. The problem with remembering things is finding them again: keyword search finds "dentist appointment" when you ask about the dentist, but not when you ask "when do I next see someone about my teeth?" That paraphrase gap is exactly what semantic recall is for. Until v2.7.5, Ankita's semantic recall had one route — a Cloudflare Workers AI account. Now there is a second one: point one environment variable at a local server and recall becomes semantic with no cloud account at all. The decision: one index, two backends The mechanism lives in src/memory/embeddings.mjs. There is a single shared base class, EmbeddingIndex, that owns everything backend-agnostic: caching, background warming, and cosine ranking. Subclasses implement exactly one method, request(texts, signal) -> vectors. The Cloudflare subclass posts to Workers AI with the qwen3-embedding-0.6b model; the new LocalEmbeddings subclass posts to {base}/embeddings with { model, input } — the OpenAI-compatible shape Ollama, LM Studio, and llama.cpp all speak. The default local model is nomic-embed-text: 768 dimensions, 8k context, ships with Ollama. Setup is two lines: EMBED_LOCAL_BASE=http://localhost:11434/v1 EMBED_LOCAL_MODEL=nomic-embed-text EMBED_PROVIDER picks the backend: auto (the default), cloudflare, local, or off. In auto mode, Cloudflare credentials win if they exist and the local endpoint is the fallback; local forces the local server; off disables semantic recall entirely even when credentials exist. The resolver, embeddingProvider(), mirrors embeddingsEnabled() exactly, so the two can never disagree about which backend is live. The part that had to be boring: the cache Vectors are cached on disk, keyed by the sha256 hash of the text — the cache file holds the vector, not the text itself. The cache directory is namespaced by a hash of namespace:model, so switching from nomic-embed-text to another model (or from local to Cloudflare) can never reuse vectors of the wrong shape. Every vector is validated on the way in: at most 4,096 dimensions, all finite, non-zero norm, then normalised to unit length. A malformed provider response never enters the cache and never replaces local recall — the test suite has a dedicated case for exactly that. Indexing happens in the background. warm() collects uncached memories and embeds them in batches of 32 under a 30-second deadline; completed batches survive for the next search even if a later batch fails. A probe() at boot establishes DNS/TLS and any provider cold start once, so the first real query does not pay for it — and a failed probe deliberately does not disable embeddings or set a backoff. The next rank() tries normally. The part that had to be fast: one deadline, then keywords Recall runs on every turn, so it may never stall the reply. rank() runs the whole operation — cold indexing, query embedding, response parsing — under a single deadline, EMBED_TIMEOUT_MS, default 2 seconds. The automatic pre-turn recall gets an even tighter budget: MEMORY_RECALL_BUDGET_MS, default 300 ms. If the embedder is slow, the foreground recall falls back to local keyword candidates and the query embedding keeps running in the background, cached, so the next explicit recall is semantic and fast. If the provider fails outright, the index backs off for 30 seconds and recall is keyword-only until then. The test names say it plainly: "one deadline covers the entire embedding operation and falls back without blocking chat" and "a failing local server falls back to keyword results instead of erroring". Each query is prefixed with an instruction — "Given a conversation message, retrieve relevant personal preferences, facts, past decisions and tasks that help answer it" — and each stored fact is embedded as its text plus its project and kind. Texts are capped at 6,000 bytes, safely below the hosted model's token window. The paraphrase test proves the point: semantic recall retrieves a memory with zero keyword overlap with the query. The honest limits Two things about this design still bother me. First, the fallback is silent. When Ollama is down, recall quietly degrades to keyword matching and nothing in the reply tells you — the companion just gets a little dumber for a while, and the only trace is a status: 'unavailable' that never reaches the user. Second, auto prefers Cloudflare whenever credentials exist, which means a local-first app silently sends your memories' embeddings to a cloud provider the moment you paste in a token for something else. Both defaults chose "never break the reply" over "never surprise the user". So here is the question I keep coming back to. Tell me I'm wrong: should a dead embedder fail loudly — a visible "recall is keyword-only right now" notice in the reply — or is silent degradation the right call for a companion that must never stall a conversation? And while you're at it: for an app that calls itself local-first, should auto prefer the local endpoint when both are configured, or is preferring the cloud the honest default because it is the one that actually works out of the box? Ankita is open source (MIT) at https://github.com/akyourowngames/A.N.K.I.T.A — the embedder code is src/memory/embeddings.mjs, the tests are test/memory/embeddings.test.mjs, and the whole thing ships in v2.7.5. More on the project at https://ankita-sable.vercel.app. If you run your own embeddings server, I'd genuinely like to know which model you use and whether you've ever caught your assistant quietly falling back to keywords. — Krish Verma, creator of Ankita AI