Local Embeddings vs. API Embeddings — Why I Chose sentence-transformers

Every RAG pipeline needs to convert text into vectors. The question is where that conversion happens. You have two options: run an embedding model locally on your own hardware, or call an API that runs the model on someone else's hardware. Both work. The right choice depends on your constraints — and understanding the tradeoffs is more useful than a recommendation. This article is about why I chose local embeddings with sentence-transformers/all-MiniLM-L6-v2 for this pipeline, and when I'd switch to an API. What Embeddings Actually Do Before the tradeoffs, a quick grounding on what's happening. An embedding model takes text and converts it into a fixed-size vector of floating-point numbers — a list of 384 numbers in the case of all-MiniLM-L6-v2. That vector encodes the semantic meaning of the text in a way that allows mathematical comparison. Two pieces of text with similar meaning produce vectors that are close together in the 384-dimensional vector space. "Authentication failed" and "login was rejected" are semantically similar — their vectors will be close. "Authentication failed" and "quarterly revenue report" are semantically distant — their vectors will be far apart. This is what makes retrieval work. When you embed a query and search for the nearest chunks, you're finding chunks that are semantically similar to the question — not just chunks that contain the same keywords. The embedding model determines the quality of this semantic matching. A better model produces vectors where semantic similarity maps more accurately to vector proximity. The Local Embedding Choice My pipeline uses sentence-transformers/all-MiniLM-L6-v2 via ChromaDB's SentenceTransformerEmbeddingFunction: from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction embedding_fn = SentenceTransformerEmbeddingFunction( model_name="sentence-transformers/all-MiniLM-L6-v2" ) This runs entirely on your local CPU. No API key, no network request, no cost per embedding, no latency from a round-trip to an external service. Why this made sense for this pipeline: Zero infrastructure. The model downloads once from HuggingFace on first use and runs locally forever after. No API account, no billing, no rate limits. For a local development pipeline, this is the right friction level. No data leaves your machine. Every document you ingest is embedded locally. Nothing is sent to an external service. For documents containing sensitive information — internal policies, security documentation, code with credentials removed but still proprietary — local embeddings are the only option that doesn't create data exposure risk. Fast enough for development scale. all-MiniLM-L6-v2 is a deliberately small model — 22 million parameters, 384 dimensions — optimised for speed rather than peak accuracy. On a modern laptop CPU, it embeds a typical paragraph in milliseconds. Ingesting hundreds of documents takes seconds to minutes, not hours. Cost is zero. At development scale this doesn't matter. At production scale — millions of embeddings per day — the cost difference between local and API embedding is significant. What You Give Up With Local Embeddings Embedding quality ceiling. all-MiniLM-L6-v2 is good for a small local model. It's not as accurate as larger API-hosted models at capturing nuanced semantic similarity. For general-purpose text, the quality gap is manageable. For domain-specific content — medical terminology, legal language, specialised technical documentation — the gap widens. No GPU by default. The pipeline runs on CPU. For small document sets this is fine. For large-scale ingestion of thousands of documents, CPU embedding becomes a bottleneck. Switching to GPU requires hardware changes, not just configuration. Model staleness. The model you download is fixed. The embedding landscape evolves rapidly — better models are released regularly. Updating the embedding model means re-embedding the entire document corpus because you need all vectors to be in the same embedding space. An API-based approach where the provider manages model updates avoids this — but introduces its own versioning challenges. No consistency guarantee across machines. Different machines running the same model from the same checkpoint produce identical embeddings — but different model versions or different precision modes may not. For a solo local pipeline this isn't an issue. For a team sharing a vector store, it matters. When You'd Switch to an API The README explicitly calls out Voyage AI as Anthropic's recommended embeddings partner — the natural pairing with Claude for generation. Here's when the switch makes sense: Production scale. When you're embedding millions of documents or serving thousands of queries per day, local CPU embedding doesn't scale. A hosted API with GPU infrastructure handles this without you managing hardware. Higher quality requirements. For domains where retrieval accuracy is critical — a medical documentation system where a wrong retrieval could mean wrong advice, or a legal research tool where missing a relevant clause has real consequences — the quality ceiling of all-MiniLM-L6-v2 may not be sufficient. Voyage AI's models are meaningfully larger and more accurate. Multi-user systems. When multiple services need to embed content into the same vector store, a centralised embedding API ensures consistency. Every service calls the same endpoint, gets the same model, produces compatible vectors. Data you're comfortable sending externally. If the content being embedded is non-sensitive — public documentation, open-source codebases, published articles — the data exposure argument against API embeddings evaporates. The swap in my pipeline is one line in rag/store.py: # Current — local from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction embedding_fn = SentenceTransformerEmbeddingFunction( model_name="sentence-transformers/all-MiniLM-L6-v2" ) # Production — Voyage AI import voyageai from chromadb.utils.embedding_functions import VoyageAIEmbeddingFunction embedding_fn = VoyageAIEmbeddingFunction( api_key=os.environ["VOYAGE_API_KEY"], model_name="voyage-2" ) The store interface doesn't change. The loader doesn't change. The pipeline doesn't change. The swap is isolated to the embedding function configuration — which is exactly why the clean component boundaries matter. The Embedding Dimension Consideration all-MiniLM-L6-v2 produces 384-dimensional vectors. Larger models produce 768, 1024, or even 1536-dimensional vectors. Higher dimensions generally mean better accuracy — more dimensions allow finer-grained semantic distinctions. They also mean larger storage requirements and slower similarity search as the vector space grows. For Chroma with a few thousand chunks, 384 dimensions is perfectly adequate — the similarity search is fast regardless. For a production system with millions of chunks, the dimension choice affects both storage cost and query latency, and the tradeoff needs to be evaluated against your accuracy requirements. The Critical Constraint: Embedding Consistency One constraint that catches people: the same embedding model must be used for both ingestion and query. When you ingest a document, you embed it with all-MiniLM-L6-v2 and store the 384-dimensional vector. When you query, you embed the question with the same model to get a 384-dimensional query vector. Similarity search finds the stored vectors closest to the query vector. If you embed documents with all-MiniLM-L6-v2 and then query with Voyage AI's voyage-2 (which produces 1024-dimensional vectors), the similarity search fails — not just with lower quality, but with an error, because you can't compare vectors of different dimensions. This is why changing the embedding model requires re-embedding the entire corpus. You can't mix vectors from different models in the same collection. My pipeline handles this by storing the embedding function configuration centrally in rag/config.py. Changing the model name in one place changes it for both ingestion and query. But it doesn't handle the corpus migration automatically — if you change the model after ingesting documents, you need to clear the Chroma collection and re-ingest. A production system would handle this with versioned collections: docs_v1 embeds with model A, docs_v2 embeds with model B, traffic cuts over when migration is complete. Simpler but less rigorous: document the embedding model version in the collection metadata so you always know which model a collection was built with. What I'd Choose for a Production Security Tool For a production RAG system specifically in a security context — scanning codebases, searching security policies, supporting threat modelling — I'd make different choices than I made here: Embeddings: Voyage AI's voyage-code-2 for code content, voyage-2 for prose. The code-specific model is trained on code and produces significantly better semantic matching for programming content than general-purpose models. But with a caveat: proprietary code going to an external embedding API is a sensitive data flow. The security team needs to approve it, the vendor's data handling policies need to be reviewed, and the data classification of the code being embedded needs to be understood. For code containing security-sensitive logic, local embeddings or an on-premise hosted model may be the only acceptable option. This is the kind of decision that sits at the intersection of AI capabilities and security policy — exactly the kind of thinking a security-focused AI engineer needs to apply. Full source at github.com/pgmpofu/rag-pipeline. The embedding configuration is in rag/store.py and rag/config.py. Next up: Chroma as a local vector store — what it is, how it works, and what you'd replace it with in production.