Semantic Search in C++ without Python, libtorch or ONNX Runtime

Ask how to run a transformer model from C++ and you get two answers: link libtorch, or convert the model and link ONNX Runtime. Both work. Both are large, both want a toolchain of their own, and both put a second inference engine inside your process. There is a third answer, and it takes four commands. mkdir kjarni-quickstart && cd kjarni-quickstart curl -sL https://github.com/olafurjohannsson/kjarni/releases/latest/download/kjarni-x86_64-linux.tar.gz | tar xz curl -sO https://raw.githubusercontent.com/olafurjohannsson/kjarni/main/crates/kjarni-ffi/examples/cpp/hello.cpp g++ -std=c++23 hello.cpp -I. -L. -lkjarni_ffi -Wl,-rpath,'$ORIGIN' -o hello && ./hello related: 0.5510 unrelated: -0.0630 That is a transformer model, downloaded, loaded and run, from an empty directory. No package manager, no Python, no model conversion step. The archive holds the shared library, kjarni.h (the C ABI) and kjarni.hpp (a header-only C++23 wrapper). macOS and Windows builds are on the same [releases page(https://github.com/olafurjohannsson/kjarni/releases). Here is what hello.cpp contains: #include "kjarni.hpp" #include int main() { // Downloaded once and cached under ~/.cache/kjarni, then loaded from disk. auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"}); if (!embedder) { std::println("{}", embedder.error().message()); return 1; } auto question = embedder->encode("How do I get my money back?"); auto related = embedder->encode("What is your refund policy?"); auto unrelated = embedder->encode("The weather in Reykjavik is unpredictable."); // No shared words with the question, but the same meaning. std::println("related: {:.4f}", kjarni::cosine(*question, *related)); std::println("unrelated: {:.4f}", kjarni::cosine(*question, *unrelated)); } "How do I get my money back?" and "What is your refund policy?" share no words at all, and score 0.55. The sentence about the weather scores below zero. That gap is the entire idea behind semantic search. How semantic search works An embedding model reads text and returns a vector, an array of floats, 384 numbers for the model above. Text with similar meaning produces vectors that point in similar directions. "refund policy" -> [0.12, -0.34, 0.56, ...] (384 numbers) "get money back" -> [0.11, -0.33, 0.55, ...] (384 numbers) [-0.45, 0.23, -0.12, ...] (384 numbers) /home/you/kjarni-quickstart/libkjarni_ffi.so libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux-x86-64.so.2 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 Kjarni, the C++ runtime, and the parts of glibc every program already uses: libc, libm, libgcc, libpthread and libdl. That is the whole list. No libtorch, no onnxruntime, no Python, no CUDA runtime. The binary is 283 KB and the library is 19.4 MB, which includes the tokenizer, the model loaders and every kernel. The -Wl,-rpath,'$ORIGIN' in the build line is what makes that first entry resolve to the library sitting next to your binary rather than something in /usr/local/lib. Keep it, and the directory you built in is a directory you can copy somewhere else and run. A dependency you cannot see in ldd is a dependency that cannot break you on a machine that is not yours. Errors are values Every fallible call returns std::expected. Nothing in the header throws except std::bad_alloc. auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"}); if (!embedder) { std::println(stderr, "could not load model: {}", embedder.error().message()); return 1; } Whether a missing model file is exceptional depends on the program. A batch job should die; a desktop application should show a message and carry on. Returning the failure lets the caller decide, and puts it in the signature where it cannot be missed. The options are a designated-initialiser aggregate, so a call names only what it changes: Embedder::create({.model = "mpnet-base-v2", .gpu = true}). Searching a corpus Encode the documents once, encode the query at search time, sort by similarity. #include "kjarni.hpp" #include #include #include #include #include int main() { auto embedder = kjarni::Embedder::create({.model = "minilm-l6-v2"}); if (!embedder) { std::println(stderr, "could not load model: {}", embedder.error().message()); return 1; } constexpr std::array docs = { std::string_view{"How do I reset my password?"}, std::string_view{"What is your refund policy?"}, std::string_view{"Do you ship internationally?"}, std::string_view{"How do I update my billing address?"}, std::string_view{"Where can I track my order?"}, }; std::vector corpus; corpus.reserve(docs.size()); for (std::string_view d : docs) { auto v = embedder->encode(d); if (!v) { std::println(stderr, "encode failed: {}", v.error().message()); return 1; } corpus.push_back(v->to_vector()); } constexpr std::string_view query = "I need to change my login credentials"; auto q = embedder->encode(query); if (!q) { std::println(stderr, "encode failed: {}", q.error().message()); return 1; } std::vector scored; for (auto [i, doc] : std::views::enumerate(docs)) scored.emplace_back(kjarni::cosine(q->values(), corpus[i]), doc); std::ranges::sort(scored, std::ranges::greater{}, &std::pair::first); std::println("query: \"{}\"", query); for (auto [score, doc] : scored) std::println(" {:.4f} {}", score, doc); } query: "I need to change my login credentials" 0.5981 How do I reset my password? 0.4067 How do I update my billing address? 0.0767 Where can I track my order? -0.0027 What is your refund policy? -0.0451 Do you ship internationally? "Change my login credentials" matches "reset my password" at 0.60 while sharing no words with it, and "update my billing address" comes second because changing account details is a related idea. That is what you cannot get from keyword matching. encode returns a kjarni::Embedding, which owns the array the C API returned and frees it in its destructor. It hands out a std::span through values(), so it drops straight into ranges, and to_vector() copies when the data has to outlive the object. There is no raw pointer to forget. Classification and reranking The other models follow the same shape. auto clf = kjarni::Classifier::create({.model = "roberta-sentiment"}); if (!clf) { std::println(stderr, "{}", clf.error().message()); return 1; } for (std::string_view t : {"I love this product!", "Terrible quality, broke after one day."}) { auto top = clf->top(t); if (!top) { std::println(stderr, "{}", top.error().message()); return 1; } if (*top) std::println(" {:9.4f} {}", r.score, docs[r.index]); 10.5139 Machine learning is a subset of artificial intelligence. -5.5301 Deep learning uses neural networks with many layers. -11.1001 The weather today is sunny. The scores are logits, not probabilities. What matters is the ordering and the size of the gap between one document and the next. Ranked gives back an index into your input rather than a copy of the text, so whatever IDs, URLs and permissions came with your documents stay attached to them. One signature detail: rerank takes std::span rather than string_view, because the C call underneath needs an array of null terminated pointers and a string_view does not promise one. The same numbers in every language Those figures are not specific to C++. The reranker scores 10.5139, -5.5301 and -11.1001 are the same values the C# document search post prints, and the similarity scores are the ones in Semantic Search in C#. That is the reason a C ABI is the right shape here. There is one engine and one set of kernels, and the C++ header is a wrapper over the same entry points the C#, Go and Python packages call. There is no second implementation to drift from the first. Choosing a model Model Dimensions Input limit Notes minilm-l6-v2 384 256 tokens Default. Fast, good quality per byte mpnet-base-v2 768 384 tokens Higher quality, slower nomic-embed-text 768 8192 tokens Long documents, though trained at 2048 bge-m3 1024 8192 tokens Large, multilingual Mind the input limit column. minilm-l6-v2 reads 256 tokens, roughly 900 characters, and silently drops the rest: no error, no warning, just a vector computed from the part it saw. The cross-encoder has its own limit, reading query and document as one sequence capped at 512 tokens. If your documents are longer than that, chunk them. There is a measurement of what the truncation costs in Your MiniLM Embeddings Are Probably Truncating at 256 Tokens. Practical notes Threading. The handles are not individually thread safe. Give each thread its own, or serialise calls. The engine already parallelises across cores inside a single call, so one embedder will use the machine. C++23 is only needed for std::expected. Everything else in kjarni.hpp is C++20, and GCC 13 or Clang 17 and newer will build it. There is no package manager integration yet. No Conan recipe, no vcpkg port. The four commands above are the install story on Linux, and the equivalent archives for macOS and Windows are on the releases page. If you are on C++11 or C++17 Plenty of codebases are, and a header that demands C++23 is not much use to them. It is worth being clear about what the requirement actually covers: kjarni.hpp needs C++23 for std::expected, and nothing else does. kjarni.h is plain C, it is the interface every other language binding is built on, and it works from C++11 upward. The whole of the C++23 convenience is one RAII wrapper and a copy: #include "kjarni.h" #include #include #include #include namespace { struct EmbedderDeleter { void operator()(KjarniEmbedder* p) const { kjarni_embedder_free(p); } }; using EmbedderPtr = std::unique_ptr; std::vector encode(KjarniEmbedder* emb, const char* text) { KjarniFloatArray arr{}; if (kjarni_embedder_encode(emb, text, &arr) != KJARNI_ERROR_CODE_OK) { std::fprintf(stderr, "encode failed: %s\n", kjarni_last_error_message()); return {}; } std::vector out(arr.data, arr.data + arr.len); kjarni_float_array_free(arr); // copied out, so release the engine's buffer return out; } } // namespace int main() { KjarniEmbedderConfig cfg = kjarni_embedder_config_default(); cfg.model_name = "minilm-l6-v2"; cfg.quiet = 1; KjarniEmbedder* raw = nullptr; if (kjarni_embedder_new(&cfg, &raw) != KJARNI_ERROR_CODE_OK) { std::fprintf(stderr, "could not load model: %s\n", kjarni_last_error_message()); return 1; } EmbedderPtr embedder(raw); const std::vector question = encode(embedder.get(), "How do I get my money back?"); const std::vector related = encode(embedder.get(), "What is your refund policy?"); const std::vector unrelated = encode(embedder.get(), "The weather in Reykjavik is unpredictable."); if (question.empty() || related.empty() || unrelated.empty()) return 1; std::printf("related: %.4f\n", kjarni_cosine_similarity(question.data(), related.data(), question.size())); std::printf("unrelated: %.4f\n", kjarni_cosine_similarity(question.data(), unrelated.data(), question.size())); } g++ -std=c++17 hello17.cpp -I. -L. -lkjarni_ffi -Wl,-rpath,'$ORIGIN' -o hello17 related: 0.5510 unrelated: -0.0630 The same numbers as the C++23 version, because it is the same engine underneath. That compiles unchanged under -std=c++11 as well. Two rules cover the manual memory. Any KjarniFloatArray you receive is freed with kjarni_float_array_free once you have copied what you need out of it, and any handle is freed with its matching _free function. Wrapping the handle in a unique_ptr with a custom deleter, as above, means the second rule takes care of itself on every return path. Error text comes from kjarni_last_error_message(), and it reports the most recent failure process wide, so read it immediately after the call that failed rather than saving it up. Compared to the alternatives libtorch ONNX Runtime Kjarni Install download SDK, match ABI package plus model conversion one archive Model format TorchScript .onnx, converted HuggingFace safetensors and GGUF directly Extra entries in ldd many its own stack none beyond libc Tokenizer bring your own bring your own included GPU CUDA toolkit CUDA or DirectML WebGPU, no toolkit The trade is scope. libtorch runs anything expressible in TorchScript. Kjarni runs the model families it implements: BERT-style encoders, cross-encoders, Llama-family decoders, T5, BART and Whisper. For an arbitrary research model, convert it and use ONNX Runtime. For embeddings, classification, reranking or chat inside a C++ program that has to ship somewhere, one archive with no toolchain is a smaller problem than either. Getting it kjarni.ai - documentation and the rest of these posts Releases - the archive used above, for Linux, macOS and Windows GitHub - source, including kjarni.hpp and the C++ examples NuGet - the same engine from C# npm - the same engine in the browser, via WebAssembly Go module - the same engine from Go crates.io - the Rust engine itself Other Resources Semantic Search in C# - The same engine and the same vectors, from .NET Build a Document Search Engine in C# - Keyword and semantic retrieval combined, with reranking Why I Built a Native ML Inference Engine in Rust - What is underneath all of this ML from the Command Line - The same models as a UNIX tool Your MiniLM Embeddings Are Probably Truncating at 256 Tokens - Measured, with the cost