I Tried Needle2 for Local Tool Calling. I Ended Up With llama.cpp + Granite

I’ve been working on a small experiment around a semantic shell. The basic idea is that the user types something like: copy report.pdf to backup and a small local model maps that to a known tool: filesystem.copy The shell then takes over. It validates arguments, asks for missing ones, shows confirmation UI when needed, and finally executes the operation. The model is not supposed to generate shell commands. It only needs to understand intent. That sounded like a very good fit for Needle2. Needle2 is tiny, focused on tool calling and structured extraction, and designed to run locally. On paper it looked almost purpose-built for what I needed. So I tried it. The first version worked. Then I started adding more realistic cases. And that’s where things became more interesting. Five tools is very different from fifty Most examples of tool-calling models are some variation of: turn on the kitchen light with two or three tools available. That’s fine as an API example, but it doesn’t tell you much about how the model behaves in a real application. Even my first filesystem package already had things like: filesystem.copy filesystem.move filesystem.delete filesystem.create_file filesystem.create_directory filesystem.list_directory filesystem.navigate filesystem.current_directory filesystem.find Needle2 uses retrieval when the number of tools gets larger. It selects a small candidate set and then resolves the tool from there. That makes sense. But it also means there are really two decisions now: Which tools should be considered? and then: Which of those tools is the right one? If the correct tool does not make it into the shortlist, the second stage never gets a chance. For many applications that may be perfectly acceptable. In my case I wasn’t comfortable relying on it without knowing more about how often it happens. I started thinking about bypassing retrieval entirely by dividing tools into groups of five and running all groups. Needle is small enough that this is not completely ridiculous. Something like: group 1 -> result group 2 -> result group 3 -> result ... then take the strongest candidate. But immediately another question appears. Are confidence scores from different candidate groups actually comparable? Maybe they are. Maybe they are not. I couldn’t find enough information to feel confident about treating them as globally calibrated scores. That does not make Needle bad. It just means that once you move away from the demo path, you start needing answers that are not obvious from the examples. Tool descriptions matter a lot Another thing I learned quickly was how sensitive a small model is to tool descriptions. A description like: Moves files from one place to another. is not enough if the neighboring tools are: filesystem.copy filesystem.navigate filesystem.rename You need to be explicit. For example, “move” should probably say that the original item no longer remains at the source, and that this is not the same thing as changing the shell’s current directory. The descriptions start looking less like documentation and more like a tiny classification dataset written in English. That was workable, but it changed how I thought about the model. It is less “small agent” and more “probabilistic semantic parser with a constrained vocabulary.” Which is actually fine for this use case. Then I tried the C++ runtime For the shell itself I wanted native integration. No Python process on the side, no extra service, and no awkward lifetime management. The C++ library looked like the right way to do it. This is where I hit the biggest practical problem. I was developing on an Intel Mac. The Cactus Engine code contains ARM NEON includes and intrinsics in many places. Not just one isolated backend file, but spread through engine and kernel code. That surprised me. If ARM is the main target, this is understandable. iOS, Android, and Apple Silicon are all important platforms. But from my side it meant x86 support was not just a matter of enabling another optimized backend. I ended up spending time patching things simply to answer a much more basic question: Is Needle good enough for my use case to justify carrying this dependency? At that point I decided to try a different route before spending more time on the port. llama.cpp and Granite So I integrated llama.cpp. Then I tried Granite 4 350M for the same tool-calling task. I expected this to be more of a baseline than a solution. It turned out to work better than I expected. For the normal cases it selected the tools I wanted. More importantly, I tried a few deliberately awkward inputs where no good tool really existed, and instead of forcing a call it simply failed to find one. For my application, that is good behavior. I would much rather see: No suitable tool found. than a confident guess. The shell can recover from that. It can ask the user to rephrase, narrow the scope, or choose between two possible interpretations. A wrong tool call is much harder to recover from once you start dealing with destructive operations. This changed the architecture less than I expected The funny part is that switching models did not really affect the shell design. The important part was already outside the model. The flow is roughly: user input ↓ intent resolution ↓ known semantic capability ↓ argument validation ↓ ask for missing arguments ↓ policy / confirmation ↓ execution If the user says: copy file the model only needs to resolve: filesystem.copy The shell can notice that source and destination are missing and show proper selectors. If two tools are plausible, for example copy versus move, the shell can show two buttons. There is no need to force the model to solve every ambiguity. That ended up being one of the more useful conclusions from the experiment. A small local model does not need to be brilliant if the surrounding system is designed to handle uncertainty. The part I wish documentation focused on more I still think Needle2 is interesting. What I missed was more information about the edges. Not another: turn on the light example. I wanted answers to things like: how accuracy changes as the tool catalogue grows; how often retrieval drops the correct tool; whether confidence remains comparable across different candidate sets; how similar tools should be described; what happens on no-match input; how confidence behaves on incorrect calls; how much fine-tuning helps with ambiguous tools; what the realistic native-platform support is. Those details are not as flashy, but they are what decides whether something works in a real system. The same applies to structured extraction. Showing that a model can extract an invoice total is useful. Showing how it behaves when there are subtotal, tax, total, previous balance, OCR errors, and two dates on the same page is much more useful. Where I am now For the moment I’m continuing with: llama.cpp Granite 4 350M typed semantic tools deterministic argument resolution small contextual UI policy-controlled execution That does not mean Needle2 is a bad model. It may be an excellent fit for the platforms and use cases Cactus is targeting. It just stopped being the obvious choice for mine. And that was probably the main thing I got from the experiment. A component can look almost perfect from the feature list and still be the wrong foundation once you start testing the boring parts: portability, failure modes, ambiguity, scaling, and integration cost. For this kind of software, I’m becoming much more interested in how a model fails than in how impressive its best demo looks.