Running a Local Model Is Not the Same as Running a Local AI Service

Getting a local language model to answer a prompt is a great first step. But after experimenting with multiple inference engines on the same machine, I started running into a different kind of problem. What happens when two models need the GPU at once? What if a larger model needs to be loaded? What happens when a request is accepted but the client never receives the reply? And, perhaps most importantly: Can I actually finish a development task with this setup, rather than just have a conversation with a model? Those questions led me to Local AI Fabric (LAF). I'm not trying to build another inference engine. I'm building the coordination layer I want around the engines I already use. TL;DR Running a model and operating an AI service are different engineering problems. LAF coordinates inference runtimes, checks available resources, and exposes a common job interface. Its terminal client can be inspected without loading a model. Real inference requires a separately configured Router and model assets. The public code is a development baseline, not a one-command installation or a finished autonomous coding system. The most useful question isn't just whether the model responded. It's whether the entire development workflow can reliably make progress. Why isn't Ollama enough? Actually, for many use cases, it is. If you want to download a model, run it locally, and talk to it, an existing runtime may already give you everything you need. I use existing inference runtimes rather than trying to replace them. The problem changes when several engines share the same machine. Imagine that a small, fast model is available for everyday requests, while a larger coding model is loaded only when needed. Now the interesting questions are no longer limited to inference: Which model should handle this request? Is there enough memory available? Should the request wait for GPU resources? Can the previous model release its resources before another one loads? If something fails, can the system report the failure without silently choosing a different model? That's the layer I'm interested in. What Local AI Fabric does The current public LAF baseline includes a Router, inference profiles, a terminal client, and an optional NPU classification service. The Router provides a shared job interface and performs resource checks before accepting work. It also bounds queued GPU work and checks resource release during model switching. The current generation profiles are: flash coder-next gpt-oss-120b These are development profiles, not a promise that all three models can run on an ordinary PC. The current implementation uses a Strata Flash path and Ollama-backed large-model profiles. Ollama remains responsible for inference; LAF coordinates work around it. The optional NPU service has a narrower purpose: fixed-shape classification. It is not a general-purpose chat model, and its existence doesn't mean that LAF already has an autonomous NPU-driven orchestration system. I want those boundaries to remain explicit. A missing NPU service should not, by itself, make GPU-based generation unusable. A terminal client should be easy to inspect I spend a lot of time in terminals. For a local AI service to be useful in my workflow, I don't want every conversation to start with manually assembling JSON requests. LAF has a small terminal client, and there is a useful distinction between checking that the client works and checking that inference works. If you already have the LAF source checkout, you can run the following from its root directory on Linux with Python 3.10 or newer: ./laf --help printf '/help\n/quit\n' | ./laf These commands do not require a running model. The first displays CLI help. The second starts the chat interface, displays its built-in help, and exits. They do not perform inference. Both checks were executed successfully against the public baseline used for this article. That's a small test, but a useful one: it separates basic client behavior from the much more complicated problem of provisioning a model service. Connecting to an existing Router Once the Router and its models have been provisioned separately, the client provides commands such as: ./laf status ./laf chat --model flash status queries the existing Router. The chat command selects the Flash profile. Neither command installs an inference engine, downloads model weights, or starts the Router for you. These commands are documented in the source, but I did not rerun live inference as part of preparing this article. For prerequisites and socket configuration, see the Router documentation and chat CLI guide. This is not yet a clean-machine Quick Start. I don't want to pretend otherwise. Giving an AI access to tools changes the problem A conversation is one thing. Letting a model interact with a development workspace is another. For example, I want the client to support a deliberately limited form of filename search. That sounds simple, but it introduces an important boundary. A model can say that it found a file. That is not the same as the client actually searching the filesystem and returning a verified result. The current client keeps workspace access opt-in and limits its filename search to names within the selected workspace. That is not permission to read arbitrary file contents, modify files, or execute shell commands. I want to preserve that distinction as the tooling grows. Generated text is not an observed tool result. It's an easy rule to state, but an essential one when an AI begins participating in real development work. A successful response doesn't mean the system succeeded There are several different kinds of success in a local AI system. A model might return a perfectly reasonable answer while the next request has no useful context. A client might lose its connection after the service has already accepted a request. A test might pass against a simulated provider while the actual inference runtime remains misconfigured. Those aren't interchangeable outcomes. They require different evidence. A unit test can verify a refusal or recovery path without loading a large model. A hardware test can establish what happened on one configured machine. Neither proves that a fresh installation will work for everyone. And neither proves that an AI can complete a software development task. That last question matters most to me. What's next? One current limitation is that the public baseline uses three fixed generation profiles. That works as a starting point for development, but it shouldn't become an assumption about everyone else's hardware. In particular, I haven't established general support for machines with 8–16 GB GPUs. Adding configurable model providers is useful, but configuration alone doesn't prove that a model will fit in memory or run acceptably. An open integration proposal extends the design with configurable providers, optional Flash, a separate Router MCP interface, and SDK tooling. At the time of this article's review, that proposal was still unmerged. It should not be mistaken for completed Codex integration. The larger goal is a development loop that can: Receive a development request. Use only explicitly permitted tools. Make a change. Run appropriate tests. Diagnose failures. Apply a justified correction and verify the result. A working chat prompt is only the beginning. Three questions I want to keep separate As I continue developing LAF, I want to be able to answer three questions independently: 1. Did the client work? Could it start, accept input, report status, and handle errors correctly? 2. Did the model run? Did the configured inference engine actually execute the request on the intended hardware? 3. Did the development task finish? Did the system produce the required change, verify it, and leave evidence that the result is correct? It's tempting to treat the first successful reply as proof of the whole system. I don't think that's enough. For me, the interesting part of local AI development begins after that first reply. Project: Local AI Fabric on GitHub LAF does not bundle model weights or vendor SDKs. Its first-party source uses the custom KISS Open License v1.0; third-party engines, models, and SDKs remain subject to their own terms. This article was prepared with ChatGPT assistance using the project's public source and recorded development results. It describes a development baseline, not a completed product release.