An Architecture to Run AI Agents Safely and Efficiently in a Linux VM on Apple Silicon

I'm the developer of Velo Workspaces, a native macOS app for disposable Linux and macOS VMs on Apple Silicon. This post is about a specific problem I kept hitting while building it, how to let an AI agent run generated code safely in a VM without giving up the GPU acceleration? I ended up with the architecture AI Bridge based on vsock, and supported by real benchmark numbers. The problem If you want to run an AI coding agent locally on your Mac today, you may choose one of the following two options. Option A: run it directly on your Mac. Most agent frameworks, like LangChain, AutoGen, SWE-Agent and Open Interpreter, execute LLM-generated code via subprocess or exec() with full access to your filesystem, network, and credentials. Docker's own engineering blog has documented real AI coding agent security incidents resulting from exactly this. A recent dev.to post lays out the same concern plainly: AI Agents Run Unsandboxed Code — How to Fix It. Indirect prompt injection makes this worse than it sounds, and a malicious instruction hidden in a file the agent reads can trigger commands with your full permissions, not just the ones you asked for. Option B: run it in a VM or container for isolation. This is the "correct" answer on paper, and it runs straight into a wall specific to Apple Silicon: Apple's Virtualization.framework does not expose the host GPU to a Linux guest like the way it does for macOS guests. This isn't a rumor, it's confirmed directly by Apple's own container team in their public repos, in response to people asking for exactly this: GPU passthrough availability? and Paravirtualized Graphics via virtio-gpu. A macOS guest gets a purpose-built Metal-based GPU driver; a Linux guest gets virtio-gpu, a paravirtualized 2D device with no path to the host's compute GPU. Running a 7B model inside the VM is dramatically slower than the same model on the host. Root cause: on Apple Silicon, "safe" (inside a VM) and "fast" (on the GPU) currently pull in opposite directions, because the GPU only exists on the host side in the case of Linux VMs. The solution: split the agent from the model The fix isn't to pick a side, it's to stop treating "run the agent" and "run the model" as a same problem. The dangerous part (executing LLM-generated code) and the expensive part (running the model) don't have to live in the same place: The agent runs inside the isolated, disposable Linux VM, and when you're done, the whole VM (and anything it wrote to disk) is gone. The model runs on the host, on Ollama or MLX, with full native Metal GPU access. AI Bridge connects the two over a direct vsock channel, so the agent inside the VM talks to the model exactly like it would talk to any local server. ┌───────────────────────────────────────────────────────────────────┐ │ macOS HOST │ │ │ │ Ollama / MLX Engine ───► Storage: ~/.ollama/models │ │ (Native Metal GPU) (single 10GB–50GB copy) │ │ ▲ │ │ │ Velo Workspaces AI Bridge │ │ │ proxies to localhost: │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ LINUX VM │ │ │ │ socat 127.0.0.1: ⇄ vsock CID 2 │ │ │ │ Env: OPENAI_API_BASE=http://127.0.0.1:/v1 │ │ │ │ Dev apps / Docker containers / VS Code / Python │ │ │ └─────────────────────────────────────────────────────┘ │ └───────────────────────────────────────────────────────────────────┘ A relay bridges a local TCP port inside the guest to the host's model-server port over vsock (CID 2 is the host, in Apple's vsock addressing). The port isn't hardcoded to Ollama, so the bridge works for MLX or any other OpenAI-compatible local server on whatever port it's actually listening on. From inside the VM, you set OPENAI_API_BASE (or whatever your tool calls it) to http://127.0.0.1:/v1 and every OpenAI-compatible client will work, such as Open Interpreter, LangChain, or a raw curl. No code changes need to be done, and no awareness that a VM boundary exists at all. The value this targets specifically: get real isolation for the part that's actually risky (arbitrary generated code execution) without paying a GPU-virtualization tax for the part that's actually expensive (inference), and without copying a 10–50GB model file into every VM you spin up. The results I wanted to know how close this actually gets to "no tax" so I benchmarked it properly rather than asserting it. Setup: Host: Mac mini, Apple M4, 10 cores, 16GB RAM, macOS 26.6.2. Guest: Ubuntu Server 26.04 (minimized), 4 vCPU, 4GB RAM. Model: qwen2.5-coder:7b (Q4_K_M) served by Ollama on the host. Bridge tuned with a larger socat buffer and nodelay set. Both sides on Python 3.12. Three tests, scripts here: ai_perf_test.py — single-request time-to-first-token and throughput, 5 runs, deterministic ai_load_test.py — 8 concurrent clients (OLLAMA_NUM_PARALLEL=2) oi_perf_test.py — full agentic-loop latency via Open Interpreter: prompt → generated code → execution → result, 5 runs × 3 tasks One methodology lesson that mattered more than anything else: the first time I ran these, I switched between the host test and the VM test without being disciplined about fully stopping the VM and restarting Ollama between them. The agentic-loop numbers were a mess on host, wildly bimodal, some iterations 10x slower than others. The fix was to stop all VMs, stop Ollama, restart ollama serve fresh, before every single run for both host and VM tests. Once I did that consistently, the bimodal pattern disappeared completely. A leftover process from the previous run contending for CPU/scheduling with the one actually being measured was enough to produce numbers that looked dramatic but meant nothing. All results below are from the controlled re-run. Raw single-request speed — this is the actual "tax," and it's small Host (loopback) VM (AI Bridge) Delta Time to first token 56 ms 59 ms +6% Throughput 22.09 tok/s 21.78 tok/s −1% These numbers reveal the actual cost of this setup: exactly one network hop over vsock. Because the model stays firmly anchored to the host's hardware, a single-digit percentage drop is practically as close to a "zero virtualization tax" as physics will allow. You aren't paying a penalty for slow GPU emulation simply because there is no GPU emulation happening at all. 8 concurrent clients — the VM won Host (loopback) VM (AI Bridge) Total wall time (8 clients) 15.04s 12.73s Avg per-client time 9.43s 8.28s The VM finished ~15% faster via the bridge！ Initially, I thought this was purely about CPU isolation on the host, the benchmark client and the Ollama server are fighting for the exact same physical cores. In the VM, the client gets its own dedicated vCPU slice. But digging into the network architecture reveals a deeper mechanical advantage. When you run 8 concurrent Python clients natively on macOS, Python relies on kqueue for async I/O. It works, but it isn't quite as efficient as Linux's epoll event loop for handling rapid socket creation. More importantly, when the VM routes traffic over vsock, it lands in Velo Workspaces's native Swift app on the host, which uses Apple's Grand Central Dispatch (GCD) to proxy the connection to Ollama. GCD acts like a highly optimized shock-absorber at the Apple Silicon hardware level. It batches and streams those requests to localhost far more efficiently than a raw Python script hammering the port directly. You are effectively putting an enterprise-grade reverse proxy in front of the model. Real agentic loop — the VM won consistently, by a believable margin Task Host mean Host std dev VM mean VM std dev Math computation 6.45s ±1.16s 5.38s ±1.28s System info 7.14s ±1.12s 5.82s ±1.07s File I/O 9.08s ±1.22s 6.56s ±1.38s The VM ran 17–28% faster, with both sides showing tight, comparable variance and no bimodal weirdness once background contention was controlled for. Seeing the VM consistently beat native macOS on tasks like File I/O and System Info might look counter-intuitive, but it makes perfect sense when you look at how agent frameworks actually execute code. Open Interpreter relies heavily on spawning background terminal shells and Python REPLs, writing temporary files, and reading stdout. Every time a script does this natively on macOS, Apple's background security daemons (XProtect, Gatekeeper, and Endpoint Security) briefly intercept the execution to scan for malicious behavior. It is a tiny delay that compounds massively over the course of an agentic loop. Inside the minimized Ubuntu VM, none of that desktop antivirus overhead exists. Linux's process creation (fork()) is fundamentally lighter than macOS's (posix_spawn), and writing to a raw ext4 filesystem is instantaneous. The vsock bridge penalty is so remarkably small that it gets completely swallowed by the sheer execution speed of Linux. How close to the ideal outcome? The goal was "near without virtualization tax" — not literally zero, since a network hop is unavoidable, but close enough that it isn't the thing you'd notice. A single-digit-percent difference on the raw inference path is that outcome. What I didn't expect going in was that the system-level numbers — concurrent load, real agentic workflows — would come out ahead in the VM rather than merely close. The isolation you get for safety turns out to also isolate you from host-side resource contention, which is a genuine bonus on top of the safety case, not just a wash. Biggest lesson overall: environment hygiene between benchmark runs matters as much as the architecture you're testing. A leftover process from the previous run can produce numbers dramatic enough to completely mislead you about which side actually won. Scripts are at github.com/xgz2025/veloworkspace-tools if you want to reproduce this on your own hardware or model. Velo Workspaces, which implements this architecture as AI Bridge, is at veloworkspaces.com.