Queue Depth Is the Real Cloud Trigger: A Secret-Class Overflow Router

On a shared lab workstation, five coding agents waited behind a single local inference process during a Friday merge window. Each request looked cheap in isolation, yet the queue wait grew longer than a typical remote round trip. A well-meaning teammate then pointed every worker at a hosted endpoint and watched repository snippets, including a staging token, leave the machine. The failure was not local inference itself; it was the missing overflow policy that could distinguish secret-bearing context from ordinary code questions. Contention breaks the local-first story A local model still serializes work when the runtime exposes a single generation slot to every client. Agent frameworks hide that slot behind async wrappers, so developers believe requests overlap when they only enqueue. Measured time-to-first-token then includes every earlier job, which makes a nearby GPU look slower than a distant server. Privacy goals remain valid, but they need a router that treats queue depth as a first-class signal rather than a footnote. Secret-bearing prompts should not follow the same overflow rule as a public README question. A staging API key inside a stack trace is a different object from a question about Python itertools. The proposed workflow below classifies the payload first, then consults queue depth, then checks whether the network is even available. Only the non-secret class may leave the machine, and only when waiting locally would exceed a declared budget. A routing contract before any endpoint The contract is deliberately small so it can sit in front of any OpenAI-compatible chat endpoint. Four fields decide the lane: secret class, local queue depth, offline flag, and a wait budget in milliseconds. The router never sends classified-secret payloads to a remote host, even when the GPU queue is painfully long. Offline machines skip overflow entirely and either wait locally or fail closed, depending on the operator policy encoded in the decision record. This article does not claim measured speedups, token quotas, or hardware ratings for any vendor. The numbers in the harness are placeholders that an operator must replace with probes from their own runtime. The useful part is the order of checks, because reversing classification and overflow is how tokens leak during an incident. Teams that already run local-first agents can drop the router in front of existing clients without rewriting tool loops. Artifact: a proposed secret-class overflow router The following Python module is a proposed harness, not a report of production measurements on a named model. Operators should replace the secret patterns with detectors that match their repositories and secret scanners. The default wait budget is a labeled placeholder, not a latency benchmark from a lab run. Step 1 — Declare the decision record The record keeps routing explainable when an incident review asks why a prompt left the workstation. Each field is serializable JSON so logs can be grepped without decoding vendor traces. Queue depth is an integer from the local runtime, not a guessed load average from the operating system. The wait budget is compared against queue_depth * estimated_ms_per_job, which the operator must supply from observation. # proposed harness — unexecuted example, not a production benchmark from __future__ import annotations from dataclasses import dataclass, asdict from enum import Enum import json import re from typing import Callable, Mapping class SecretClass(str, Enum): SECRET = "secret" NON_SECRET = "non_secret" UNKNOWN = "unknown" class Lane(str, Enum): LOCAL_WAIT = "local_wait" LOCAL_FAIL_CLOSED = "local_fail_closed" REMOTE_OVERFLOW = "remote_overflow" @dataclass(frozen=True) class RouteDecision: secret_class: SecretClass queue_depth: int offline: bool wait_budget_ms: int estimated_wait_ms: int lane: Lane reason: str def to_json(self) -> str: payload = asdict(self) payload["secret_class"] = self.secret_class.value payload["lane"] = self.lane.value return json.dumps(payload, sort_keys=True) Step 2 — Classify the prompt before any network call Classification must run on the workstation, because a remote classifier would receive the secret in order to decide that it is a secret. The patterns below are a starting list for source-shaped agent traffic, not a substitute for a dedicated secret scanner. Unknown class fails closed and stays local, which is slower and safer than a false non-secret label. Operators should extend the pattern list from their own incident tickets rather than copying a public gist blindly. SECRET_PATTERNS = ( re.compile(r"(?i)api[_-]?key\s*[=:]\s*['\"][^'\"]+['\"]"), re.compile(r"(?i)authorization:\s*bearer\s+\S+"), re.compile(r"(?i)-----BEGIN (RSA |OPENSSH |EC )?PRIVATE KEY-----"), re.compile(r"(?i)(aws_secret_access_key|x-amz-security-token)\s*[=:]",), re.compile(r"ghp_[A-Za-z0-9]{20,}"), re.compile(r"(?i)postgres(ql)?://[^\s:]+:[^@\s]+@"), ) def classify_prompt(text: str) -> SecretClass: if not text or not text.strip(): return SecretClass.UNKNOWN for pattern in SECRET_PATTERNS: if pattern.search(text): return SecretClass.SECRET # proposed heuristic: unknown beats optimistic non-secret on binary blobs if "\x00" in text: return SecretClass.UNKNOWN return SecretClass.NON_SECRET Step 3 — Probe queue depth and liveness locally Queue depth should come from the inference server that actually holds the generation lock. A process listing of GPU utilities is a weak proxy, because the lock may live in a user-space queue that nvidia-smi never sees. The probe below expects a tiny local status endpoint; replace the callable if the runtime only exposes a file or a mutex. Offline detection is a separate probe so a dead overflow host cannot masquerade as a healthy remote lane. class ProbeError(RuntimeError): pass def read_queue_depth(status_get: Callable[[], Mapping[str, object]]) -> int: try: payload = status_get() except Exception as exc: # proposed: caller logs the exception raise ProbeError("local status endpoint unreachable") from exc depth = payload.get("queue_depth") if not isinstance(depth, int) or depth < 0: raise ProbeError("queue_depth missing or invalid") return depth def remote_is_reachable(health_get: Callable[[], bool]) -> bool: try: return bool(health_get()) except Exception: return False Step 4 — Route with fail-closed defaults The router applies classification first, then offline state, then estimated wait against the budget. Secret and unknown classes never take the remote lane, which keeps overflow from becoming a disguised pastebin. When the local status probe fails, the router also fails closed instead of guessing that the GPU is free. Estimated milliseconds per queued job remains an operator-supplied constant, labeled here so nobody treats it as a published benchmark. ESTIMATED_MS_PER_JOB = 4_000 # placeholder — replace from local observation DEFAULT_WAIT_BUDGET_MS = 8_000 # placeholder — not a vendor SLA def route_request( prompt: str, *, status_get: Callable[[], Mapping[str, object]], health_get: Callable[[], bool], wait_budget_ms: int = DEFAULT_WAIT_BUDGET_MS, ) -> RouteDecision: secret_class = classify_prompt(prompt) try: depth = read_queue_depth(status_get) except ProbeError: return RouteDecision( secret_class=secret_class, queue_depth=-1, offline=not remote_is_reachable(health_get), wait_budget_ms=wait_budget_ms, estimated_wait_ms=-1, lane=Lane.LOCAL_FAIL_CLOSED, reason="local_queue_probe_failed", ) estimated_wait_ms = depth * ESTIMATED_MS_PER_JOB offline = not remote_is_reachable(health_get) if secret_class in (SecretClass.SECRET, SecretClass.UNKNOWN): lane = Lane.LOCAL_WAIT reason = "secret_or_unknown_stays_local" elif offline: lane = Lane.LOCAL_WAIT reason = "overflow_unreachable_stay_local" elif estimated_wait_ms > wait_budget_ms: lane = Lane.REMOTE_OVERFLOW reason = "queue_exceeds_wait_budget" else: lane = Lane.LOCAL_WAIT reason = "local_queue_inside_budget" return RouteDecision( secret_class=secret_class, queue_depth=depth, offline=offline, wait_budget_ms=wait_budget_ms, estimated_wait_ms=estimated_wait_ms, lane=lane, reason=reason, ) A thin client can then dispatch on decision.lane without mixing policy into HTTP headers. Local wait posts to the workstation endpoint; remote overflow posts only after the lane is REMOTE_OVERFLOW. Fail-closed returns an application error that the agent can surface as a retryable local condition. Logging decision.to_json() beside the request identifier is enough for a later audit without storing the raw prompt in the same file. Step 5 — Exercise the harness with a short test plan The tests below are a plan for operators to run against fixtures, not evidence that a particular model passed them. Each case pins one field of the contract so a later refactor cannot silently reopen the remote lane. Replace the status and health callables with fakes that encode the incident the team actually fears. Keep the secret fixture out of shared logs when the suite runs in continuous integration. # proposed tests — run locally against fixtures def test_secret_never_overflows_even_when_queue_is_long(): prompt = "export API_KEY='stg-live-example'" decision = route_request( prompt, status_get=lambda: {"queue_depth": 12}, health_get=lambda: True, ) assert decision.lane == Lane.LOCAL_WAIT assert decision.secret_class == SecretClass.SECRET def test_public_question_overflows_when_budget_exceeded(): prompt = "Explain itertools.groupby with a short Python example." decision = route_request( prompt, status_get=lambda: {"queue_depth": 5}, health_get=lambda: True, ) assert decision.lane == Lane.REMOTE_OVERFLOW assert decision.reason == "queue_exceeds_wait_budget" def test_public_question_stays_local_when_overflow_is_offline(): prompt = "Explain itertools.groupby with a short Python example." decision = route_request( prompt, status_get=lambda: {"queue_depth": 5}, health_get=lambda: False, ) assert decision.lane == Lane.LOCAL_WAIT assert decision.offline is True def test_status_probe_failure_does_not_open_remote_lane(): def boom(): raise RuntimeError("connection refused") decision = route_request( "Explain itertools.groupby with a short Python example.", status_get=boom, health_get=lambda: True, ) assert decision.lane == Lane.LOCAL_FAIL_CLOSED Run the suite with the project interpreter after the module is saved beside the agent client. Command shape stays ordinary so the plan works on a developer laptop without extra cluster services. Capture the JSON decisions from a dry-run log if the team wants a before-and-after review during a merge freeze. Do not treat a green suite as permission to disable secret scanning elsewhere in the pipeline. python -m pip install pytest python -m pytest overflow_router_test.py -q Decision table for operators Secret class Queue vs budget Overflow host Lane secret any any local wait unknown any any local wait non_secret wait inside budget healthy local wait non_secret wait exceeds budget healthy remote overflow non_secret wait exceeds budget offline local wait any probe failed any local fail closed The table is the entire policy in a form that a security review can annotate. Adding a fifth signal, such as prompt length or tool-name allowlists, should extend the table rather than hide branches in HTTP middleware. If a row cannot be tested with a fixture, it does not belong in the router. Teams that need dual control can require two reviewers to change the secret pattern list and the wait budget together. When a free remote lane actually wins A free remote lane wins when the payload is already non-secret, the local queue is saturated, and the overflow host is reachable. It also wins when the workstation must stay interactive for an editor while batch agents drain elsewhere. It does not win when the prompt includes environment files, customer transcripts, or private key material, regardless of how long the GPU queue looks. Offline laptops remain local-first by construction, because the health probe returns false and the table forbids overflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. For experiments that already pass the non-secret row, MonkeyCode's free model access and free server option can serve as that overflow endpoint without standing up a second GPU. Wire it only behind the router above, and keep secret-class traffic on the workstation even if the remote lane is idle. Readers who do not want a third-party overflow host can point the same lane at any compatible endpoint they already operate. Limitations and who should not use this Regex classification will miss novel secret formats, encoded blobs, and screenshots of credentials pasted as text. False non-secret labels are more dangerous than extra local waiting, which is why unknown class never overflows in this design. Estimated milliseconds per job drift as context windows grow, so a stale constant can overflow too early or wait too long. The harness also ignores disk residency of prompt logs, which is a separate audit from routing. Teams under contractual isolation, regulated health or payment data, or air-gapped rules should not add an overflow lane at all. Student laptops without a local status endpoint should not fake queue depth as zero, because that silently disables the only contention signal. Multi-tenant CI runners should not share one classifier config across repositories with different secret shapes. If the agent can call tools that fetch production data, classify the tool output again before any later overflow, rather than trusting the original user prompt. Local-first remains the default lane in this workflow, and the remote path is a pressure valve rather than a new home for every token. Queue depth, secret class, and liveness are cheaper signals than another round of model shopping. Operators who keep those three checks in order can use a free server when it genuinely wins, and can keep secrets on the workstation when it does not. The Friday merge window still gets slower under load, but it no longer has to choose between a stalled GPU and an accidental leak.