Part 2: 4-Gate AST Verification & Static Call Inventories (4B Edge NPU vs. Frontier)

In Part 1, we introduced SpaghettiLens-AI (v0.5.0), showing how running a compact 4B parameter model (qwen3-4b-instruct-2507) on a local Snapdragon NPU exposed multi-tier orchestration weaknesses in agentic frameworks. Following community feedback—specifically noting that models of different scales may pass the same routing tests while altering how they chain tool calls—we packaged our verification gates and a static call-inventory extractor into the repository (dev branch). This article documents our two public synthetic fixtures, our 4 scoped verification gates, an AST-based static call inventory extractor, and an exploratory comparison between our local 4B NPU setup and Google's gemini-3.8-flash. 1. What Is Reproducible Offline vs. Live Because readers evaluating this repository will not generally have access to a local Qualcomm Snapdragon X Elite NPU (npurun) and may not wish to run 100 paid cloud API evaluations, we split our benchmark scripts into two distinct modes: Offline Gate Verifier & Telemetry Replay (benchmarks/run_eval.py --mode dry-run): Runs in list[str]: known_callables = ( set(getattr(agent, "tools", {}).keys()) | set(getattr(agent, "managed_agents", {}).keys()) | {"final_answer"} ) inventory: list[str] = [] for step in agent.memory.steps: if isinstance(step, ActionStep) and getattr(step, "code_action", None): try: tree = ast.parse(step.code_action) calls = [ node for node in ast.walk(tree) if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id in known_callables ] calls.sort(key=lambda n: (getattr(n, "lineno", 0), getattr(n, "col_offset", 0))) for call_node in calls: inventory.append(call_node.func.id) except SyntaxError: pass return inventory Scope of this Extractor: Sorting ast.Call nodes by (lineno, col_offset) produces a static source-order call inventory, not a dynamic runtime execution trace. If a model writes conditional branches, loops, or helper functions inside step.code_action, static AST inspection will list calls in source order regardless of whether a branch executed at runtime. Observed Step-Chaining Difference on legacy_billing.pl In our live run of benchmarks/run_live_cloud_trace.py with gemini-3.8-flash: Step 1 (2,387 input tokens | 142 output tokens): extracted_schema = extract_sql_schema(legacy_code=legacy_code) print(extracted_schema) Step 2 (5,145 input tokens | 224 output tokens): safe_code = synthesize_safe_fastapi_service(schema_json=extracted_schema) print(safe_code) final_answer(safe_code) Extracted Static Call Inventory: ['extract_sql_schema', 'synthesize_safe_fastapi_service', 'final_answer'] across 2 steps (7,532 cumulative input tokens, 366 output tokens). In this experiment, gemini-3.8-flash chained synthesize_safe_fastapi_service(...) and final_answer(...) inside a single Python block in Step 2. By contrast, under our local 4B configuration—where Jinja2 templates restrict agents to one tool call per step to avoid exceeding the 4,096-token context limit—the model invokes the synthesis tool in Step 2, observes the output, and calls final_answer in Step 3. Furthermore, the unconstrained Step 2 input size (5,145 tokens) illustrates the prompt-growth pressure that necessitated sliding-window pruning on our 4,096-token local setup. 5. Conclusion From this exploratory evaluation, we draw two practical takeaways for our codebase: Constrained Contexts Surface Prompt and Routing Overhead Early: Operating under a 4,096-token configured context limit on local 4B hardware forced us to confront prompt growth, truncated JSON outputs, and multi-tier routing visibility—issues that did not cause immediate failures when running the same task against a larger-context cloud model in our test. Combine Scoped AST Gates with Static Call Inventories: Even when two configurations produce equivalent final code, inspecting the AST of step.code_action alongside scoped syntax, SQL, and Decimal checks reveals how different models and prompt constraints alter multi-step tool chaining. Repository and benchmark scripts: github.com/giuseppecamiolo-dev/spagnettilens (dev branch).