Free Inference Should Not Gate a Merge

A free-tier completion can annotate a pull request. It cannot be the required status that lets that pull request land. The moment an unpaid, uncontracted model holds a merge vote, the CI graph stops being a control plane and starts being a suggestion box with a green checkmark glued on. This is not an argument about model quality. Quality varies by day, by queue, and by the hidden routing behind a free pool. The failure is architectural. A merge gate needs a stable principal, a fail-closed policy, and an audit trail that still makes sense after the endpoint disappears. Free inference offers none of those as a guarantee. Think of the required check as a deadbolt. Chat is a hallway conversation. Mixing them is how teams discover, after the fact, that a lockfile change shipped because a model typed approval into a comment. The analogy is blunt because the failure mode is blunt. Git hosts already know how to distinguish a non-blocking comment from a required context. Policy should use that distinction. A free pool is a crowd, not a reviewer. Completions may be served by different underlying systems, different safety filters, and different rate limits. The same prompt is not the same signer. Status checks, by contrast, are names in a protection rule. If the name is llm-review and the body is a free completion, the protection rule has been bound to weather. The pull request itself is hostile input. Titles, descriptions, commit messages, and diffs are attacker-controlled documents. Prompt injection in a PR body is not exotic. It is the default shape of the artifact under review. A model asked to approve if the tests look fine will often read the story in the description instead of the hunk that swaps a dependency. That is not a reason to ban drafts. It is a reason to keep drafts off the deadbolt. Latency makes the problem operational, not only security-shaped. Merge queues punish tail latency. Free endpoints throttle, stall, and return empty. A job that retries until the model emits approval is no longer reviewing. It is sampling. Sampling does not belong in a required check. The first red flag is a required context whose only implementation is an HTTP call to a free inference URL. If that URL returns 429, the branch cannot merge, or worse, the workflow treats timeout as success. Either interpretation is a policy bug. Timeouts on an unpaid pool are normal weather. They are not a signal about the code. The second red flag is a bot account that can dismiss reviews or write success to a required check from a free-model JSON blob. Dismissal is a privileged act. It should not be a side effect of a completion. If the workflow can both comment and approve, split those permissions. The comment token should not be the approval token. The third red flag is using the model output as the audit log. The model said it was fine is not evidence. Evidence is a test report, a signed provenance document, a human review, or a scanner with a pinned version. When the free endpoint changes behavior, yesterday's approval cannot be replayed. An auditor will ask who signed. The answer cannot be whichever replica answered. The fourth red flag is tool access from the review job: git push, gh pr merge, package publish, or cloud credentials. A draft generator that can mutate the repo is not a draft generator. Earlier articles on this DEV account treated tool access and persistence as separate boundaries. The merge gate is another boundary. It is not a place to collapse them. Exit criteria follow from those flags. Remove free inference from the merge path when the job is required to merge, when it can write a success status, when it can dismiss a review, when the completion is stored as the official rationale, or when the job can call a tool that mutates shared state. If even one item is true, the model is holding a key. Take the key back. The artifact below is a small policy module plus tests. It does not call a model. It decides whether a CI job is allowed to sign a required status, given how the completion was produced. Treat it as executable policy, not as a benchmark of any vendor. The tests are a local harness. They are not reported results from a production org. # merge_gate_policy.py from __future__ import annotations import json import os import sys from dataclasses import dataclass ALLOWED_REQUIRED = {"tests", "license-scan", "typecheck", "human-review"} BLOCKED_PROVIDERS = {"free-inference", "unmetered-pool", "anonymous"} MUTATING_TOOLS = {"git_push", "gh_merge", "publish", "apply_tf"} @dataclass(frozen=True) class ReviewJob: context: str required: bool provider: str can_write_status: bool can_dismiss_review: bool tools: tuple[str, ...] def may_sign_status(job: ReviewJob) -> tuple[bool, str]: if job.required and job.provider in BLOCKED_PROVIDERS: return False, "free inference cannot be a required context" if job.can_write_status and job.provider in BLOCKED_PROVIDERS: return False, "free inference cannot write check status" if job.can_dismiss_review: return False, "review dismissal is not a model action" if MUTATING_TOOLS.intersection(job.tools): return False, "mutating tools are outside the draft path" if job.required and job.context not in ALLOWED_REQUIRED: return False, f"unknown required context: {job.context}" return True, "ok" def sidecar_comment_allowed(job: ReviewJob) -> bool: if job.required or job.can_write_status or job.can_dismiss_review: return False return job.tools == ("post_comment",) def job_from_env() -> ReviewJob | None: raw = os.environ.get("REVIEW_JOB_JSON") if not raw: return None data = json.loads(raw) return ReviewJob( context=data["context"], required=bool(data["required"]), provider=data["provider"], can_write_status=bool(data["can_write_status"]), can_dismiss_review=bool(data["can_dismiss_review"]), tools=tuple(data.get("tools", ())), ) if __name__ == "__main__": job = job_from_env() if job is None: sys.exit(0) ok, reason = may_sign_status(job) print(reason) sys.exit(0 if ok else 1) The function returns a reason string so CI logs stay readable. A boolean without a reason becomes another silent green check. Tests pin the intended failures. # test_merge_gate_policy.py from merge_gate_policy import ReviewJob, may_sign_status, sidecar_comment_allowed def test_free_model_cannot_be_required(): job = ReviewJob( context="llm-review", required=True, provider="free-inference", can_write_status=True, can_dismiss_review=False, tools=("post_comment",), ) ok, reason = may_sign_status(job) assert ok is False assert "required context" in reason def test_sidecar_comment_on_free_pool_is_allowed(): job = ReviewJob( context="llm-draft-notes", required=False, provider="free-inference", can_write_status=False, can_dismiss_review=False, tools=("post_comment",), ) assert sidecar_comment_allowed(job) is True ok, reason = may_sign_status(job) assert ok is True assert reason == "ok" def test_mutating_tool_is_rejected_even_if_optional(): job = ReviewJob( context="llm-draft-notes", required=False, provider="free-inference", can_write_status=False, can_dismiss_review=False, tools=("post_comment", "gh_merge"), ) ok, reason = may_sign_status(job) assert ok is False assert "mutating" in reason def test_unknown_required_context_fails_closed(): job = ReviewJob( context="model-lgtm", required=True, provider="paid-contracted", can_write_status=True, can_dismiss_review=False, tools=(), ) ok, reason = may_sign_status(job) assert ok is False assert "unknown required context" in reason Run the tests with a pinned interpreter. The command is ordinary on purpose. Policy should not depend on a model's availability. python -m pip install pytest python -m pytest -q test_merge_gate_policy.py A workflow file should enforce the same split at the permission layer. The draft job gets pull-requests: write and nothing else. The required jobs never receive a model provider. The environment blob is assembled from branch protection inventory, not from the completion text. # .github/workflows/review-sidecar.yml name: llm-draft-notes on: pull_request: types: [opened, synchronize, reopened] jobs: draft-notes: runs-on: ubuntu-latest permissions: contents: read pull-requests: write checks: read env: REVIEW_JOB_JSON: >- {"context":"llm-draft-notes","required":false, "provider":"free-inference","can_write_status":false, "can_dismiss_review":false,"tools":["post_comment"]} steps: - uses: actions/checkout@v4 - name: fail closed if this job could sign run: python merge_gate_policy.py Wire the required field from the repository ruleset, not from the model. The model does not get to declare itself optional. Humans do that in the branch rules. A merge queue that cannot see this job as a required context is the correct outcome, even if the comment is insightful. Static checks already cover a large fraction of what teams hope a model will notice. Compilers, linters, license scanners, secret detectors, and the test suite are signers with pinned versions. They belong on the deadbolt. A language model does not replace them, paid or free. Human review remains the correct signer for intent. Intent lives in product language, threat models, and the reason a dependency changed. No completion, free or otherwise, should be the only reader of that layer on a production repo. A contracted inference endpoint can be a second, non-required opinion if the team wants one. Contract here means an identifiable vendor, an account, a bill, and a place to file an incident. That is a different principal from a free pool. It is still not a substitute for tests. Draft generation can stay on a free path. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are a reasonable place to render those non-blocking notes, because the job described above only posts a comment. The policy module does not import that product. Delete the name and the gate still fails closed. Hobby branches with no users, no secrets, and no required checks are not merge gates. A personal experiment that auto-merges into a private sandbox is a different shape of risk. This guide would only add friction there. Teams that have already banned generated text from the repository can skip the sidecar as well. The policy is for groups that want the draft and need a hard line in front of main. It is not a claim that free inference is useless. It is a claim that uselessness and danger are different, and the merge path is where they get confused. Do not treat the tests above as a measurement of model accuracy. They never call a model. They encode an admission-control decision. If a team needs accuracy numbers, that is a separate eval harness with pinned prompts, pinned dates, and a dataset that is not the open PR queue. A loop that asks a free model whether the loop should continue is the same bug in a different jacket. Recent discussion of agent loops keeps rediscovering that the useful part is still a conditional. The merge gate is already that conditional. Adding a free completion does not give it judgment. It gives the conditional a noisy coin and a tempting green icon. The closing move is administrative, not rhetorical. Inventory every required context on the default branch. If any context is implemented by a free inference call, demote it to a comment job the same day. Keep the free server for drafts that cannot open the lock. Readers who want that scratch path can try MonkeyCode's free model access and free server on a workflow that is forbidden from signing status.