Schema Evals for a One-Key Node.js Text Classification API

Short answer: replace separate provider clients with one scoped classification API key and a strict JSON contract, keep chat completion model routing behind aliases, and require every route change to pass the same eval suite. The deciding trade-off is control versus convenience. A Node.js application can submit the same label request every time while an internal gateway selects a chat completion model. The important part isn't the shared key. It is the eval suite and output schema that make a route change measurable and reversible. This is a small architecture, on purpose. If an application currently has separate OpenAI, Claude, and Gemini branches, replacing the SDK switch statements is only half the job. A useful migration also normalizes labels, refusal handling, retry ownership, observability, and the criteria for promoting a model. Otherwise, one API key merely hides three behaviors behind one URL. How should a Node.js text classification API handle chat completions and model routing? Treat the Node.js side as a caller of a stable internal classification contract. It sends an item identifier, text, taxonomy version, and routing profile such as interactive or batch. A Python classification worker turns that request into a chat completion, asks for a strict JSON object, validates the response, and returns a provider-neutral result. The routing layer maps the profile to a concrete model. Application code never receives a provider model ID and never chooses one. The flow is straightforward: the product emits an item, the classifier builds a versioned prompt, the gateway routes the request, the classifier validates the returned label, and an event records the result plus the versions that produced it. That last record is what lets a team replay yesterday's examples against tomorrow's candidate route. Without it, model routing is guesswork — quick to configure, impossible to defend. Boundary Credential shape Best fit Main limitation Direct provider clients One key per provider Workloads that need native controls Application owns divergent clients and response handling Shared compatibility gateway One scoped gateway key Closed-schema classification across routes Common contract may omit native features Internal inference service One internal service credential Restricted data or self-managed models Team owns deployment and capacity Keep the output contract deliberately boring. For a support-ticket example, it might contain label, confidence, and reason, with label restricted to a closed set. Confidence is useful for queues and review thresholds, but it should not be treated as calibrated probability until an eval demonstrates calibration on the application's own data. I'm not sure a model's self-reported confidence is useful at all for some taxonomies; a held-out confusion matrix resolves that question better than intuition. Run the contract before debating routes The following worker is a compact implementation of that boundary. It uses one endpoint and one credential, while the routing_profile chooses an alias rather than a vendor model name. The endpoint is supplied as configuration because the code should work with an internal gateway, a managed compatibility layer, or a service the team operates itself. from __future__ import annotations import json import os import urllib.error import urllib.request from dataclasses import dataclass from typing import Literal Label = Literal["billing", "bug", "feature_request", "other"] ALLOWED_LABELS = {"billing", "bug", "feature_request", "other"} ROUTES = { "interactive": "classifier-fast", "batch": "classifier-throughput", } @dataclass(frozen=True) class Classification: label: Label confidence: float reason: str def validate_result(raw: object) -> Classification: if not isinstance(raw, dict): raise ValueError("E_CLASSIFY_SHAPE: result must be an object") label = raw.get("label") confidence = raw.get("confidence") reason = raw.get("reason") if label not in ALLOWED_LABELS: raise ValueError(f"E_CLASSIFY_LABEL: unexpected label {label!r}") if not isinstance(confidence, (int, float)) or isinstance(confidence, bool): raise ValueError("E_CLASSIFY_CONFIDENCE: confidence must be numeric") if not 0 dict[str, object]: if not rows: raise ValueError("evaluation set must not be empty") correct = sum(row.expected == row.predicted for row in rows) invalid = sum(not row.valid for row in rows) confusions = Counter( (row.expected, row.predicted) for row in rows if row.expected != row.predicted ) return { "examples": len(rows), "accuracy": correct / len(rows), "invalid_rate": invalid / len(rows), "confusions": { f"{expected}->{predicted}": count for (expected, predicted), count in sorted(confusions.items()) }, } Run the same frozen set against the current and candidate aliases, store raw outputs, and inspect changed predictions rather than looking only at the summary. Then shadow a small production sample without using the candidate's labels in the product. This sequence catches schema drift in the notebook, distribution drift in live traffic, and operational differences before a route starts changing user-visible tags. Batching needs its own decision. An asynchronous batch interface is a good fit for backfills and nightly tagging because the caller doesn't need an immediate label, while an interactive request needs a latency budget and bounded retries. OpenAI documents a dedicated Batch API, but adopting a provider-specific batch format creates a separate execution path that a generic chat-completion gateway may not reproduce. Keep a common result envelope and eval suite if the transport diverges. Portability lives in the contract, not in pretending every execution mode is identical. What does one API key fail to solve? One application credential reduces secret distribution and client configuration. It does not merge provider data-processing terms, regional availability, retention policies, quotas, or incident domains. The gateway becomes a critical dependency too, so its authentication, request logs, timeout budget, and access controls need the same scrutiny as any other production service. A shared key should be scoped to classification, stored in a secret manager, rotated, and kept out of browser code; “one” must never mean “used everywhere.” The catch is the common contract. It works well for short text classification with a closed JSON schema, but it can hide provider-specific controls that matter for another workload. Stick with a native provider interface when a required safety control, batch facility, data boundary, or output feature isn't represented faithfully by the shared layer. Keep separate routes when legal or organizational isolation requires separate credentials. And if text cannot leave the team's network, a hosted multi-provider gateway is not suitable; use an internal inference service and accept the capacity-planning work. Retries are another boundary. The synchronous request path should not sleep through a long series of attempts. Return a typed retryable result or enqueue the item, apply exponential backoff with jitter in one owner, and place a maximum age on the job. A 429 is operational evidence, not permission for every application instance to start its own retry loop. Invalid labels are different: retrying the same prompt against the same route may repeat the same answer, so record E_CLASSIFY_LABEL, send the item to review, and use that example to improve the next eval set. Operate classification as a data pipeline Before release, version the taxonomy, prompt, schema, route mapping, and eval dataset independently. Log those versions with the item ID, selected alias, latency, token usage, validation outcome, and final label, while excluding raw sensitive text unless retention rules explicitly allow it. Put dashboards on invalid-output rate, queue age, per-label volume shifts, and the share sent to human review. A sudden fall in feature_request may indicate a product trend, a prompt change, or routing drift; versioned events let an operator tell the difference. Deployment should move from offline replay to shadow traffic, then to a limited route allocation with an automatic rollback threshold. Reclassifying old data is a migration, so write new labels beside old ones until downstream reports have been checked. Don't silently overwrite history. For interactive calls, define a timeout and a product fallback such as “unclassified”; for batch work, make jobs idempotent and resume from item-level checkpoints. Review access to the shared credential, rehearse its rotation, and verify that the previous route mapping can be restored without an application deploy. The final production decision is compact: one classification contract, one scoped application key, model aliases owned by the routing layer, strict schema validation, and promotion by repeatable evals. This keeps provider changes inexpensive without pretending they are risk-free. More importantly, it turns model choice from a preference into an observable deployment decision. Sources https://platform.openai.com/docs/guides/structured-outputs https://platform.openai.com/docs/guides/batch