"OpenAI-compatible" is a spectrum, not a boolean — here's an 11-check conformance suite

Two endpoints both say "OpenAI-compatible" on the tin. You point your app at the first one and everything works. You point it at the second one and everything works too — for about a week. Then a streamed tool call comes back with arguments split across chunks in a way your accumulator didn't expect, your JSON parser throws inside a retry loop, and the retry loop hammers the endpoint because the error body doesn't have the field your backoff code reads. Nothing lied to you. "OpenAI-compatible" was never a boolean. It's a surface area, and every implementation covers a different subset of it. I maintain a gateway, so I read a lot of compatibility bug reports. Disclosure: I work on daoxe, a multi-model gateway that speaks the OpenAI protocol among others. The checklist below is deliberately written so you can run it against us and against anyone else, and the script at the end doesn't know or care which endpoint you point it at. If it makes us look bad on a check, that's the correct output. The nine surfaces 1. Streaming deltas. The reference behaviour is specific: the first chat.completion.chunk carries choices[0].delta.role = "assistant" and usually empty content, middle chunks carry delta.content fragments, the last content-bearing chunk carries finish_reason, and the stream terminates with a literal data: [DONE] line. Implementations diverge on all four. Some never send the role chunk, which breaks clients that use it to open a message. Some put finish_reason on a trailing chunk with an empty delta. Some omit [DONE] entirely and just close the connection, which is fine for a client that reads to EOF and fatal for one that blocks waiting for the sentinel. 2. Tool / function calling. Two traps here. First, function.arguments is a JSON-encoded string, not an object — endpoints that "helpfully" return a parsed object break every client that calls json.loads on it. Second, in streaming mode, tool calls arrive as fragments that you reassemble by the index field, with id and function.name typically only present on the first fragment. An endpoint that re-sends id on every fragment, or that omits index when there's only one call, will work with your naive accumulator and fail the moment a model emits two parallel calls. Also check that finish_reason is tool_calls and not stop — agent loops branch on that value. 3. response_format. Three tiers, and they're commonly conflated: no support, {"type": "json_object"} (valid JSON, any shape), and {"type": "json_schema", "json_schema": {..., "strict": true}} (constrained decoding against your schema). The dangerous middle case is an endpoint that accepts the parameter and ignores it. You get prose with a code fence around it, your parser fails one request in fifty, and it looks like a model quality problem. 4. logprobs / top_logprobs. Usually the first thing a proxy layer drops, because almost nobody notices. If you do classification by comparing token probabilities, or you use logprobs for confidence gating, this is load-bearing and you should test it explicitly. 5. temperature and sampling params. Reasoning-style models reject temperature outright on some upstreams, accept-and-ignore it on others, and honour it on a third set. All three are defensible; not knowing which one you're on is not. The same applies to top_p, presence_penalty, and max_tokens vs max_completion_tokens. 6. stop sequences. Does the endpoint honour an array of stop strings? Is the stop sequence included in or excluded from the returned content (the reference excludes it)? Does finish_reason come back as "stop"? A surprising number of shims implement stop by post-truncating the full completion, which means you pay for tokens you never see — and the usage numbers will show it. 7. Usage accounting. Non-streaming responses should carry usage.prompt_tokens, completion_tokens, total_tokens. Streaming responses only include usage if you pass stream_options: {"include_usage": true}, and then it arrives in a final chunk with an empty choices array — a shape that crashes clients which assume choices[0] always exists. If you do cost attribution per request, also check whether cached-prompt and reasoning-token breakdowns survive. 8. Error-body shape. Every retry layer you've ever written parses this, and nobody tests it. The reference is {"error": {"message": ..., "type": ..., "param": ..., "code": ...}} with an HTTP status that matches the semantics. Real-world variations: a 200 with an error object in the body (fatal for retry logic — you'll cheerfully return an error string to your user), an HTML error page from an intermediate proxy, or a 500 where a 400 belonged, which turns a permanent client error into an infinite retry storm. 9. /v1/models fidelity. Does it exist, does it return {"object": "list", "data": [{"id": ...}]}, and — the part that matters — does it list the IDs your key can actually call? A catalogue endpoint that returns everything the vendor sells, rather than everything your key is entitled to, is worse than no catalogue at all, because you'll build CI checks on top of it. The script Stdlib only, Python 3.8+, no install. It runs eleven checks and prints a verdict table. #!/usr/bin/env python3 """compat_probe.py — how OpenAI-compatible is this endpoint, really? export LLM_BASE_URL=https://api.example.com/v1 export LLM_API_KEY=sk-... python3 compat_probe.py --model """ import argparse, json, os, sys, urllib.error, urllib.request BASE = os.environ.get("LLM_BASE_URL", "").rstrip("/") KEY = os.environ.get("LLM_API_KEY", "") OUT = [] def _open(path, payload=None, method="GET", timeout=90): data = json.dumps(payload).encode() if payload is not None else None req = urllib.request.Request(BASE + path, data=data, method=method) req.add_header("Authorization", "Bearer " + KEY) if data is not None: req.add_header("Content-Type", "application/json") try: return urllib.request.urlopen(req, timeout=timeout) except urllib.error.HTTPError as exc: # still a readable file object return exc def call(path, payload=None, method="GET"): resp = _open(path, payload, method) raw = resp.read().decode("utf-8", "replace") try: return resp.getcode(), json.loads(raw) except ValueError: return resp.getcode(), raw def stream(payload): payload = dict(payload, stream=True) resp = _open("/chat/completions", payload, "POST") chunks, saw_done = [], False for line in resp: line = line.decode("utf-8", "replace").strip() if not line.startswith("data:"): continue body = line[5:].strip() if body == "[DONE]": saw_done = True break try: chunks.append(json.loads(body)) except ValueError: pass return chunks, saw_done def record(name, ok, detail): OUT.append((name, "PASS" if ok is True else "FAIL" if ok is False else "PARTIAL", detail)) def main(): ap = argparse.ArgumentParser() ap.add_argument("--model", required=True) model = ap.parse_args().model ask = {"model": model, "messages": [{"role": "user", "content": "Say hi."}], "max_tokens": 24} # 1 — catalogue code, body = call("/models") ids = [m.get("id") for m in body.get("data", [])] if isinstance(body, dict) else [] record("models_endpoint", code == 200 and bool(ids), f"HTTP {code}, {len(ids)} ids, target listed: {model in ids}") # 2 — basic completion + model echo + usage code, body = call("/chat/completions", ask, "POST") msg = (body.get("choices") or [{}])[0].get("message", {}) if isinstance(body, dict) else {} usage = body.get("usage") if isinstance(body, dict) else None record("basic_chat", bool(msg.get("content")), f"HTTP {code}") record("model_echo", isinstance(body, dict) and body.get("model") == model, f"asked {model!r}, got {body.get('model')!r}" if isinstance(body, dict) else "n/a") record("usage_fields", bool(usage and usage.get("total_tokens") is not None), str(usage)) # 3 — streaming shape chunks, done = stream(ask) role = any((c.get("choices") or [{}])[0].get("delta", {}).get("role") for c in chunks) fin = any((c.get("choices") or [{}])[0].get("finish_reason") for c in chunks) record("stream_shape", bool(chunks) and role and fin and done, f"{len(chunks)} chunks, role_chunk={role}, finish_reason={fin}, [DONE]={done}") # 4 — usage on stream chunks, _ = stream(dict(ask, stream_options={"include_usage": True})) record("stream_usage", any(c.get("usage") for c in chunks), "final usage chunk present" if any(c.get("usage") for c in chunks) else "absent") # 5 — tool calling tool = {"type": "function", "function": {"name": "get_weather", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}} code, body = call("/chat/completions", dict( ask, messages=[{"role": "user", "content": "Weather in Osaka? Use the tool."}], tools=[tool], tool_choice="auto"), "POST") choice = (body.get("choices") or [{}])[0] if isinstance(body, dict) else {} calls = choice.get("message", {}).get("tool_calls") or [] args_is_str = bool(calls) and isinstance(calls[0].get("function", {}).get("arguments"), str) record("tool_calls", bool(calls) and args_is_str and choice.get("finish_reason") == "tool_calls", f"n={len(calls)} arguments_is_string={args_is_str} finish={choice.get('finish_reason')}") # 6 — json mode code, body = call("/chat/completions", dict( ask, messages=[{"role": "user", "content": "Return {\"ok\": true} as JSON."}], response_format={"type": "json_object"}), "POST") text = ((body.get("choices") or [{}])[0].get("message", {}).get("content") if isinstance(body, dict) else "") or "" try: json.loads(text); parsed = True except ValueError: parsed = False record("json_object", code == 200 and parsed, f"HTTP {code}, parses={parsed}") # 7 — strict schema schema = {"type": "json_schema", "json_schema": {"name": "r", "strict": True, "schema": { "type": "object", "properties": {"ok": {"type": "boolean"}}, "required": ["ok"], "additionalProperties": False}}} code, _ = call("/chat/completions", dict(ask, response_format=schema), "POST") record("json_schema_strict", code == 200, f"HTTP {code}") # 8 — logprobs code, body = call("/chat/completions", dict(ask, logprobs=True, top_logprobs=3), "POST") lp = ((body.get("choices") or [{}])[0].get("logprobs") if isinstance(body, dict) else None) record("logprobs", bool(lp and lp.get("content")), f"HTTP {code}") # 9 — temperature: accepted / rejected / ignored is three different worlds code, _ = call("/chat/completions", dict(ask, temperature=0.5), "POST") record("temperature", None if code == 400 else code == 200, "rejected with 400 (reasoning model?)" if code == 400 else f"HTTP {code}") # 10 — stop sequences code, body = call("/chat/completions", dict( ask, messages=[{"role": "user", "content": "Count: one two three four five"}], stop=["three"], max_tokens=48), "POST") text = ((body.get("choices") or [{}])[0].get("message", {}).get("content") if isinstance(body, dict) else "") or "" record("stop_sequences", "three" not in text, f"stop string leaked into content: {'three' in text}") # 11 — error shape on a model that cannot exist code, body = call("/chat/completions", dict(ask, model="definitely-not-a-model-xyz"), "POST") shaped = isinstance(body, dict) and isinstance(body.get("error"), dict) record("error_shape", 400