Four different things are called "replay" in our agent runtime. I read the ledger.

The short version replay means four different things in our codebase. Here they are up front: # What it is Entry point Re-executes? Cost 1 Evidence replay GET /api/v1/observe/runs/{run_id}/replay No One database read 2 Catch-up after a dropped connection GET /api/v1/runs/{run_id}/stream?last_event_id=... No One database read, then resubscribe 3 Idempotent replay of a tool call Inside the tool gateway, same idempotency key arriving twice No — returns last time's result One database read 4 Actual re-execution POST /api/v1/workflows/{id}/runs/{run_id}/replay and two other paths Yes Runs again, spends money again All four rest on one thing: the ledger that hits the database first is the authority. Not the logs. Not the event stream. Not the SSE feed scrolling in your console. The rows in the tables. That sounds unremarkable until you notice it is what lets three of those four survive a process restart. 1. Why the word needs splitting In a demo, these three sentences look like one feature: "Here's every step of that run." "Lost your connection? Refresh — the missing steps come back." "Bad answer? Hit replay." Engineering-wise they are nothing alike. The first is a read. The second is a read plus a subscription. The third is a write — it calls the model again, sends the HTTP request again, spends the money again. The cost of collapsing them into one word is a user who assumes "replay" is safe and sends two emails. So the order below goes from cheapest to most expensive. 2. The tables, and one number that shows up three times Five tables, all in one file (server/app/kernel/runtime/db/models/runs.py, 396 lines): runs one execution Run :25 run_steps one step inside it RunStep :120 run_step_tool_calls execution control for one tool call :180 run_artifacts files this execution produced RunArtifact :242 run_cost_entries usage and cost for one metered call :284 Three more cover long-running work (models/tasks.py, 98 lines): tasks, task_checkpoints, task_events. Section 7 uses them. Now the detail worth stopping on: 8192 appears three times in this ledger, and it means something different each time. Twice on the run and the step, where summaries are truncated: input_summary=input_summary[:8192] if input_summary else None, Once on a tool call result, where anything larger is offloaded to object storage and the ledger keeps a pointer, a byte count and a sha256: if len(encoded_result) > 8192: ... artifact = self.trace_writer.create_artifact( run_id=record.run_id, step_id=record.run_step_id, artifact_type="json", storage_key=storage_key, mime="application/json", size_bytes=len(encoded_result), sha256=hashlib.sha256(encoded_result).hexdigest(), meta={"kind": "tool_result", "tool_call_id": record.tool_call_id}, ) The asymmetry is deliberate. Summaries are for humans; losing the tail is fine. Tool results get reconciled and replayed; losing a byte is not fine. Section 12 covers a consequence of that asymmetry we have not handled well yet. 3. Replay #1: reassembling the evidence The cheap one. A GET: GET /api/v1/observe/runs/{run_id}/replay One sentence of behaviour: query the five record types by run id, add approvals and feedback, return the bundle. The implementation (server/app/modules/observe/application/service.py:212) returns seven keys: return { "run": run, "steps": steps, "artifacts": artifacts, "costs": costs, "approvals": approvals, "feedback": feedback, "trace_spec": to_runtrace_spec(run, steps, artifacts, costs), } Six raw record sets, plus trace_spec — the same data flattened into something you can hand to a tracing backend (kernel/runtime/runs/exporter.py:88). That spec carries two rollups alongside the timeline: usage_summary (prompt tokens, completion tokens, embeddings, reranks, milliseconds, storage bytes, requests, vectors) and charge_summary (amounts grouped by currency). Nothing here executes. No model call, no tool call, no cost. It is a database read, so you can call it at any point after the run ended, and the ten-thousandth call costs what the first one did. Every query carries tenant_id and workspace_id in its where clause — reading another workspace's ledger is closed off at the SQL level, not at a middleware you can misconfigure. 4. Replay #2: catching up after the connection drops The second-cheapest, for the "tab is open, wifi died" case: GET /api/v1/runs/{run_id}/stream?last_event_id=st_xxxx Handled at server/app/api/v1/workflow/streaming.py:401. The part that matters: if last_event_id: step_query = select(RunStep).where( and_( RunStep.id == last_event_id, RunStep.run_id == run_id, ... ) ) last_step = db.exec(step_query).first() if last_step: last_step_time = last_step.created_at known_step_ids.add(last_step.id) steps_query = select(RunStep).where( and_( RunStep.run_id == run_id, ... RunStep.created_at > last_step_time if last_step_time else True, ) ).order_by(RunStep.created_at) Look at where it reads from: select(RunStep). The database. Not an in-memory ring buffer, not a broker offset. That choice buys a specific property: you can reconnect an hour after the run finished, hand over your last_event_id, and still get the steps you missed. An in-memory buffer cannot do that — a restart empties it. A broker can, but then you need a broker. The SSE id: field is the step's primary key (streaming.py:432), so the Last-Event-ID that browsers resend automatically is already a row id in the ledger. No second cursor scheme to keep in sync. One more detail worth borrowing: that query sets populate_existing=True, with a comment explaining why — the execution side writes from its own session, so this tailer has to bypass anything its own session cached earlier. That is the kind of line nobody can reconstruct three months later without the comment. 5. Replay #3: the same idempotency key, twice This one happens below the surface, inside the tool gateway. Every tool call gets a run_step_tool_calls row. The table carries three unique constraints (models/runs.py:182): UniqueConstraint("tenant_id", "workspace_id", "run_step_id", ...) UniqueConstraint("tenant_id", "workspace_id", "run_id", "tool_call_id", ...) UniqueConstraint("tenant_id", "workspace_id", "idempotency_key", ...) The third is the interesting one. When the same key arrives again and the row is already terminal: if existing.status in {"succeeded", "failed"}: payload = existing.result_json or {} ... return ToolExecutionClaim( record=existing, run_step=step, replayed=True, cached_response=ToolResponse( result=payload.get("result"), success=existing.status == "succeeded", error=existing.error_message, metadata={..., "idempotent_replay": True}, ), ) Last time's result comes back; nothing leaves the process. The metadata carries idempotent_replay: True so callers can tell this apart from a fresh execution. If the earlier result was large enough to live in object storage, load_cached_response (tool_calls.py:636) fetches the artifact — after checking tenant, workspace, run and step all match, and raising Tool result artifact scope mismatch if any of them does not. The point of this layer: replay #4 is only safe to offer because this one exists. When you re-run, the tool calls whose idempotency keys did not change are not actually executed a second time. 6. A status that admits we don't know This is the design I would point at first if someone asked what is unusual about this ledger. Claiming a tool call takes a lease (60 seconds by default, widened by the gateway to the tool's timeout). An expired lease means the executor may be dead. Retry or not? The code answers by asking whether the request actually left (tool_calls.py:309): lease_expired = ( existing.lease_expires_at is not None and _aware_utc(existing.lease_expires_at) dict[str, Any] | None: if not run.input_summary: return None import json try: parsed = json.loads(run.input_summary) ... It json.loads the summary. So a run whose inputs exceeded 8KB will fail to parse on replay (truncated JSON generally is not valid) and return Replay requires inputs or a parseable run input_summary. Workaround today: pass inputs explicitly instead of letting it read from the ledger — both endpoints accept an override. Intended fix: send large inputs to an artifact and keep a pointer, exactly like tool results in section 2. Same status: issue intended, not yet filed. ③ generate_ulid() does not generate a ULID. The source says so itself (kernel/commons/ids.py:9): def generate_ulid() -> str: """Generate a ULID-like sortable ID. For now, we use UUID4 with prefix. In production, consider using python-ulid or similar library for true ULID generation. """ return f"id_{uuid.uuid4().hex}" UUID4 is random. Not sortable at all — neither the name nor the "sortable" in the docstring holds. Bounded but real impact: everything that needs chronological order has to use created_at rather than the id. The catch-up in section 4 does exactly that. Arguably forced into the correct implementation. ④ Catch-up uses a strict greater-than. Following from ③: created_at comes from Python's datetime.now(UTC), and the filter is RunStep.created_at > last_step_time. Theoretical consequence: if two steps land on an identical timestamp and the client's last received event was one of them, the other is skipped by the strict comparison. I did not reproduce this. datetime.now() resolves to microseconds on modern Linux, and two steps in one run colliding on the same microsecond takes unusual conditions. It is listed as a design fragility, not an observed bug — please don't repeat it as one. The fix is easy once ③ is done: order by (created_at, id). ⑤ Ids in the ledger come in three shapes. Because generate_ulid() already returns an id_-prefixed string, anything that adds its own prefix ends up double-prefixed: Table How it is generated What you see run_step_tool_calls f"rstc_{generate_ulid()}" rstc_id_xxxx tasks f"task_{generate_ulid()}" task_id_xxxx run_cost_entries default_factory=generate_ulid id_xxxx All three work. The third just gives no hint which table the row belongs to. Cosmetic, no functional impact — but you notice it the moment you start reading rows. ⑥ The Run.status docstring lists 6 statuses; there are 11. On the model (models/runs.py:79): """Status: queued, running, paused, succeeded, failed, canceled.""" ExecutionStatus has eleven: those six plus preparing, waiting_input, waiting_approval, retrying, expired. Steps add skipped on top. Impact: anyone writing a client from that comment misses five states. Documentation drift; a one-line fix. ⑦ The soit runs replay line in the console is display copy — that CLI does not exist. From the run detail adapter (web/app/console/adapters/run-detail.ts:227): ledger_code: { command: `soit runs replay ${run.id} --dry-run`, output: `replaying ${detail.steps.length} steps · verdict on record: ${run.status}`, }, It renders as a code sample explaining what that panel shows. But there is no soit CLI in the open-source repo — server/pyproject.toml has no [project.scripts], and server/scripts/ has no matching entry point. The thing that does work is the HTTP endpoint from section 3. There is a replay script in the repo, but it is for the outbox (server/scripts/replay_outbox_event.py, 41 lines — it returns one terminally failed domain event to the pending queue), which is a different thing entirely. I went back and forth on including this. Including it says we haven't kept our own console copy honest. Leaving it out means a reader types the command from a screenshot and gets nothing. Included, in the end — the gap between demo copy and real capability is exactly the kind of thing a reader is entitled to know. Coming clean No fresh live run behind this piece. Every conclusion comes from reading soit/ at commit fb46f20, plus the tests already in the repo. I did not stand up an environment, execute a run and then call the replay endpoint. Every claim carries a file and a line number; go check them. Item ④ in section 12 is an inference, not an observation. I did not construct the colliding-timestamp case. It is listed because a design should not depend on timestamp uniqueness, not because we have seen it break. Replay does not promise identical output. Replay #4 genuinely runs again — models have temperature, tools talk to real systems, external data moves. What is promised is the same inputs, the same governance policy and complete evidence. There is no record-and-stub harness for tools in the repo. This is all the community edition. Every path above is in github.com/soit-ai/soit and readable right now. Disclosure: I maintain SOIT. One-line version "Replayable" here is not an adjective. It is a field the platform computes, that you can query, and that can come back fail — backed by five database tables, four replay paths with very different costs, and one status willing to admit we don't know whether the other side did the thing. Try it, and come argue The repo is github.com/soit-ai/soit. To check the claims above: Start it, send any message, take the run_id. GET /api/v1/runs/{run_id} and look at replay_ready among the thirteen evidence items — if it is fail, missing names what is absent. GET /api/v1/observe/runs/{run_id}/replay and see what the seven keys hold. If any of the seven items in section 12 is wrong, open an issue and say so. I would rather learn where this doesn't line up than be told the design is nice.