Event-driven AI agents: Build multi-agent workflows that survive production failures

AI agents become fragile when they are connected as long synchronous chains. An event bus lets them work independently, wait for people and tools, recover after restarts, and place policy between a model's recommendation and a real action. Most AI agent demos fit inside a single request: User -> Agent -> Tool -> Agent -> Response The agent makes a plan, calls a tool, gets an answer, and returns a response. It is easy to understand and easy to demo. Then the same agent meets a real workflow. It needs data from several systems. One tool takes five minutes. Another agent has to review the result. A production change needs human approval. While the approval is pending, one of the services restarts. The model is no longer the hardest part. Coordination is. At that point, adding a better prompt will not fix the system. The agents need a way to work independently, survive failures, and resume after the original request has ended. That is an event-driven architecture problem. Synchronous chains work until somebody has to wait Imagine an operations agent investigating a slow checkout service. It needs to collect metrics, inspect dependencies, run a diagnostic job, propose a recovery action, wait for approval, execute the change, and verify recovery. With direct calls, each component knows what comes next. The first agent waits for the second. The second waits for a tool. The request stays open while a person decides whether to approve the change. This works when every step is fast and available. In production, that assumption does not last long. A timeout may leave a tool running after its caller has given up. A retry may execute the same action twice. A restart can erase the current plan. Adding a security review means changing an integration that was already working. Human approval exposes the problem most clearly. A decision may take minutes or hours. Holding an HTTP request open across that wait is a poor way to preserve a workflow. Replace direct calls with facts In an event-driven design, an agent publishes what happened. Other components decide whether that fact matters to them. ServiceLatencyIncreased | v DiagnosisRequested | v DependencyFailureSuspected | v RecoveryProposed | v HumanApprovalRequired | v RecoveryApproved | v RecoveryCompleted The service health agent can publish DiagnosisRequested without knowing which diagnostic agent will handle it. The diagnostic agent can publish DependencyFailureSuspected without calling the remediation agent directly. An audit service, an observability pipeline, and a security agent can all react to the same event. If one consumer is temporarily unavailable, it can process the event after it recovers, subject to the broker's retention settings. This is the useful part of an event bus. Components participate in the workflow without being wired directly to one another. It is the same producer and consumer separation described in AWS Prescriptive Guidance for event-driven AI, but the pattern is not tied to any cloud or broker. An event is not a command Agent systems get dangerous when every message is treated as interchangeable. An event records a fact: { "type": "ServiceLatencyIncreased", "eventId": "evt-204-01", "incidentId": "incident-204", "service": "checkout-api", "p95LatencyMs": 1840 } The exact schema is up to the system. When events cross team or platform boundaries, the vendor-neutral CloudEvents specification provides a common envelope for event metadata. A command requests an action: { "type": "ScaleService", "commandId": "cmd-204-01", "incidentId": "incident-204", "service": "checkout-api", "targetReplicas": 12, "idempotencyKey": "incident-204:scale:12" } An agent decision is a recommendation. It can include evidence and confidence, but it is still a proposal. These distinctions matter. An LLM saying "scale the service" does not mean the service was scaled. It also does not mean the action was authorized. A safer path is: Event -> Agent decision -> Proposed command -> Policy check -> Authorized command -> Execution -> Outcome event The agent interprets the situation and proposes an action. A policy layer checks permissions, limits, and approval requirements. Deterministic code performs the change. The executor then publishes what actually happened. That separation remains useful even when the model changes. The model can become more capable without quietly gaining permission to restart production or issue a refund. Agent memory is not workflow state Agent platforms talk a lot about memory. Conversational memory and execution state solve different problems. Memory may contain a conversation summary, user preferences, or retrieved knowledge. Workflow state answers operational questions: Which steps finished? Which command is waiting for approval? Was this event already processed? What should resume after a restart? A conversation transcript is a fragile place to store those answers. It may be summarized, truncated, or interpreted differently after a model update. If the only record that a refund was issued is a sentence in a context window, the system may eventually issue it again. Store workflow state explicitly. Keep event IDs, command status, approvals, retry counts, and execution results in durable storage. Give the model only the context it needs for its current decision. The broker will not make the system reliable for you Moving work onto an event bus changes the failure modes. It does not remove them. Duplicate events Design consumers so they can see the same message more than once. Track stable event IDs, and require idempotency keys for tools that change state. Restarting a service twice or issuing the same refund twice is not a retry strategy. Events arriving out of order An approval may arrive after a proposal has expired. A recovery result may appear after a newer action has started. Put versions and timestamps on messages, and reject transitions that no longer match the current workflow state. Agent feedback loops Agent A publishes an event that wakes Agent B. Agent B responds with an event that wakes Agent A. Both agents can keep generating messages and spending tokens without completing useful work. Track causation depth. Set limits for time, tokens, and workflow steps. Route repeated failures to human review. Conflicting decisions Two agents may propose opposite actions for the same resource. Do not allow both commands to run. Serialize changes for that resource or use a version check before execution. The consumer itself should be boring. That is a compliment: def handle(event, store, agent, policy, executor): if store.already_processed(event.id): return workflow = store.load_workflow(event.incident_id) if not workflow.accepts(event.type, event.version): store.mark_processed(event.id, outcome="stale") return decision = agent.decide(workflow.context_for(event)) verdict = policy.evaluate(decision) if verdict.requires_human: store.request_approval(workflow, decision) return if not verdict.allowed: store.record_denied(workflow, decision, verdict.reason) return result = executor.run( decision.command, idempotency_key=decision.command.idempotency_key, ) store.append_event(workflow, result.to_event()) store.mark_processed(event.id, outcome="applied") The model performs the ambiguous reasoning. The surrounding code handles state, policy, deduplication, and execution in ways that can be tested. Not every agent needs an event bus A document summarizer does not need a distributed control plane. A synchronous request is usually enough when one caller expects an immediate answer, the task finishes quickly, and a retry cannot cause harm. Events begin to earn their cost when work continues past a request timeout, several agents act independently, a human interrupts the workflow, or an action has operational or financial consequences. There is a price. Teams have to manage message schemas, retention, tracing, failed-message handling, and asynchronous debugging. Use an event-driven design when those costs buy reliability, not because the architecture sounds more sophisticated. Start with the step that blocks You do not need to rebuild a synchronous agent system all at once. Find the step that blocks the longest. It is often a human approval or a slow external tool. Give that step a durable state record and an event that can resume the workflow. Then add an idempotency key to every command that changes something outside the agent system. That small change answers an important question: does separating the workflow from the original request make the system easier to recover and operate? If it does, move the next boundary. Better models will improve agent decisions. They will not recover a lost approval, prevent a duplicate command, or resume a half-finished workflow. Once agents work across services and over time, they inherit distributed-system failures. An event bus does not solve every one of those failures. It gives the system somewhere durable to acknowledge them, contain them, and continue. Take one agent workflow you already run and draw the waits, retries, approvals, and side effects. If those boundaries are hidden inside one synchronous chain, that is where the first event belongs.