Routing Computer-Use Agent Pauses to Human Sales Reps: A Sub-30-Second WebRTC Handoff Playbook

Browser-driving AI agents now run real workflows, such as filling forms, navigating checkouts and collecting pricing. Sooner or later every one of them hits a login wall, an MFA prompt, a CAPTCHA, or an action that needs a human decision. A static automation stalls at that point. The session times out, and the opportunity sits in an error log. This playbook shows how to build an event-driven handoff pipeline. It detects a stall, alerts a human in Slack with context, and gives them a live, controllable view of the agent’s browser. It also states what the pattern can and cannot do. Several claims often made about this topic, including some in earlier versions of this article, do not hold up. 0. Scope: what this pattern is, and what it isn’t You can only intercept agents you run. OpenAI’s computer-use tooling is a client-driven loop. Your code sends screenshots to the model, receives actions such as click, type and scroll, executes them in a browser, and repeats. There is no hosted service that sends webhooks to a vendor when a buyer’s agent hits your checkout. The events in this article come from your own agent harness, for example a sales-assist or procurement agent you operate on behalf of a user. You can still detect agent-driven traffic on your own site, but that is a different problem (bot detection and agent identification) and it does not give you control of the visitor’s browser. “OpenAI Operator” no longer exists as a product. OpenAI launched Operator on January 23, 2025. On July 17, 2025 it folded Operator into ChatGPT agent and said the standalone site would be sunset, and standalone access ended on August 31, 2025. Consumer packaging has kept changing since, so check OpenAI’s current help pages before naming a product in customer-facing material. What developers can build on today is the computer-use tool in the Responses API, which exposes the same kind of Computer-Using Agent (CUA) loop. The research-preview tool launched in March 2025 as computer-use-preview, and current docs describe a simpler computer tool type. Handoffs need authorization. Passing a stalled session to a human is appropriate when the agent is acting for your user or your own organization, on accounts and flows you’re authorized to use. Using a sales rep to defeat a CAPTCHA, MFA or 3-D Secure challenge on a third-party site is a different matter. Those controls exist to prove that a specific, authorized person is present, and bypassing them can breach terms of service or law. When the challenge belongs to the end user (their MFA code, their card authentication), the right design hands control to that user, not to your rep. With that scoped, the rest of the article covers a real and useful pattern. An agent pauses at a human-needed moment, the right person is alerted fast, and they pick up the live session with context. 1. Architecture overview Decouple the agent’s execution runtime from the human-facing presentation layer. [Agent harness: Responses API computer-use loop] │ (pending_safety_checks / auth wall / CAPTCHA / high-value state) ▼ [Event Gateway (FastAPI / Node.js)] ── verifies signature, dedupes, scores │ ├───────────────────────────────┐ ▼ ▼ [Slack message with Block Kit] [Session broker: mints short-lived takeover token] │ │ └──────────────┬────────────────┘ ▼ [Rep clicks "Take over" → authenticates via SSO] │ ▼ [Takeover web app: WebRTC video in, input events out] │ ▼ [Input relayed to the agent's browser via CDP; agent loop paused] Components Agent harness. Your code runs the model loop and a browser (Playwright or similar, usually in an isolated container). The browser is typically headed inside a virtual display, not truly headless, because the screen has to be captured. Event gateway. Receives pause events from the harness and decides whether to alert a human. Slack layer. Turns events into an actionable message with context and a link. Media path. Streams the browser’s viewport to the rep. WebRTC is the usual choice for low latency. You can run your own SFU (Selective Forwarding Unit) such as mediasoup or Janus, or use a cloud-browser provider’s embeddable live view. Control path. Carries the rep’s mouse and keyboard input back into the browser, and arbitrates between human and agent control. 2. Detecting the moment to hand off You do not need to guess at stalls. The computer-use API gives you a first-class signal. When the model plans an action that looks sensitive, such as a possible prompt injection, a login, a payment or another irreversible step, the returned computer_call carries pending_safety_checks. The action does not proceed until your code echoes those checks back as acknowledged_safety_checks in the next computer_call_output. OpenAI’s design intent is that a human reviews them. That makes it a natural handoff trigger. Do not auto-acknowledge safety checks in production flows. Route them to a person. Useful handoff triggers: Trigger Source pending_safety_checks present Model/API signal Login, SSO or MFA page detected Your DOM/URL heuristics or the model’s own report CAPTCHA frame present DOM heuristics (for example an iframe[src*='recaptcha']) Repeated identical actions or no page change across N steps Your loop’s stall detector Task reaches a commercial decision (pricing tier, contract terms, payment) Your task logic Example event payload { "event_id": "evt_9981", "timestamp": "2026-11-07T07:32:00Z", "session_id": "sess_7749", "trigger_type": "SAFETY_CHECK_PENDING", "intent_score": 92, "account": { "company_name": "Acme Global Enterprise", "estimated_arr_usd": 250000 }, "page": { "current_url": "https://vendor-portal.example.com/auth/mfa-challenge", "blocking_element": "iframe[src*='recaptcha']" }, "screenshot_url": "https://cdn.internal.example.com/snaps/sess_7749.png" } intent_score and estimated_arr_usd come from your own CRM and scoring systems. They are not produced by OpenAI or any agent platform. Sign every event from the harness to the gateway, for example with an HMAC over the body plus a timestamp. Dedupe on event_id so retries don’t page a rep twice. 3. Streaming the browser to the rep Option A: your own pipeline Run Chrome inside a virtual framebuffer and encode the screen to an SFU. This sketch captures the display with GStreamer and sends VP8 RTP to a local endpoint, such as a mediasoup PlainTransport: # Virtual display for the browser container Xvfb :99 -screen 0 1280x800x24 & export DISPLAY=:99 # Chrome with CDP bound to localhost only. # Recent Chrome versions require a non-default --user-data-dir to honor # --remote-debugging-port. google-chrome \ --remote-debugging-port=9222 \ --remote-debugging-address=127.0.0.1 \ --user-data-dir=/tmp/agent-profile & # Capture the display, encode VP8, send RTP to the SFU gst-launch-1.0 ximagesrc display-name=:99 use-damage=false ! videoconvert \ ! video/x-raw,format=I420 \ ! vp8enc deadline=1 cpu-used=8 target-bitrate=2000000 \ ! rtpvp8pay pt=96 \ ! udpsink host=127.0.0.1 port=5004 Two cautions. First, never expose the CDP port beyond the container. Anyone who can reach it has full control of the browser and its cookies. Second, --no-sandbox, which is common in container recipes, weakens Chrome’s isolation. Prefer a properly configured container sandbox, and if you must disable Chrome’s, treat the pod as untrusted and isolate it at the network level. Keeping the stream running from the start of every session means the video is already flowing when a human arrives. That removes a connection-setup step from the critical path. The cost is encoding CPU and bandwidth for every session, including the ones that never stall. Option B: a hosted cloud browser with live view Several cloud-browser providers ship embeddable live views so you don’t build the media path. Browserbase documents a live view that began as a way to watch automation run in an iframe and then gained two-way interaction, with pause, click and type, so a person can handle a CAPTCHA or an SMS-delivered code and then resume. Anchor Browser documents an embeddable live_view_url that you can place in an iframe and make non-interactive with pointer-events: none. Interactive sessions need a generous session timeout. In one open-source project’s fix notes, a session created with the platform default ended after roughly two minutes and killed the live view mid-takeover, so they set 15 minutes for human-facing sessions. Check each provider’s current docs for limits and pricing. 4. Alerting the rep in Slack Slack’s Block Kit supports headers, fields, images and buttons. Three platform constraints matter here: Incoming webhooks are bound to one channel. Slack’s docs say you cannot override the channel, username or icon in the payload. A "channel" key in a webhook payload is ignored. To route by account, territory or severity, create one webhook per destination or use chat.postMessage with a bot token. Always include a top-level text fallback. Without it, notifications and previews can show no body when you send only blocks. Slack fetches image URLs itself. The image_url must be reachable by Slack’s servers, so a screenshot behind your VPN won’t render. Use a short-lived signed URL, and think about what the screenshot exposes: it can contain personal data or secrets typed on the page. A button with a url simply opens that link in the user’s browser. The link itself is not authentication, so the takeover app must authenticate the rep, ideally through your SSO. import os import httpx SLACK_WEBHOOK_URL = os.environ["SLACK_WEBHOOK_URL"] # bound to one channel async def send_takeover_alert(evt: dict, takeover_url: str) -> None: acct = evt["account"] message = { # Fallback for notifications and previews "text": f"AI agent needs a human: {acct['company_name']} ({evt['trigger_type']})", "blocks": [ {"type": "header", "text": {"type": "plain_text", "text": "AI agent needs a human"}}, {"type": "section", "fields": [ {"type": "mrkdwn", "text": f"*Account:*\n{acct['company_name']}"}, {"type": "mrkdwn", "text": f"*Est. ARR:*\n${acct['estimated_arr_usd']:,}"}, {"type": "mrkdwn", "text": f"*Intent score:*\n{evt['intent_score']}/100"}, {"type": "mrkdwn", "text": f"*Trigger:*\n`{evt['trigger_type']}`"}, ]}, {"type": "image", "image_url": evt["screenshot_url"], "alt_text": "Current browser state"}, {"type": "actions", "elements": [ {"type": "button", "text": {"type": "plain_text", "text": "Take over browser"}, "style": "primary", "url": takeover_url, "action_id": "browser_takeover"}, ]}, ], } async with httpx.AsyncClient(timeout=5) as client: resp = await client.post(SLACK_WEBHOOK_URL, json=message) resp.raise_for_status() Incoming webhooks are rate limited to roughly one message per second per channel, which is fine for handoffs but worth remembering if you alert on many sessions at once. 5. The event gateway This FastAPI handler verifies the sender, dedupes events, mints a short-lived signed takeover link, and posts to Slack in the background. It is a starting point, not a drop-in service. In production, replace the in-memory dedupe set with Redis or a database. import hashlib import hmac import os import time from fastapi import BackgroundTasks, FastAPI, Header, HTTPException, Request app = FastAPI(title="Agent handoff gateway") INGEST_SECRET = os.environ["INGEST_SECRET"].encode() TAKEOVER_SECRET = os.environ["TAKEOVER_SECRET"].encode() INTENT_THRESHOLD = int(os.getenv("INTENT_THRESHOLD", "80")) seen_events: set[str] = set() # use Redis with a TTL in production def verify_signature(body: bytes, timestamp: str, signature: str) -> None: if abs(time.time() - int(timestamp)) > 300: raise HTTPException(401, "stale request") expected = hmac.new(INGEST_SECRET, f"{timestamp}.".encode() + body, hashlib.sha256).hexdigest() if not hmac.compare_digest(expected, signature): raise HTTPException(401, "bad signature") def mint_takeover_url(session_id: str, ttl: int = 600) -> str: exp = int(time.time()) + ttl sig = hmac.new(TAKEOVER_SECRET, f"{session_id}.{exp}".encode(), hashlib.sha256).hexdigest() # The takeover app must still require SSO login and check the rep's role. return f"https://app.example.com/takeover/{session_id}?exp={exp}&sig={sig}" @app.post("/api/v1/agent/events") async def handle_event( request: Request, background: BackgroundTasks, x_timestamp: str = Header(...), x_signature: str = Header(...), ): body = await request.body() verify_signature(body, x_timestamp, x_signature) evt = await request.json() if evt["event_id"] in seen_events: return {"status": "duplicate"} seen_events.add(evt["event_id"]) if evt["trigger_type"] != "SAFETY_CHECK_PENDING" and evt["intent_score"] < INTENT_THRESHOLD: return {"status": "below_threshold"} url = mint_takeover_url(evt["session_id"]) background.add_task(send_takeover_alert, evt, url) # defined in section 4 return {"status": "alerted"} 6. The takeover interface When the rep clicks the button, the takeover app should do five things: Authenticate the rep through SSO and confirm they’re allowed to take over this account’s sessions. Validate the signed link and its expiry. Open a signaling channel, typically a WebSocket, and negotiate WebRTC with your SFU through an SDP offer/answer exchange plus ICE candidates. Provide a TURN server, because many corporate networks block direct UDP paths. Show the video with a status bar, and a clear “you have control” indicator. Send input back over a WebRTC data channel or WebSocket to a small relay beside the browser. The relay converts the rep’s pointer and keyboard events into Chrome DevTools Protocol commands, namely Input.dispatchMouseEvent and Input.dispatchKeyEvent. It also needs to map coordinates from the video’s scaled size to the real viewport. Arbitrating control Never let the human and the agent act at once. A safe protocol: Pause. When a rep takes over, stop the agent loop. Do not send further actions, and hold any in-flight computer_call. Control. Relay the rep’s input to the browser. Resume. When the rep clicks “Return to agent”, capture a fresh screenshot and send it as the next computer_call_output. If a safety check was pending, include the checks the human has now reviewed in acknowledged_safety_checks. Tell the model what changed, for example that the user completed authentication, so it doesn’t repeat work. Log every takeover with who, when, which session, and the start and end of control. If the rep types credentials, card numbers or MFA codes, those pass through your infrastructure, so decide ahead of time what you record. Don’t record keystrokes or video of sensitive fields unless you have to, and apply your data-retention and privacy obligations to anything you keep. 7. A latency budget you can test “Sub-30-second” is a target you must measure, not a property WebRTC gives you. Break the path into stages, put a timestamp at each, and track the 95th percentile: Stage Example design target Agent detects the stall and emits the event 1–3 s Gateway validation, scoring, Slack post 1–2 s Slack delivery and notification 1–5 s, outside your control Rep notices and clicks 5–15 s, usually the largest variable SSO check, signaling, WebRTC connection 2–4 s First video frame under 1 s These are planning numbers for your own testing, not published benchmarks. Human attention is the dominant variable, so the biggest improvements come from alert routing, such as on-call rotation, escalation to a second rep after a short timeout, and mobile push, rather than from shaving milliseconds off the media path. Practical optimizations: Keep the media path warm for sessions likely to stall, such as those at checkout or authentication steps, rather than for all sessions, to control encoding cost. Pre-warm browser containers so a new session doesn’t cold-start. Measure how much this saves in your environment. Place the SFU and TURN servers near your reps. Round-trip time between rep and media server drives interaction feel. Measure it rather than assuming a number. Filter alerts. Page a human only when the trigger is a pending safety check or the intent score clears a threshold you’ve tuned on your own data. Alert fatigue will hurt your response time more than any network setting. Add escalation. If nobody claims a takeover within a set window, re-alert a wider group and let the agent either wait or time out gracefully. 8. Security and compliance checklist Treat page content as untrusted. Prompt injection is a main reason the safety checks exist. A human taking over mid-session should see what the agent saw and be warned if the page contained suspicious instructions. Isolate each session in its own container with no access to internal networks, and destroy it afterwards. Keep secrets out of screenshots and Slack. Slack messages and image URLs persist and are visible to channel members. Use short-lived, signed, single-use links with SSO and role checks, as in section 5. Don’t bypass anti-abuse controls you aren’t authorized to bypass. See the scope section above. Respect privacy law when a session includes a customer’s personal data, including consent, retention and access logging. Summary A well-built handoff pipeline has four parts: reliable trigger detection, using the computer-use API’s pending_safety_checks and your own heuristics; a fast and context-rich Slack alert; a live view the rep can control; and a clean pause-and-resume protocol so the agent picks up where the human left off. Treat the 30-second goal as a metric you instrument and improve, and build the pattern for agents you run, with the authorization to use the accounts they act on. Originally published at https://instachime.com/blog/routing-computer-use-agent-pauses-to-human-sales-reps-a-sub-30-second-webrtc-handoff-playbook