Add livekit transport opt-in alongside aiortc#119
Conversation
Side-by-side WebRTC transport support for the inference server's new livekit path. aiortc stays the default and is fully back-compat. - packages/sdk/src/realtime/transports/livekit.ts: new LiveKitConnection. Public surface (connect/send/cleanup/ getPeerConnection/websocketMessagesEmitter/setImageBase64/state) matches WebRTCConnection so WebRTCManager can swap implementations. Control WS is identical (prompt / set_image / session_id / tick acks); the only differences are the media handshake (livekit_join → livekit_room_info, then Room.connect + publishTrack). - packages/sdk/src/realtime/transports/index.ts: shared TransportKind type + re-exports. - packages/sdk/src/realtime/webrtc-manager.ts: gains an optional transport: "aiortc" | "livekit" field in WebRTCConfig. The constructor dispatches to LiveKitConnection when opted in, WebRTCConnection otherwise. All manager state machine logic (reconnect, buffer, emit) is transport-agnostic. - packages/sdk/src/realtime/client.ts: RealTimeClientConnectOptions now accepts `transport`; it's threaded into the manager config. - package.json: adds livekit-client ^2.0.0. Typecheck passes; all 145 existing unit tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| rejectConnect = reject; | ||
| }); | ||
| connectAbort.catch(() => {}); | ||
| this.connectionReject = (error) => rejectConnect(error); |
There was a problem hiding this comment.
connectAbort promise created but never raced
High Severity
The connectAbort promise is created and this.connectionReject is wired to reject it, but unlike WebRTCConnection (which uses Promise.race([..., connectAbort]) for every phase), the LiveKitConnection never races connectAbort against openControlWs, requestRoomInfo, joinRoom, or sendInitialPrompt. If the control WebSocket closes during these operations, the rejection is silently swallowed by connectAbort.catch(() => {}), and the connection flow won't abort until individual timeouts expire (up to 30 seconds).
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ac1e2f0. Configure here.
| // Phase 3 — optional: send initial prompt over control WS. | ||
| if (this.callbacks.initialPrompt) { | ||
| await this.sendInitialPrompt(this.callbacks.initialPrompt); | ||
| } |
There was a problem hiding this comment.
Initial image silently dropped in LiveKit transport
High Severity
The initialImage field is declared in LiveKitCallbacks but never sent during connect(). The aiortc transport sends the initial image via setImageBase64 before the media handshake, but the LiveKit transport only checks this.callbacks.initialPrompt and ignores this.callbacks.initialImage. Users providing an initialImage with the LiveKit transport will have it silently discarded.
Reviewed by Cursor Bugbot for commit ac1e2f0. Configure here.
| type: "prompt", | ||
| prompt: prompt.text, | ||
| enhance: prompt.enhance ?? false, | ||
| } as unknown as OutgoingWebRTCMessage; |
There was a problem hiding this comment.
Wrong field name and default in sendInitialPrompt
Medium Severity
The sendInitialPrompt method sends enhance with a default of false, but the server protocol (defined in types.ts as PromptMessage) expects the field enhance_prompt, and the aiortc transport defaults to true. This means the server won't recognize the enhance flag, and behavior will differ from the aiortc transport even when the same options are passed.
Reviewed by Cursor Bugbot for commit ac1e2f0. Configure here.
index.html now has aiortc | livekit radios that feed
realtime.connect({ transport }), so the dev demo at sdk.decart.local
can flip between the two transports without a code change. Default
stays aiortc so existing sanity tests are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 4 total unresolved issues (including 3 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit edb61bd. Configure here.
| this.websocketMessagesEmitter.emit("generationTick", typed); | ||
| break; | ||
| } | ||
| } |
There was a problem hiding this comment.
Server error messages silently dropped after initial handshake
Medium Severity
handleControlMessage has no case for error type messages in its switch statement. After requestRoomInfo() completes and pendingRoomInfoResolvers is empty, server-sent error messages (e.g. insufficient credits, session rejected) are silently ignored. The aiortc transport's handleSignalingMessage always calls callbacks.onError for error messages, ensuring the SDK user is notified.
Reviewed by Cursor Bugbot for commit edb61bd. Configure here.
Inference server gained an opt-in periodic `{"type": "server_metrics"}`
WS emission (DecartAI/api PR forthcoming) that the webrtc-bench tool
subscribes to for per-session fps / latency / queue-depth numbers.
Surface it through the SDK so consumers can do:
rtClient.on("serverMetrics", (msg) => ...)
Changes:
- types.ts: new ServerMetricsMessage type; added to IncomingWebRTCMessage.
- webrtc-connection.ts (aiortc): parse `type: "server_metrics"` and emit
on the internal websocketMessagesEmitter.
- transports/livekit.ts: same, inside handleControlMessage switch.
- client.ts: add `serverMetrics` to public Events, wire the listener so
the internal emitter fans out to the public RealTimeClient.on surface.
Default off — the server only emits when the client's realtime URL has
`?emit_server_metrics=1`. Normal SDK consumers see nothing unless they
explicitly opt in.
Typecheck passes; 145/145 unit tests still green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Forwards the inference server's E2E pixel-latency handshake (message type "marker_config") to SDK consumers. Symmetric with serverMetrics — opt-in via ?pixel_latency=1 on the realtime WS URL. The webrtc-bench tool uses this to align its PixelMarkerReader's search window with the server's actual stamp dimensions (which can differ from the client stamp dims when the server transcodes). Normal consumers ignore the event. - types.ts: MarkerConfigMessage + add to IncomingWebRTCMessage union. - webrtc-connection.ts + transports/livekit.ts: parse type == "marker_config" and emit on the transport's websocketMessagesEmitter. - client.ts: expose as a public markerConfig event on RealTimeClient, via the same emitOrBuffer path as serverMetrics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
E2E pixel-latency no longer negotiates stamp dimensions between client and server — both sides use a fixed protocol and auto-detect the received scale. The marker_config WS message is gone, so drop the MarkerConfigMessage type and the event plumbing across client.ts, webrtc-connection.ts, transports/livekit.ts, and types.ts. Reverts the prior markerConfig addition on this branch; the webrtc-bench tool in api#1095 handles scale detection inside its PixelMarkerReader. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rate
Two fixes that let non-aiortc transports see the same `stats` event stream
and that keep the reported outbound bitrate sensible under simulcast:
1. Transport-agnostic stats source.
Introduce `StatsProvider`: `{ getStats(): Promise<RTCStatsReport> }`.
`RTCPeerConnection` already satisfies it (aiortc path, back-compat);
LiveKitConnection now supplies an aggregator that walks every local
and remote track in the Room, calls `track.getRTCStatsReport()`, and
merges the per-track reports into one RTCStatsReport-shaped Map.
That's the minimum surface `WebRTCStatsCollector.parse()` needs — it
iterates with `.forEach` and keys off `report.type`.
Before: LiveKitConnection.getPeerConnection() returned null, so the
SDK never started its stats collector for livekit sessions and no
`stats` events fired. Now livekit sessions emit stats on the same
cadence (and with the same payload shape) as aiortc.
Client code (`startStatsCollection` / `handleConnectionStateChange`)
now consults `manager.getStatsProvider()` instead of
`manager.getPeerConnection()`. The identity check (so we don't
restart the collector on every state change) still works because
both the provider and the PC are stable references per connection.
2. Simulcast-safe outbound bitrate.
Simulcast emits one `outbound-rtp` report per spatial layer (3 layers
is typical). The parser used to overwrite `outboundVideo` with
whichever layer `forEach` visited last — each layer has its own
`bytesSent` counter, so across ticks the "last visited" layer would
alternate and `bytesSent - prevBytesSentVideo` went violently
negative. We saw `bitrateOutKbps` down to -6589 in bench results.
Accumulate `bytesSent` + `packetsSent` across every outbound-rtp
video report; compute the bitrate once, after the forEach, against
the summed total. Also clamp the result to `Math.max(0, ...)` since
`bytesSent` can transiently drop when tracks are added/removed
mid-session (new simulcast layer ramping up, publisher swap).
For scalar fields (resolution, fps, qualityLimitationReason), pick
the highest-resolution active layer so reported frame dimensions
match what's on the wire.
Verified against staging: 3-region x 2-transport smoke produces 0
negative `bitrateOutKbps` samples and livekit scenarios now report
bitrate/fps/rtt/jitter/resolution alongside aiortc.
1 participant
Summary
transports/livekit.ts—LiveKitConnectionwith the same public surface asWebRTCConnection(connect/send/cleanup/getPeerConnection/websocketMessagesEmitter/setImageBase64/state)WebRTCManagergains atransport?: "aiortc" | "livekit"option; default is"aiortc"(fully back-compat)RealTimeClientConnectOptionsthreadstransportthrough; control-WS behavior is identical for both transports — only the media handshake differs (livekit_join→livekit_room_info→Room.connect)livekit-client ^2.0.0depPairs with the api branch that lands side-by-side aiortc + livekit on the inference server + bouncer + k8s helm chart; details in the plan file alongside that PR.
Test plan
pnpm typecheckcleanpnpm test— 145/145 existing unit tests passcreateRealTimeClient({ transport: "livekit" })against a locallivekit-serverdeployed via the api repo'sjust initonce the api PR lands🤖 Generated with Claude Code
Note
Medium Risk
Adds an alternate realtime media transport and threads a new
transportoption through connection setup, which can impact connection reliability and state handling. Default remainsaiortc, but the new LiveKit handshake and dependency surface introduces integration risk with server-side support.Overview
Adds an opt-in LiveKit-based realtime transport alongside the existing
aiortcflow, selectable perrealtime.connect()call via a newtransport: "aiortc" | "livekit"option (defaulting toaiortc).Implements
LiveKitConnectionto mirror the existing connection interface while switching the media handshake to alivekit_join→livekit_room_infocontrol-WS exchange followed by joining/publishing to a LiveKit SFU room;WebRTCManagernow instantiates the appropriate transport. The SDK testindex.htmlis updated to allow choosing the transport, andlivekit-clientis added as a dependency.Reviewed by Cursor Bugbot for commit edb61bd. Bugbot is set up for automated code reviews on this repo. Configure here.