Quickstart

For Claude Code or Cursor, install the shared Realtime Avatar plugin. It connects your account's avatars, credits and clips to the agent, with opt-in writes and a shared workflow for building an avatar app. The plugin source and packaging also document an optional, separately installed hosted call preview.

npm install --save-exact realtime-avatar@0.17.0

Start the call on your server. Your key never touches a browser.

import { RealtimeAvatar, isQueued } from "realtime-avatar";

const rta = new RealtimeAvatar({ apiKey: process.env.REALTIME_AVATAR_API_KEY! });

const call = await rta.startCall({
  avatarId: "ava_…",
  instructions: "You are Rin. Short, warm, specific sentences.",
  maxSeconds: 600,
});

if (isQueued(call)) return { queued: true, position: call.position };
return call.raw;            // relay to the browser, byte-for-byte

Join it from your client.

import { AvatarCall, createProxyClient } from "realtime-avatar/react";

// Holds a URL, not a credential. Your route holds the key.
const client = createProxyClient({ proxyUrl: "/api/realtime-avatar" });

<AvatarCall client={client} avatarId="ava_…" onEnded={({ reason }) => router.push("/")} />;

That's the whole integration. Two halves, one payload between them.

Why this one

How it fits together

your client  ──▶  your backend  ──▶  Realtime Avatar
                  (holds the key,      (capacity, listening,
                   decides the call)    thinking, speaking, rendering)
     ◀────────── live audio + video, and they're still listening ──────────

Two rules fall out of that picture, and they drive everything else:

1. The key is server-only. A browser holding it can start unlimited calls on your account. The constructor throws in a browser runtime, so that fails loudly instead of quietly costing you money.
2. The connection payload is opaque. Relay call.raw untouched — the client validates it strictly and will reject a reshaped object.

Built with an AI agent?

Most people integrating this in 2026 aren't typing it by hand. So the reference is written for the reader that shows up: AGENTS.md is the entire surface in one file, plus the twelve rules that are expensive to learn the hard way.

"Add a voice call to my Next.js app using the Realtime Avatar SDK.
 Read AGENTS.md in node_modules/realtime-avatar first."

CLAUDE.md symlinks to it, so Claude Code picks it up with no prompting. Codex, Cursor and Copilot read AGENTS.md directly. There's also an MCP server if you'd rather your agent query your avatars, balance and bill live instead of reading about them.

The SDK exports composable validation schemas, so Zod 4.4.3 or a compatible newer Zod 4 is a required peer. Modern npm/Bun installs peers automatically; applications that already use Zod should declare one compatible version at their root. This is a JavaScript dependency and introduces no native module.

Packages

One. npm install --save-exact realtime-avatar@0.17.0 and you have the server client, the route adapters, the React bindings and the browser helpers. Everything is a subpath, treeshaken per entry, so a server-only app still ships 18.8 KB with no React and no LiveKit in it.

The credential boundary is enforced by the build, not by the package name: importing a key-holding entry into a browser bundle throws at bundle time. realtime-avatar-mcp is a separate name only because it is a CLI you run with npx and never install.

A server-only app pays for nothing it does not import. ., /server, the route adapters, /browser and /tools have zero external imports — npm run externals prints that per entry and fails the build if it stops being true.

One runtime dependency comes along: zod (4.3 MB), which /react and /react-native need for runtime validation. It is deliberately not bundled — inlining it costs every React app +64 KB gzipped in shipped bytes (15 KB → 79 KB gzip, measured), and shipped bytes are what your users feel, while node_modules is what a developer feels once. React, livekit-client, and the @livekit/* packages are optional peers — install the group for your platform. Native compatibility and the exact verified dependency tuple are documented in the package README.

Examples

Every example runs standalone — copy one out of the repo and it still works.

apps/quickstart/ — the smallest correct integration

apps/demo/ — the showcases

See apps/README.md for what a new example has to contain and which tier it belongs in.

API

One class, one types file. The full surface is libs/http-client/src/types.ts — no import chasing.

// calls
rta.startCall({ avatarId, mode?, instructions?, context?, maxSeconds?, transcription?, video?, recording?, connectionHistory?, transcript?, metadata?, llm? })
rta.endCall(sessionId, { reason? })     // free an abandoned call's slot; idempotent, never throws
rta.leaveQueue(queueTicketId, { reason? }) // give up a queued call's place in line; idempotent, never throws

// optional recordings; server only, requires recordings:read
rta.listRecordings({ sessionId, limit?, cursor? })
rta.getRecording(recordingId)          // durable metadata and processing state
rta.getRecordingAccess(recordingId)    // { recordingId, url, expiresAt }; renew when needed
rta.getConnectionHistory(sessionId)    // bounded LiveKit observations; requires usage:read

// avatars
rta.createAvatarFromImage({ displayName, imageUrl, motionPrompt?, voice? })  // the only lane
rta.createAvatarFromVideo({ displayName, videoUrl, voice? })                 // DEPRECATED — closed, 422
rta.listAvatars()
rta.getAvatar(avatarId)
rta.updateAvatar(avatarId, patch)       // re-point displayName / defaultVoiceId
rta.swapSource(avatarId, { sourceAssetId, anchorTimeMs? })  // re-shoot her: new loop, library re-renders
rta.retimeAnchor(avatarId, anchorTimeMs)                    // same loop, different rest frame
rta.deleteAvatar(avatarId)

// clip library
rta.setClipLibrary(avatarId, { expectedRevision, clips, idle?, on?, actions? }) // full declaration; revision required
rta.setLoop(avatarId, { motionPrompt })     // re-direct the RESTING LOOP; clips untouched
rta.waitForLoop(avatarId)               // block until it settles; THROWS if it failed
rta.waitForClips(avatarId)              // block until nothing is still rendering
rta.listClips(avatarId)                 // rows + revision, anchor, eligibility

// assets
rta.createRemoteAsset({ kind, remoteUrl })
rta.uploadAsset(file, { kind })

// billing
rta.creditBalance()                                 // balance + reserved
rta.listSessions({ from, to, endUserId })           // per-session: when, how long, what it cost
rta.iterateSessions({ from, to })                   // the same, paging handled

// webhooks
verifyTranscript(rawBytes, headers, secret)

Recording defaults to off. "participants" keeps one continuous user file and one character file, each with only its own voice and permitted video. Camera toggles do not split the user file. "tracks" stores separate published audio/video tracks and replacement-camera segments; list them by session after capture. Its initial grant omits recording fields for compatibility with older clients. Your server can choose "audio", "video", "audio_video", "participants", or "tracks" after obtaining consent. Keep recordingId and sessionId for your admin views; fetch temporary playback access when needed. File retention (retainedUntil, normally 30 days) is independent of URL expiry (expiresAt, up to one hour). Join transcripts and your script revisions by session ID. See recording details.

Connection history is separately opt-in: set connectionHistory: true in the server policy. When RTA grants it, the browser SDK batches existing LiveKit connection snapshots automatically (no WebRTC statistics polling). Read the bounded result with getConnectionHistory(sessionId) using the usage:read scope.

Errors worth branching on

Only a capacity-queue response returns { queued: true, position, retryAfterMs }. concurrency_limit_reached throws: active, connecting and starting sessions all count toward the workspace limit. End a session or wait for pending starts to clear, then retry. Avoid duplicate parallel starts. The HTTP error retains safe counts at .concurrency and the correlation ID at .requestId; the proxy and browser client preserve the refusal. Showing an error there is the most common bad first impression.

The realtime-avatar/* route adapters relay every platform failure as JSON with its status, code, requestId (also in X-Request-ID, with cache-control: no-store) and the platform's own retryable verdict, never its private diagnostics. A platform 401 or 403 is the route's own key, so it reaches the page as a 500 internal_error with retryable: false, not a sign-in wall, a plan wall or a retry button. These failures are answered, not thrown, so a framework error handler no longer sees them; the route logs each once with console.error (operation, status, code, requestId, no secrets).

The route spends at most timeoutMs (default 50s) on one platform request, retries and backoff included, and answers 504 upstream_timeout when that runs out. That budget sits inside createProxyClient's default 60s wait so the page always hears the route's answer; raise both together, and keep the route's under your host's max function duration.

In the browser, createProxyClient never retries a mint: the server client inside your route is the one retry owner, and a browser retry on top would multiply each click into up to nine mints. A refusal throws a RealtimeAvatarApiError carrying .status, .code, .retryable and .requestId; offer a retry button rather than looping. A deadline that runs out throws the same class with code: "upstream_timeout", status: 504 and response: null. The route may still be minting, so do not re-send it automatically: release the session if an id reached you, otherwise wait for the server-side join timeout, or leave the retry to the user.

A route on another origin. If the page is app.example.com and the route is api.example.com and authorizes on a cookie, pass credentials: "include" and have the route's CORS answer with Access-Control-Allow-Credentials: true and the page's exact origin. createProxyClient applies that mode to every request, the release it sends when the page is hidden included. Unset, it sets no credentials at all, so fetch's default (same-origin) or your own fetch wrapper decides. The page-hide release is a sendBeacon where a beacon's fixed include matches (a same-origin route, or credentials: "include"), and otherwise a keepalive fetch through your fetch. Neither carries per-request requestOptions.headers, and a beacon carries no header at all, so a route that authorizes the release by header sees none. Only Chromium has been measured; Firefox before 133 ignores keepalive, which matters only for a cross-origin route without credentials: "include".

const client = createProxyClient({ proxyUrl: "https://api.example.com/realtime-avatar", credentials: "include" });

Speech recognition hints

Your server can supply the languages and names relevant to a call:

const call = await rta.startCall({
  avatarId,
  transcription: {
    languageCodes: ["en-US", "zh-CN"],
    customVocabulary: ["Mira", "Acme"],
  },
});

Omit languageCodes, or use [], for automatic multilingual detection. Prefer at most 100 relevant vocabulary terms; the limit is 1,000 terms, 160 characters each. Hints bias recognition rather than guaranteeing a spelling or faster response. They do not change character instructions. This configuration is set when the call starts; changing the object does not update an active call.

Input source on room messages

useRealtimeSession on React and React Native observes normal sendTurn calls as text automatically. If your app already recognizes speech, bind its final-text callback once per session:

await session.sendTurn("Hello");
const sendTranscript = session.createTranscriptSender({ inputSource: "client_stt" });
await sendTranscript("Recognized words");
await sendTranscript("Typed correction", { inputSource: "text" });
// A single send can also declare its source:
await session.sendTurn("Recognized words", { inputSource: "client_stt" });

The adapter defaults to client_stt and captures its source at creation. An explicit per-send declaration wins for that send only. Only text and client_stt are accepted as client declarations; the transport observation stays text. The adapter accepts recognized text; it does not run speech recognition. Existing instructions still work. retryTurn() retains the original resolved declaration and scope, even after a timeout, and sends a new turn_id linked to the previous attempt by retry_of_turn_id.

Consumers subscribed to the same room's lk.chat text streams can read these attributes directly. The existing useChat hook exposes them on received messages:

import { useChat } from "realtime-avatar/react";

// Inside a component connected to the receiving room.
const { chatMessages } = useChat();
const inputs = chatMessages.map((message) => ({
  id: message.id,
  text: message.message,
  turnId: message.attributes?.["rta.turn_id"],
  observedSource: message.attributes?.["rta.observed_input_source"] ?? "unknown",
  declaredSource: message.attributes?.["rta.declared_input_source"],
  declarationScope: message.attributes?.["rta.input_source_declaration_scope"],
}));

For an imperative consumer that owns its room's text handler:

room.registerTextStreamHandler("lk.chat", async (reader, participant) => {
  const text = await reader.readAll();
  await consumeInput({
    text,
    senderIdentity: participant.identity,
    attributes: reader.info.attributes ?? {},
  });
});

Choose one handler owner per topic on a room instance; do not register another lk.chat handler alongside useChat. Stream updates share a message ID; update the existing record rather than inserting each partial update as a new input. The receiving consumer must be connected and subscribed when the message arrives. Room messages are not a durable history or an automatic push to an HTTP backend. Source attributes require text-stream reception; the legacy lk-chat-topic compatibility message does not carry these attributes.

This room-message path requires no inference Worker or platform contract changes. Attributes are client-reported metadata, not verified facts about audio recognition. Automatically tagging RTA's own server STT and returning provenance in the final transcript webhook are separate capabilities; this SDK release does not establish them. Existing webhook types and behavior are unchanged.

Let the character see your camera

Authorize camera sharing in your server's call policy:

const call = await rta.startCall({ avatarId, camera: true, maxSeconds: 600 });

This permits sharing; it does not open the camera. Keep camera in your server policy rather than forwarding it from an end-user request. Use the in-room useAvatarCamera hook for camera controls on web and native. With AvatarCall, render your camera control as a child; with a custom view, place it inside the existing RealtimeAvatarLiveKitRoom.

Capture defaults to off and uses a front camera at 640×360, 5 fps. The server grant must allow camera publication. Use the hook's participant/publication with the platform's VideoTrack for preview. Stop sharing when your app goes into the background, including native AppState transitions, and render the hook's error.

The character receives short-lived visual observations while normal conversation continues. Sharing does not enable recording. This requires the matching platform and inference release; an older grant does not authorize capture.

For an in-room control on either web or native, use useAvatarCamera({ allowed: grant.camera === true, active: foreground }). It returns enabled, pending, error, toggle, setEnabled, publication and participant. Bind active to your page/app foreground and live-call state. Capture stays off until an explicit action, stops when inactive or unmounted, and cancelled permission requests cannot publish it. Capture and publication are separate operations; a late captured track is stopped before it reaches the room. The existing room video prop remains a lower-level LiveKit escape hatch. Do not drive it alongside this hook; choose one camera controller.

Contributing

Issues and PRs welcome — see CONTRIBUTING.md.

npm install
npm run check     # typecheck + build + tests, all of it

Found a security issue? Please don't open a public issue — SECURITY.md has the disclosure path.

Support

• 📖 Docs — https://realtimeavatar.ai/docs
• 🐛 Bugs and feature requests — open an issue
• 🔀 Versioning — the API is pinned at /api/v1. Breaking changes get a new version, never a silent edit.