Realtime Audio Conversations

PickLayer provides a multi-provider realtime audio conversation API that mirrors OpenAI’s Realtime API event format. The system acts as a thin normalization layer, translating between the canonical format and each provider’s native WebSocket protocol.

Supported Providers

OpenAI: openai/gpt-4o-realtime-preview-* models
Google Gemini: gemini/gemini-2.0-flash-exp and other Live API models
xAI (Grok): xai/grok-* models (experimental)

Connection Protocols

PickLayer supports two connection protocols:

Native WebSocket (OpenAI SDK compatible)


wss://{HOST}/v1/realtime

Connect using standard WebSocket with Authorization: Bearer {token} header. This endpoint is compatible with the OpenAI Realtime SDK.

Socket.IO (enhanced features)


wss://{HOST}/v1/realtime-socketio

Connect using Socket.IO client with token in auth/query params. Supports automatic reconnection and room-based session management.

Query Parameters

Both protocols accept the following query parameters:

Parameter	Description
`token` or `authToken`	Bearer JWT
`org` / `organizationId`	Organization identifier
`app` / `applicationId`	Application identifier
`env`	Environment (default: `prod`)
`model`	Provider-prefixed model (e.g., `openai/gpt-4o-realtime-preview-2024-12-17`)
`sessionId` / `session_id`	For reconnection to existing sessions
`voice`	Voice selection (e.g., `alloy`, `echo`, `nova`)
`modalities`	Comma-separated: `text,audio`
`instructions`	System prompt

Turn Detection (VAD) Parameters

Parameter	Description
`turn_detection_type`	`server_vad` (default) or `semantic_vad`
`vad_threshold`	Activation threshold 0-1 (default: `0.5`)
`silence_duration_ms`	Silence duration to detect speech stop (default: `500`)
`create_response`	Auto-respond when speech stops: `true` (default) or `false`
`interrupt_response`	Allow interrupting model audio: `true` (default) or `false`

Session Lifecycle

Connect → session.created event with session configuration
Exchange audio/text via canonical events
Disconnect triggers billing finalization
Reconnection supported via sessionId within TTL (90s default)

Event Format (OpenAI GA API)

Client → Server Events

Event	Description
`input_audio_buffer.append`	Stream audio: `{ type: "input_audio_buffer.append", audio: "<base64>" }`
`input_audio_buffer.commit`	Commit audio buffer
`input_audio_buffer.clear`	Clear uncommitted audio
`conversation.item.create`	Add text/audio items
`response.create`	Request a response
`session.update`	Update session config

Server → Client Events

Event	Description
`session.created`	Session initialized
`session.updated`	Config changes applied
`conversation.item.created`	New conversation item
`conversation.item.input_audio_transcription.completed`	User speech transcription
`response.created`	Response started
`response.output_audio.delta`	Audio chunk: `{ delta: "<base64>" }`
`response.output_audio.done`	Audio stream complete
`response.output_audio_transcript.delta`	Model speech transcript chunk
`response.output_audio_transcript.done`	Model speech transcript complete
`response.text.delta`	Text response chunk
`response.done`	Response complete with usage
`error`	Error details
`warning`	Non-fatal warnings (e.g., unsupported parameters)

Session Configuration

Important: output_modalities only supports ["text"] OR ["audio"], not both together. For voice conversations, use ["audio"] - transcripts still come via response.output_audio_transcript.delta events.


{
  "type": "session.created",
  "session": {
    "id": "<session-id>",
    "object": "realtime.session",
    "model": "gpt-4o-realtime-preview-2024-12-17",
    "output_modalities": ["audio"],
    "instructions": "...",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "transcription": { "model": "whisper-1" },
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.5,
          "silence_duration_ms": 500,
          "create_response": true,
          "interrupt_response": true
        }
      },
      "output": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "voice": "alloy"
      }
    }
  }
}

Audio Format

Default: PCM16 (16-bit, 24kHz, mono)
Configurable via input_audio_format / output_audio_format

Legacy Events (backwards compatibility)

Legacy Event	Canonical Equivalent
`input_text { text }`	`conversation.item.create`
`input_audio { data\|audio }`	`input_audio_buffer.append` + `commit`

Billing

Reserve on connect (estimated 5 minutes)
Finalize on session expiry with actual usage
Meters: audio seconds in/out, text tokens in/out, session duration
Guard actions: guard.realtime-reserve / guard.realtime-finalize

Example: Native WebSocket (JavaScript)


// Native WebSocket - works with OpenAI SDK
const ws = new WebSocket(
  "wss://api.picklayer.dev/v1/realtime?model=openai/gpt-4o-realtime-preview-2024-12-17&voice=alloy",
  {
    headers: {
      "Authorization": "Bearer <JWT>"
    }
  }
);
 
ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  switch (data.type) {
    case "session.created":
      console.log("Session ready:", data.session.id);
      break;
    case "response.output_audio.delta":
      playAudio(data.delta); // base64 PCM16
      break;
    case "response.done":
      console.log("Usage:", data.response.usage);
      break;
  }
};
 
// Send audio
ws.send(JSON.stringify({
  type: "input_audio_buffer.append",
  audio: audioBase64
}));
ws.send(JSON.stringify({ type: "input_audio_buffer.commit" }));

Example: Socket.IO (JavaScript)


import { io } from "socket.io-client";
 
const socket = io("wss://api.picklayer.dev", {
  path: "/socket.io",
  auth: {
    token: "<JWT>",
    org: "acme",
    app: "voice-assistant",
    model: "openai/gpt-4o-realtime-preview-2024-12-17",
    voice: "alloy",
    modalities: "text,audio"
  }
});
 
// Connect to the realtime namespace
socket.on("connect", () => {
  socket.emit("join", "/v1/realtime-socketio");
});
 
socket.on("session.created", (data) => {
  console.log("Session ready:", data.session.id);
});
 
socket.on("response.output_audio.delta", (data) => {
  playAudio(data.delta);
});

Example: Python SDK


from picklayer import PickLayerClient
from picklayer.realtime import RealtimeSession, RealtimeSessionConfig
 
client = PickLayerClient("org", "app")
client.set_token(access_token)
 
config = RealtimeSessionConfig(
    model="openai/gpt-4o-realtime-preview-2024-12-17",
    voice="alloy",
)
 
# Native WebSocket (OpenAI compatible)
session = RealtimeSession(client, config, protocol="websocket")
 
# Or Socket.IO (enhanced features)
# session = RealtimeSession(client, config, protocol="socketio")
 
@session.on("response.output_audio.delta")
def handle_audio(event):
    play_audio(event.delta)
 
session.connect()
session.send_audio(audio_base64)
session.disconnect()

Live Transcription

PickLayer also supports live streaming speech-to-text over the realtime endpoints. Use transcription models instead of conversation models.

Supported Transcription Models

openai/gpt-4o-mini-transcribe
openai/gpt-4o-transcribe

Transcription Mode Behavior

Session type is transcription (not realtime)
Output is text-only (output_modalities: ["text"])
No audio output config is sent (no voice, no output audio format)
Primary events:
- conversation.item.input_audio_transcription.delta
- conversation.item.input_audio_transcription.completed
Billing uses audio_stt per-second rates (guard.audio-stt-reserve / guard.audio-stt-finalize)

Transcription Query Parameters

Parameter	Description
`language`	ISO-639-1 hint (e.g., `en`, `es`, `ja`)
`noise_reduction`	`near_field` or `far_field`

Example: Native WebSocket (JavaScript)


const ws = new WebSocket(
  "wss://api.picklayer.dev/v1/realtime?model=openai/gpt-4o-mini-transcribe&language=en",
  { headers: { Authorization: "Bearer <JWT>" } }
);
 
ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  if (data.type === "conversation.item.input_audio_transcription.delta") {
    process.stdout.write(data.delta);
  }
  if (data.type === "conversation.item.input_audio_transcription.completed") {
    console.log("\nFinal:", data.transcript);
  }
};
 
ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: audioBase64 }));
ws.send(JSON.stringify({ type: "input_audio_buffer.commit" }));

Example: Socket.IO (JavaScript)


import { io } from "socket.io-client";
 
const socket = io("wss://api.picklayer.dev", {
  path: "/socket.io",
  auth: {
    token: "<JWT>",
    model: "openai/gpt-4o-mini-transcribe",
    modalities: "text",
    language: "en",
  },
});
 
socket.on("event", (event) => {
  if (event.type === "conversation.item.input_audio_transcription.delta") {
    process.stdout.write(event.delta);
  }
  if (event.type === "conversation.item.input_audio_transcription.completed") {
    console.log("\nFinal:", event.transcript);
  }
});
 
socket.emit("event", { type: "input_audio_buffer.append", audio: audioBase64 });
socket.emit("event", { type: "input_audio_buffer.commit" });

Example: Python SDK


from picklayer.realtime import RealtimeSession, RealtimeSessionConfig
 
config = RealtimeSessionConfig.transcription(
    model="openai/gpt-4o-mini-transcribe",
    language="en",
)
session = RealtimeSession(client, config)
 
@session.on("conversation.item.input_audio_transcription.delta")
def on_transcript_delta(event):
    print(event.delta, end="", flush=True)
 
@session.on("conversation.item.input_audio_transcription.completed")
def on_transcript_done(event):
    print("\nFinal:", event.transcript)
 
session.connect()
session.send_audio(audio_base64)
session.disconnect()

Provider Translation

Provider	Behavior
OpenAI	Passthrough (canonical format)
Gemini	Translates to/from Gemini Live API format (`realtimeInput.mediaChunks`, `serverContent.modelTurn`, etc.)
xAI	OpenAI-compatible with voice mapping (alloy→Ara, echo→Rex, etc.)

Error Handling

error events include type, code, message
Unsupported provider features emit warning events with unsupported_options array