Frank Fu's Blog

Recently, I’ve been researching real-time voice conversation implementations and discovered that ElevenLabs Agents Platform provides a very powerful WebSocket API. After some exploration, I completed a real-time voice conversation demo that can run directly in the browser. Today, I’ll share the implementation details and usage experience of this project.

1. Why Choose ElevenLabs?

Before we begin, you might be wondering why I chose ElevenLabs over other solutions. I compared ElevenLabs with OpenAI Realtime API and found that ElevenLabs has unique advantages in voice selection, model flexibility, and other aspects. However, I’ll elaborate on this comparison in detail later in the article.

2. Project Overview

demo link: https://demo.navtalk.ai/11labs/en/index.html

This demo is implemented based on the ElevenLabs Agents Platform WebSocket API and supports:

✅ Complete WebSocket connection management

✅ Real-time voice input and output

✅ Text message support

✅ Rich custom configuration options

✅ Complete message handling mechanism

The entire project can run directly in the browser without a backend server, making it perfect for rapid prototyping and learning.

3. Core Features

3.1 Complete WebSocket Connection

The project implements complete WebSocket connection management, including:

▪️ Automatic signature URL retrieval

▪️ Secure WSS connection establishment

▪️ Comprehensive connection status and error handling

3.2 Real-time Voice Conversation

Voice processing is the core functionality, including:

▪️ Microphone audio capture

▪️ 16kHz PCM audio encoding

▪️ Real-time audio stream transmission

▪️ Agent audio playback

3.3 Complete Message Handling

Supports all message types provided by ElevenLabs:

▪️ `conversation_initiation_metadata` – Session initialization

▪️ `user_transcript` – User speech-to-text

▪️ `agent_response` – Agent text response

▪️ `agent_response_correction` – Agent response correction

▪️ `audio` – Agent audio response

▪️ `interruption` – Interruption detection

▪️ `ping/pong` – Heartbeat detection

▪️ `client_tool_call` – Tool call support

▪️ `contextual_update` – Context update

▪️ `vad_score` – Voice activity detection score

3.4 Text Message Support

In addition to voice input, it also supports sending text messages to the Agent, with a very practical feature: text messages can interrupt the Agent’s ongoing voice response, making conversations more natural.

3.5 Custom Configuration

Provides rich configuration options:

▪️ Custom Agent Prompt

▪️ Custom first message

▪️ Language override

▪️ TTS voice ID override

▪️ Dynamic variable support

▪️ Custom LLM parameters (temperature / max_tokens)

4. Detailed Usage Instructions

4.1 Prepare Configuration

4.1.1 Open File

Simply open the link https://demo.navtalk.ai/11labs/en/index.html in your browser to get started.

4.1.2 Required Configuration Items

API Key (xi-api-key):

▪️ ElevenLabs API Key

▪️ Format: `sk-…` or `xi-api-key`

▪️ How to obtain: Log in to the ElevenLabs Console(https://elevenlabs.io/app/settings/api-keys), create or view API Key

Agent ID:

▪️ ElevenLabs Agent ID

▪️ Format: `agent_…`

▪️ How to obtain: Create or view an Agent on the ElevenLabs Agents page(https://elevenlabs.io/app/agents), then copy the Agent ID

4.1.3 Optional Configuration Items (in interface order)

Custom Prompt:

▪️ Override the Agent’s default prompt

▪️ Leave empty to use the default prompt from Agent configuration

▪️ Can be used to temporarily modify the Agent’s behavior and conversation style

First Message:

▪️ The first sentence the Agent says after connection

▪️ Leave empty to use the default first message from Agent configuration

▪️ Example: “Hello, I’m your AI assistant. How can I help you?”

Language:

▪️ Override the Agent’s default language setting

▪️ Supported language codes: `en` (English), `zh` (Chinese), `es` (Spanish), `fr` (French), `de` (German), `ja` (Japanese), etc.

▪️ Leave empty to use the default language from Agent configuration

TTS Voice:

▪️ Override the Agent’s default voice setting

▪️ Select different voice IDs from the dropdown menu

▪️ Leave empty to use the default voice from Agent configuration

▪️ Note: You need to fill in the API Key first to load the voice list

Dynamic Variables:

▪️ Used to dynamically replace variable placeholders in the Prompt during conversation

▪️ Format: JSON object, for example `{“user_name”: “John”, “greeting”: “Hello”}`

▪️ Use case: When the Agent’s Prompt contains variables (such as `{{user_name}}`, `{{greeting}}`), you can pass actual values through dynamic variables

▪️ Example:

  {

    “user_name”: “John”,

    “company”: “ABC Company”,

    “product”: “Smart Assistant”

  }

▪️ If the Agent’s Prompt contains `Hello, {{user_name}}, welcome to use {{product}}`, the dynamic variables will automatically replace it with `Hello, John, welcome to use Smart Assistant`

▪️ Leave empty to not use dynamic variables

LLM Temperature:

▪️ Controls the randomness and creativity of LLM text generation

▪️ Value range: 0.0 – 2.0

▪️ Lower values produce more deterministic and consistent output (more conservative); higher values produce more random and creative output (more flexible)

▪️ Recommended value: 0.7 – 1.0 (balanced creativity and consistency)

▪️ Leave empty to use the default value from Agent configuration

LLM Max Tokens:

▪️ Limits the maximum number of tokens for a single LLM response

▪️ Value range: Positive integers

▪️ Used to control response length and avoid overly long replies

▪️ Leave empty to use the default value from Agent configuration

4.2 Start Conversation

1. Click the “Connect and Start Conversation” button

2. The browser will request microphone permission, please allow it

3. Recording will start automatically after successful connection

4. Start speaking, and the Agent will respond in real-time

4.3 Function Operations

▪️ Stop Recording: Stop sending audio but keep the connection

▪️ Disconnect: Completely disconnect the WebSocket connection

▪️ Text Message: Enter a message in the text input box and send it

5. API Documentation Reference

The demo implementation is based on ElevenLabs Agents Platform WebSocket API(https://elevenlabs.io/docs/agents-platform/api-reference/agents-platform/websocket)

5.1 WebSocket Endpoint

wss://api.elevenlabs.io/v1/convai/conversation?agent_id={agent_id}

5.2 Complete Call Flow

5.2.1 Connection Establishment Phase

Step 1: Establish WebSocket Connection

Client → Server: Establish WebSocket connection

wss://api.elevenlabs.io/v1/convai/conversation?agent_id={agent_id}

Step 2: Send Initialization Data

▪️ Immediately after successful connection, send `conversation_initiation_client_data` message

▪️ Contains Agent configuration overrides (optional), dynamic variables (optional), custom LLM parameters (optional)

▪️ Wait for server to return `conversation_initiation_metadata` event

Step 3: Receive Session Metadata

▪️ Server returns `conversation_initiation_metadata` event

▪️ Content to handle:

  – Save `conversation_id` (for subsequent session management)

  – Record audio format information (`agent_output_audio_format`, `user_input_audio_format`)

  – Start audio capture (call `getUserMedia` to get microphone permission)

5.2.2 Conversation Phase

Audio Input Flow:

User speaks → Microphone capture → Audio processing (downsample to 16kHz) → Convert to 16-bit PCM → Base64 encode → Send user_audio_chunk

Server Response Flow:

Server receives audio → Speech recognition (ASR) → Send user_transcript → LLM processing → Generate response → Send agent_response → TTS synthesis → Send audio chunks

Key Event Handling Sequence:

1. When user speaks:

   ▪️ Continuously send `user_audio_chunk` (send once every 4096 samples)

   ▪️ Server processes audio stream, may return `vad_score` (voice activity detection score)

2. Server recognizes user speech:

  ▪️ Receive `user_transcript` event

▪️ Can display what the user said in the UI (for debugging)

3. Server generates response:

▪️ Receive `agent_response` event

▪️ Can display the Agent’s text response in the UI

▪️ May receive `agent_response_correction` (if the Agent corrects the response)

4. Server sends audio:

   ▪️ Receive `audio` event (may occur multiple times, streamed)

   ▪️ Processing method:

      – Decode Base64 audio data

      – Add to audio playback queue

      – Play audio chunks in order

5. Interruption handling:

   ▪️ If the user sends a new message while the Agent is speaking, may receive `interruption` event

   ▪️ Need to immediately stop current audio playback and clear the audio queue

5.2.3 Heartbeat Maintenance Phase

Heartbeat Mechanism:

▪️ Server periodically sends `ping` event

▪️ Need to immediately respond with `pong` message, containing the same `event_id`

▪️ Used to keep connection alive and detect connection status

5.2.4 Tool Call Flow (if enabled)

Tool Call Steps:

1. Server sends `client_tool_call` event

2. Processing flow:

   ▪️ Parse tool call information (`tool_name`, `parameters`, `tool_call_id`)

   ▪️ Execute the corresponding tool/function

   ▪️ Send `client_tool_result` to return results

3. Server continues processing, may send new `agent_response` and `audio`

5.2.5 Context Update Flow (if enabled)

Context Update:

▪️ Client can actively send `contextual_update` to update conversation context

▪️ Server may also send `contextual_update` event

▪️ Handle context updates according to business requirements

5.2.6 Text Message Flow

Send Text Message:

▪️ Client sends `user_message` event

▪️ Feature: Can interrupt the Agent’s ongoing audio response (ElevenLabs unique feature)

▪️ Processing method:

  – If the Agent is playing audio, immediately stop playback (receive `interruption` event)

  – Wait for server to process text message and return new response

5.2.7 Connection Close Phase

Normal Close:

▪️ Stop sending audio (call `stopRecording`)

▪️ Close WebSocket connection

▪️ Release audio resources (close AudioContext, stop MediaStream)

Exception Handling:

▪️ Listen to WebSocket `error` and `close` events

▪️ Implement reconnection logic (optional)

▪️ Clean up all resources

5.3 Detailed Event Handling

5.3.1 Events Client Needs to Handle

5.3.2 When Client Sends Messages

6. Audio Format Requirements

ElevenLabs has clear requirements for audio format:

▪️ Sample Rate: 16kHz

▪️ Channels: Mono

▪️ Encoding: 16-bit PCM

▪️ Format: Base64 encoded binary data

7. Technical Implementation

7.1 Audio Processing Flow

1. Capture: Use `getUserMedia` API to get microphone audio stream

2. Process: Use `AudioContext` and `ScriptProcessorNode` to process audio

3. Downsample: If sample rate is not 16kHz, automatically downsample

4. Encode: Convert Float32 audio data to 16-bit PCM

5. Encode: Base64 encode and send via WebSocket

7.2 Audio Playback Flow

1. Receive: Receive Base64 encoded audio from WebSocket

2. Decode: Base64 decode to binary data

3. Play: Try to play as MP3 first, if fails, play as PCM

8. ElevenLabs vs OpenAI Realtime API Detailed Comparison

During development, I also researched OpenAI Realtime API and found that both platforms have their own characteristics. Below is my detailed comparison:

8.1 Quick Comparison Overview

For detailed comparison information, please see the detailed explanations of each feature point below.

8.2 Detailed Comparison of Key Points

8.3.1 Multimodal Support (Camera Recognition)

Explanation: OpenAI Realtime API is based on GPT-4o model, supports multimodal input, and can process image and video content. ElevenLabs currently focuses on voice conversation scenarios and does not support visual input.

Reference Sources:

▪️ ElevenLabs: Official WebSocket API Documentation – Does not mention visual input support

▪️ OpenAI: Realtime API Official Documentation – Supports GPT-4o multimodal capabilities

8.3.2 Voice Selection Comparison

Detailed Comparison:

ElevenLabs: Provides over 100 preset voices, covering multiple languages, ages, genders, and styles. Supports voice cloning, can create custom voices from a small number of samples. Supports emotion and tone control, can adjust voice expression. High voice quality, suitable for professional applications.

OpenAI: TTS API provides 10 preset voices (alloy, echo, fable, onyx, nova, shimmer…), relatively limited selection. Does not support voice cloning, weak voice control capability.

Reference Sources:

▪️ OpenAI: TTS API Documentation – Lists 10 available voices

▪️ ElevenLabs: Official Voice Library – Shows large number of preset voices

▪️ ElevenLabs: Voice Cloning Documentation – Supports custom voice cloning

8.2.3 Supported LLM Models

List of Models Supported by ElevenLabs Agents Platform:

ElevenLabs Proprietary Models:

▪️ GLM-4.5-Air: Suitable for agentic use cases, latency ~631ms, cost ~$0.0600/minute

▪️ Qwen3-30B-A3B: Ultra-low latency, latency ~163ms, cost ~$0.0168/minute

▪️ GPT-OSS-120B: Experimental model (OpenAI open-source model), latency ~314ms, cost ~$0.0126/minute

Other Provider Models (available on ElevenLabs platform):

OpenAI Models:

▪️ GPT-5 series: GPT-5 (latency ~1.14s, cost ~$0.0826/minute), GPT-5.1, GPT-5 Mini (latency ~855ms, cost ~$0.0165/minute), GPT-5 Nano (latency ~788ms, cost ~$0.0033/minute)

▪️ GPT-4.1 series: GPT-4.1 (latency ~803ms, cost ~$0.1298/minute), GPT-4.1 Mini, GPT-4.1 Nano (latency ~478ms, cost ~$0.0065/minute)

▪️ GPT-4o (latency ~771ms, cost ~$0.1623/minute), GPT-4o Mini (latency ~738ms, cost ~$0.0097/minute)

▪️ GPT-4 Turbo (latency ~1.28s, cost ~$0.6461/minute), GPT-3.5 Turbo (latency ~494ms, cost ~$0.0323/minute)

Google Models:

▪️ Gemini 3 Pro Preview (latency ~3.87s, cost ~$0.1310/minute)

▪️ Gemini 2.5 Flash (latency ~752ms, cost ~$0.0097/minute), Gemini 2.5 Flash Lite (latency ~505ms, cost ~$0.0065/minute)

▪️ Gemini 2.0 Flash (latency ~564ms, cost ~$0.0065/minute), Gemini 2.0 Flash Lite (latency ~547ms, cost ~$0.0049/minute)

Anthropic Models:

▪️ Claude Sonnet 4.5 (latency ~1.5s, cost ~$0.1956/minute), Claude Sonnet 4 (latency ~1.31s, cost ~$0.1956/minute)

▪️ Claude Haiku 4.5 (latency ~703ms, cost ~$0.0652/minute)

▪️ Claude 3.7 Sonnet (latency ~1.12s, cost ~$0.1956/minute), Claude 3.5 Sonnet (latency ~1.14s, cost ~$0.1956/minute)

▪️ Claude 3 Haiku (latency ~608ms, cost ~$0.0163/minute)

Custom Models:

▪️ Supports adding custom LLMs

The above image shows the list of selectable LLM models in ElevenLabs Agents Platform, including latency and pricing information

Detailed Explanation:

– ElevenLabs: Provides rich model selection, including proprietary models and models from multiple third-party providers. Users can choose the most suitable model based on latency, cost, and functional requirements. Supports customizing LLM parameters (such as temperature, max_tokens) through `custom_llm_extra_body`.

– OpenAI: Clearly supports GPT-4o (supports multimodal, stronger reasoning capabilities) and GPT-4o-mini (faster, lower cost), users can choose according to needs. Both models support real-time conversation.

Reference Sources:

▪️ ElevenLabs: [Agents Platform Documentation](https://elevenlabs.io/docs/agents-platform) – Model selection interface

▪️ ElevenLabs: [WebSocket API – Custom LLM Parameters](https://elevenlabs.io/docs/agents-platform/api-reference/agents-platform/websocket#custom-llm-extra-body)

▪️ OpenAI: [Realtime API Documentation](https://platform.openai.com/docs/guides/realtime) – Supports GPT-4o and GPT-4o-mini

▪️ OpenAI: [Model Comparison Documentation](https://platform.openai.com/docs/models) – Detailed model information

8.2.4 Knowledge Base Support

Detailed Explanation:

– ElevenLabs: Supports knowledge base functionality in Agent configuration, can upload documents for Agent reference. Knowledge base content will be automatically referenced in conversations.

– OpenAI: Can create assistants with knowledge base through Assistants API (supports file upload and vector storage), or access external data sources and APIs through function calling, achieving more flexible knowledge retrieval.

Reference Sources:

▪️ ElevenLabs: [Agents Platform Documentation](https://elevenlabs.io/docs/agents-platform) – Mentions knowledge base support

▪️ ElevenLabs: [Agent Configuration Documentation](https://elevenlabs.io/docs/agents-platform/agent-configuration) – Knowledge base configuration instructions

▪️ OpenAI: [Assistants API Documentation](https://platform.openai.com/docs/assistants) – Knowledge base and file upload functionality

▪️ OpenAI: [Function Calling Documentation](https://platform.openai.com/docs/guides/function-calling) – External data access

8.2.5 Function Call Support

Detailed Comparison:

– ElevenLabs: Uses `client_tool_call` event to request client to execute tools, returns results through `client_tool_result`. Tools are defined in Agent configuration.

– OpenAI: Uses standard function calling mechanism, triggered through `tool_calls` event, returns results through `tool_results`. Supports dynamically defining tools in sessions.

Reference Sources:

▪️ ElevenLabs: [WebSocket API – client_tool_call](https://elevenlabs.io/docs/agents-platform/api-reference/agents-platform/websocket#client-tool-call) – Tool calling implementation

▪️ ElevenLabs: [Agent Configuration](https://elevenlabs.io/docs/agents-platform/agent-configuration) – Tool definition

▪️ OpenAI: [Realtime API Function Calling](https://platform.openai.com/docs/guides/realtime/function-calling) – Real-time API tool calling

▪️ OpenAI: [Function Calling Guide](https://platform.openai.com/docs/guides/function-calling) – Detailed implementation instructions

8.2.6 Text Interrupt AI Response

Detailed Comparison:

– ElevenLabs: Supports interrupting AI’s ongoing response by sending text messages. When user sends text message while AI is speaking, AI will immediately stop current response and process new text input, making conversations more natural and smooth, similar to interruption behavior in real human conversations.

– OpenAI: Does not support text message interruption feature. If AI is responding, text messages sent by user need to wait for current response to complete before being processed, which may affect conversation fluency and real-time performance.

Use Cases:

– ElevenLabs: Suitable for scenarios requiring fast interaction and interruption, such as real-time customer service, quick Q&A, etc.

– OpenAI: Suitable for scenarios requiring complete responses, but interaction may not be flexible enough

8.2.7 Latency Comparison

Detailed Explanation:

– ElevenLabs: Latency depends on the selected LLM model. If selecting low-latency models (such as Qwen3-30B-A3B ~163ms, GPT-3.5 Turbo ~494ms), latency can be very low, suitable for real-time interaction. If selecting high-performance models (such as GPT-5 ~1.14s, Claude Sonnet ~1.5s), latency will be higher but reasoning capabilities stronger. Supports streaming audio response, reducing first-byte latency.

– OpenAI: Latency is relatively stable, GPT-4o-mini usually responds faster than GPT-4o. Supports streaming response optimization.

Actual latency will be affected by the following factors:

– Network conditions and geographic location

– Model selection (ElevenLabs platform has multiple models to choose from, OpenAI mainly GPT-4o vs GPT-4o-mini)

– Request complexity

– Server load

The above data are typical values, actual performance may vary depending on usage scenarios.

Reference Sources:

▪️ ElevenLabs: [Agents Platform Documentation](https://elevenlabs.io/docs/agents-platform) – Emphasizes low-latency optimization

▪️ OpenAI: [Realtime API Documentation](https://platform.openai.com/docs/guides/realtime) – Real-time performance description

▪️ OpenAI: [Latency Optimization Guide](https://platform.openai.com/docs/guides/realtime/optimizing-latency) – Performance optimization recommendations

8.2.8 Pricing Comparison

Detailed Comparison:

– ElevenLabs: Uses per-conversation minute billing model, price depends on selected LLM model. Usually includes comprehensive fees for voice synthesis, speech recognition, and LLM calls, billing method is simple and clear. For specific model prices, please refer to the “Supported LLM Models” section above.

– OpenAI: Uses per-token billing model, prices vary significantly between different models:

  – GPT-4o-mini: More economical, suitable for high-frequency usage scenarios

  – GPT-4o: Stronger functionality but higher price, suitable for scenarios requiring multimodal or stronger reasoning capabilities

  – Audio processing billed separately per minute

Cost Estimation Examples (for reference only):

– Short conversation scenario (5 minutes, approximately 1000 tokens): OpenAI GPT-4o-mini approximately $0.0015 + $0.075 = $0.0765

– Long conversation scenario (30 minutes, approximately 5000 tokens): OpenAI GPT-4o-mini approximately $0.0075 + $0.45 = $0.4575

Recommendations: Choose the appropriate platform based on actual usage scenarios and budget:

– If mainly using voice conversation with high usage volume, ElevenLabs’ per-minute billing may be simpler, can choose different models according to needs to balance cost and performance

– If need multimodal capabilities or stronger LLM capabilities, OpenAI may be more suitable

– For high-frequency usage, GPT-4o-mini is usually more economical

Reference Sources:

▪️ ElevenLabs: [Official Pricing Page](https://elevenlabs.io/pricing) – Latest pricing information

▪️ ElevenLabs: [Agents Platform Documentation](https://elevenlabs.io/docs/agents-platform) – Billing instructions

▪️ OpenAI: [Official Pricing Page](https://platform.openai.com/pricing) – Latest pricing information (2024-2025)

▪️ OpenAI: [Realtime API Documentation](https://platform.openai.com/docs/guides/realtime) – Billing details

9. Conclusion

ElevenLabs Agents Platform WebSocket API provides powerful support for real-time voice conversations. Through this demo, I implemented complete real-time voice conversation functionality, including audio capture, processing, transmission, and playback.

Compared to OpenAI Realtime API, ElevenLabs has obvious advantages in voice selection, model flexibility, and other aspects, especially suitable for scenarios requiring specific voices or voice cloning. However, if multimodal capabilities are needed, OpenAI may be a better choice.

If you also want to try implementing real-time voice conversations, this demo should provide a good starting point. The project code is open source, and you can use it directly or extend it based on this foundation.