--- title: Start a conversational AI agent description: 'Create and start a Conversational AI agent instance. ' sidebar_position: 1 platform: android exported_from: https://docs.agora.io/en/conversational-ai/rest-api/agent/join exported_on: '2026-06-11T15:04:03.270039Z' exported_file: join.md --- > For a complete site index fetch https://docs.agora.io/llms.txt. For all pages in this product fetch https://docs.agora.io/en/conversational-ai/overview/product-overview.md [HTML Version](https://docs.agora.io/en/conversational-ai/rest-api/agent/join) # Start a conversational AI agent **Method:** POST **Endpoint:** `https://api.agora.io/api/conversational-ai-agent/v2/projects/{appid}/join` Use this endpoint to create and start a Conversational AI agent instance. ## Request ### Path parameters - **appid** (string, required): The App ID of the project. ### Request body APPLICATION/JSON **BODY** - **name** (string, required): The unique identifier of the agent. The same identifier cannot be used repeatedly. - **preset** (string, optional): A comma-separated string of one or more presets. Each preset provides a predefined configuration for ASR, LLM, and TTS. You can specify a preset for any or all of ASR, LLM, and TTS. When a preset is specified, you do not need to provide the endpoint URL, API key, or model for the preset providers. Use the `asr`, `llm`, and `tts` fields to configure additional settings. - **pipeline_id** (string, optional): The unique ID of a published agent in [Agent Studio](https://docs-md.agora.io/en/conversational-ai/studio/overview.md). When provided, the saved agent configuration is used as the base configuration. Any fields specified in `properties` override the corresponding agent settings. When you specify a `pipeline_id`, the `asr`, `tts`, and `llm` fields in `properties` are optional. - **properties** (object, required): Configuration details of the agent. - **channel** (string, required): The name of the channel to join. - **token** (string, required): The authentication token used by the agent to join the channel. - **agent_rtc_uid** (string, required): The user ID of the agent in the channel. All UIDs within an RTC channel must be unique. Ensure no other user or service bot is using this UID. A value of `0` means that a unique random UID is generated and assigned. Set the `token` accordingly. - **remote_rtc_uids** (array[string], required): A list of user IDs that the agent subscribes to in the channel. Only subscribed users can interact with the agent.Currently, only one user ID is supported. - **enable_string_uid** (boolean, optional, default: `false`): Whether to enable String uid: - `true`: Both agent and subscriber user IDs use strings. - `false`: Both agent and subscriber user IDs must be integers. - **idle_timeout** (integer, optional, default: `30`, possible values: `[0`, `259200]`): The maximum idle time in seconds after all users specified in `remote_rtc_uids` are detected to have left the channel. When the idle time exceeds this value, the agent automatically stops and exits the channel. The valid range is `0` to `259200` (72 hours). - When set to `0`, the agent does not exit due to channel idle timeout. - Regardless of the `idle_timeout` value, the maximum running time for a single session is 72 hours. The agent automatically exits after 72 hours.For precise and reliable control over the agent's lifecycle, use the [`leave`](https://docs-md.agora.io/en/conversational-ai/rest-api/agent/leave.md) API to terminate the agent as soon as its task is complete. - **geofence** (string, optional): Regional access restriction configuration. Use this to limit which Agora servers the Conversational AI Engine can access based on geographic regions. - **area** (string, required, possible values: `GLOBAL`, `NORTH_AMERICA`, `EUROPE`, `ASIA`, `INDIA`, `JAPAN`): The allowed region for server access. - **exclude_area** (string, optional, possible values: `NORTH_AMERICA`, `EUROPE`, `ASIA`, `INDIA`, `JAPAN`): The excluded region. Only available when `area` is set to `GLOBAL`. - **advanced_features** (object, optional): Advanced features configuration. - **enable_mllm** (boolean, optional, default: `false`): Use [mllm.enable](#properties-mllm-enable) instead. Enable Multimodal Large Language Model for voice-to-voice processing. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly. See `turn_detection.type` for turn detection options available with MLLM. - **enable_rtm** (boolean, optional, default: `false`): Whether to enable the Signaling (RTM) service. When enabled, the agent can combine the capabilities provided by Signaling to implement advanced functions, such as delivering [custom information](https://docs-md.agora.io/en/conversational-ai/develop/custom-information.md).Before enabling the Signaling service, make sure the token includes both RTC and RTM privileges. When an agent joins an RTM channel, it reuses the token specified in the `token` field. For more information, see ["How can I generate a token with both RTC and Signaling privileges?"](https://docs-md.agora.io/en/help/integration-issues/rtc_rtm_token.md). - **enable_sal** (boolean, optional, default: `false`): Enable Selective Attention Locking (SAL). When enabled, configure the `sal` field to set up speaker recognition or locking modes. See the `sal` parameter for configuration details. - **enable_tools** (boolean, optional, default: `false`): Enable tool invocation. When enabled, the agent can invoke tools provided by the MCP server to implement advanced functionality. - **asr** (object, optional): Automatic Speech Recognition (ASR) configuration. - **language** (string, optional, default: `en-US`): The BCP-47 language tag identifying the primary language used for agent interaction. If `params` contains a vendor-specific language code, it takes precedence over this setting. - **vendor** (string, optional, default: `ares`, possible values: `ares`, `microsoft`, `deepgram`, `openai`, `google`, `amazon`, `assemblyai`, `sarvam`, `xai`): ASR provider: - `ares`: [Adaptive Recognition Engine for Speech](https://docs-md.agora.io/en/conversational-ai/models/asr/ares.md) - `microsoft`: [Microsoft Azure](https://docs-md.agora.io/en/conversational-ai/models/asr/microsoft.md) - `deepgram`: [Deepgram](https://docs-md.agora.io/en/conversational-ai/models/asr/deepgram.md) - `openai`: [OpenAI](https://docs-md.agora.io/en/conversational-ai/models/asr/openai.md) - `speechmatics`: [Speechmatics](https://docs-md.agora.io/en/conversational-ai/models/asr/speechmatics.md) - `assemblyai`: [AssemblyAI](https://docs-md.agora.io/en/conversational-ai/models/asr/assembly-ai.md) - `amazon`: [Amazon Transcribe](https://docs-md.agora.io/en/conversational-ai/models/asr/amazon.md) - `google`: [Google](https://docs-md.agora.io/en/conversational-ai/models/asr/google.md) - `sarvam`: [Sarvam](https://docs-md.agora.io/en/conversational-ai/models/asr/sarvam.md) - `xai`: [xAI](https://docs-md.agora.io/en/conversational-ai/models/asr/xai.md) - **params** (object, required): The configuration parameters for the ASR vendor. See [ASR Overview](https://docs-md.agora.io/en/conversational-ai/models/asr/overview.md) for details. - **tts** (object, required): Text-to-speech (TTS) module configuration. - **vendor** (string, required, possible values: `microsoft`, `elevenlabs`, `minimax`, `deepgram`, `cartesia`, `openai`, `humeai`, `rime`, `fishaudio`, `google`, `amazon`, `sarvam`, `xai`): TTS provider. - `microsoft`: [Microsoft Azure](https://docs-md.agora.io/en/conversational-ai/models/tts/microsoft.md) - `elevenlabs`: [ElevenLabs](https://docs-md.agora.io/en/conversational-ai/models/tts/elevenlabs.md) - `minimax`: [MiniMax](https://docs-md.agora.io/en/conversational-ai/models/tts/minimax.md) - `deepgram`: [Deepgram](https://docs-md.agora.io/en/conversational-ai/models/tts/deepgram.md) - `murf`: [Murf](https://docs-md.agora.io/en/conversational-ai/models/tts/murf.md) - `cartesia` : [Cartesia](https://docs-md.agora.io/en/conversational-ai/models/tts/cartesia.md) - `openai`: [OpenAI](https://docs-md.agora.io/en/conversational-ai/models/tts/openai.md) - `humeai`: [Hume AI](https://docs-md.agora.io/en/conversational-ai/models/tts/humeai.md) - `rime`: [Rime](https://docs-md.agora.io/en/conversational-ai/models/tts/rime.md) - `fishaudio`: [Fish Audio](https://docs-md.agora.io/en/conversational-ai/models/tts/fish-audio.md) - `google`: [Google](https://docs-md.agora.io/en/conversational-ai/models/tts/google.md) - `amazon`: [Amazon Polly](https://docs-md.agora.io/en/conversational-ai/models/tts/amazon.md) - `sarvam`: [Sarvam](https://docs-md.agora.io/en/conversational-ai/models/tts/sarvam.md) - `xai`: [xAI](https://docs-md.agora.io/en/conversational-ai/models/tts/xai.md) - **params** (object, required): The configuration parameters for the TTS vendor. See [TTS Overview](https://docs-md.agora.io/en/conversational-ai/models/tts/overview.md) for details. - **skip_patterns** (array[integer], optional): Controls whether the TTS module skips bracketed content when reading LLM response text. This prevents the agent from vocalizing structural prompt information like tone indicators, action descriptions, and system prompts, creating a more natural and immersive listening experience. Enable this feature by specifying one or more values: - `1`: Skip content in Chinese parentheses `（）` - `2`: Skip content in Chinese square brackets `【】` - `3`: Skip content in parentheses `( )` - `4`: Skip content in square brackets `[ ]` - `5`: Skip content in curly braces `{ }`- **Nested brackets**: When input text contains nested brackets and multiple bracket types are configured to be skipped, the system processes only the outermost brackets. The system matches from the beginning of the text and skips the first outermost bracket pair that meets the skip rule, including all nested content. - **Agent memory**: The agent's short-term memory always contains the complete, unfiltered LLM text, regardless of live captioning settings. - **Real-time transcript**: When enabled, transcript excludes filtered content during TTS playback but restores the complete text after each sentence finishes. - **llm** (object, required): Large language model (LLM) configuration. - **url** (string, required): The LLM callback address. - **api_key** (string, optional): The LLM verification API key. The default value is an empty string. Ensure that you enable the API key in a production environment. - **system_messages** (array[object], optional): A set of predefined information used as input to the LLM, including prompt words and examples. - **params** (object, optional): Additional LLM configuration parameters, such as the `model` used, and the maximum token limit. For details about each supported LLM, refer to [Supported LLMs](https://docs-md.agora.io/en/conversational-ai/models/llm/overview.md). - **max_history** (integer, optional, default: `32`, possible values: `[1`, `1024]`): The number of conversation history messages cached in the LLM. History includes user and agent dialog messages, tool call information, and timestamps. Agent and user messages are recorded separately. - **input_modalities** (array[string], optional, default: `["text"]`): LLM input modalities: - `["text"]`: Text only - `["text", "image"]`: Text plus image. Recommended configuration, requires the selected LLM to support visual input - **output_modalities** (array[string], optional, default: `["text"]`): LLM output modalities: - `["text"]`: The output text is converted to speech by the TTS module and then published to the RTC channel. - `["audio"]`: Voice only. Voice is published directly to the RTC channel. - `["text", "audio"]`: Text plus voice. Write your own logic to process the output of LLM as needed. - **greeting_configs** (object, optional): Agent greeting broadcast configuration. - **mode** (string, optional, default: `single_every`, possible values: `single_every`, `single_first`): Determines when the agent sends greeting messages to users joining the channel. - `single_every`: Broadcasts a greeting every time a user joins the channel. - `single_first`: Broadcasts a greeting only once to the first user who joins the channel. - **delay_ms** (integer, optional, possible values: `[0`, `5000]`): The delay in milliseconds before the agent plays the greeting message after a user joins the channel. - **interruptable** (boolean, optional, default: `true`): - `true`: Follows the global `interruption` configuration. - `false`: Uninterruptible. The greeting plays in its entirety. If the user speaks multiple times while the greeting plays, the system merges the speech segments after the greeting ends and sends them to the LLM for a single response. - **greeting_message** (string, optional): Agent greeting. If provided, the first user in the channel is automatically greeted with the message upon joining. - **failure_message** (string, optional): Prompt for agent activation failure. If provided, it is returned through TTS when the custom LLM call fails. - **vendor** (string, optional, possible values: `openai`, `azure`, `xai`, `custom`): LLM provider, supports the following settings: - `openai`: [OpenAI](https://docs-md.agora.io/en/conversational-ai/models/llm/openai.md) - `azure`: [Azure OpenAI](https://docs-md.agora.io/en/conversational-ai/models/llm/azure-openai.md) - `xai`: [xAI Grok](https://docs-md.agora.io/en/conversational-ai/models/llm/xai.md) - `custom`: [Custom LLM](https://docs-md.agora.io/en/conversational-ai/develop/custom-llm.md). When you set this option, the agent includes the following fields, in addition to `role` and `content` when making requests to the custom LLM: - `turn_id`: A unique identifier for each conversation turn. It starts from `0` and increments with each turn. One user-agent interaction corresponds to one `turn_id`. - `timestamp`: The request timestamp, in milliseconds. - **style** (string, optional, default: `openai`, possible values: `openai`, `gemini`, `anthropic`, `dify`): The request style for chat completion: - `openai`: For OpenAI and OpenAI-compatible APIs - `gemini`: For Google Gemini and Google Vertex API format - `anthropic`: For Anthropic Claude API format - `dify`: For Dify API format For details, refer to [Supported LLMs](https://docs-md.agora.io/en/conversational-ai/models/llm/overview.md). - **template_variables** (object, optional): Template parameter configuration used to insert variables into the agent's `system_messages`, `greeting_message`, `failure_message`, and `parameters.silence_config.content` text. Uses key-value pairs, where the key is the variable name and the value is the variable's value. Template variables, combined with prompt customization and SIP outbound calling functionality, enable dynamic content injection, automating processes such as automatic hang-up, voicemail recognition, automatic message leaving, and call transfer. To insert defined variables in the prompt text, use the syntax `{{variable_name}}`. The system automatically replaces each variable with the corresponding value defined in `template_variables`.Variable values cannot reference other variables. For example, if you define `"farewell": "Looking forward to seeing you again, {{name}}"`, the `{{name}}` variable will not be resolved. - **mcp_servers** (array, optional): MCP (Model Context Protocol) server configuration. By configuring MCP servers, agents can call tools provided by external services to implement advanced functionality. - **name** (string, required): A unique identifier for the MCP server. Maximum 48 characters. Accepts only English letters and numbers. - **endpoint** (string, required): The endpoint address of the MCP server. The agent uses this to communicate with the MCP server. - **transport** (string, optional, possible values: `streamable_http`): Transport protocol type. - `streamable_http`: Streaming HTTP protocol - **headers** (object, optional): HTTP header information to include when requesting the MCP server, such as authentication information. - **allowed_tools** (array, optional): A list of tools that the agent is allowed to invoke. The agent can only use tools on this list. **Behavior:** - **Empty or omitted**: All tools are enabled. - **Empty array `[]`**: No tools are enabled. - **`["*"]`**: All tools are enabled. - **Specific tools `["aa", "bb", "cc"]`**: Only `aa`, `bb`, and `cc` are enabled. - **Mix with wildcard `["aa", "bb", "*"]`**: All tools are enabled (wildcard takes precedence). - **timeout_ms** (integer, optional): The MCP server request timeout in milliseconds. After timeout, the agent stops waiting for the MCP server's response and continues executing subsequent logic. - **headers** (object, optional): Custom headers to include in requests to the LLM. Use this field to pass business-specific information such as custom fields or tenant identifiers.- These headers are merged with the headers generated by the Conversational AI Engine. If a key conflict occurs, the engine-generated header takes precedence. For example, authentication-related headers. - Header keys are merged using exact string matching and are case-sensitive. Agora recommends using standard capitalization to avoid ambiguity from duplicate keys with different casing. - **mllm** (object, optional): Multimodal Large Language Model (MLLM) configuration for real-time audio and text processing. `mllm` is an exclusive alternative to the standard `asr` + `llm` + `tts` pipeline." - **enable** (boolean, optional): Enable Multimodal Large Language Model for voice-to-voice processing. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly. Replaces the deprecated `advanced_features.enable_mllm`. - **url** (string, optional): The MLLM WebSocket URL for real-time communication. - **api_key** (string, required): The API key used for MLLM authentication. - **messages** (array[object], optional): Array of conversation items used for short-term memory management. Uses the same structure as `item.content` from theOpenAI Realtime API. - **params** (object, optional): Additional MLLM configuration parameters. - **Modalities override**: The `modalities` setting in params is overridden by `input_modalities` and `output_modalities`. - **Turn detection override**: The `turn_detection` setting in `params` is overridden by `mllm.turn_detection`. See [MLLM Overview](https://docs-md.agora.io/en/conversational-ai/models/mllm/overview.md) for details. - **input_modalities** (array[string], optional, default: `["audio"]`): MLLM input modalities: - `["audio"]`: Audio only - `["audio", "text"]`: Audio plus text - **output_modalities** (array[string], optional, default: `["text", "audio"]`): MLLM output modalities: - `["text", "audio"]`: Text plus audio - **greeting_message** (string, optional): Agent greeting message. If provided, the first user in the channel is automatically greeted with this message upon joining. - **vendor** (string, optional, possible values: `openai`, `gemini`, `vertexai`, `xai`): MLLM provider. Currently supports: - `openai`: [OpenAI Realtime API](https://docs-md.agora.io/en/conversational-ai/models/mllm/openai.md) - `gemini`: [Google Gemini Live](https://docs-md.agora.io/en/conversational-ai/models/mllm/gemini.md) - `vertexai`: [Google Gemini Live (Vertex AI)](https://docs-md.agora.io/en/conversational-ai/models/mllm/google-vertex-ai.md) - `xai`: [xAI Grok](https://docs-md.agora.io/en/conversational-ai/models/mllm/xai.md) - **turn_detection** (object, optional): Turn detection configuration for the MLLM module.When `mllm.turn_detection` is defined, the top-level `turn_detection` object has no effect. - **mode** (string, optional, possible values: `agora_vad`, `server_vad`, `semantic_vad`): - `agora_vad`: Agora VAD-based detection. - `server_vad`: Vendor-side VAD-based detection. Supported by OpenAI Realtime API and Gemini Live. - `semantic_vad`: Semantic-based detection. Supported by OpenAI Realtime API only. - **agora_vad_config** (object, optional): Configuration for Agora VAD-based turn detection. Applicable when `mode` is `agora_vad`. - **interrupt_duration_ms** (integer, optional): Minimum duration of speech in milliseconds required to trigger an interruption. - **prefix_padding_ms** (integer, optional): Duration of audio in milliseconds to include before the detected speech start. - **silence_duration_ms** (integer, optional): Duration of silence in milliseconds required to determine end of speech. - **threshold** (number, optional): VAD sensitivity threshold. A higher value reduces false positives. - **server_vad_config** (object, optional): Configuration for vendor-side VAD-based turn detection. Applicable when `mode` is `server_vad`. Parameters are passed through to the vendor. - **prefix_padding_ms** (integer, optional): Duration of audio in milliseconds to include before the detected speech start. - **silence_duration_ms** (integer, optional): Duration of silence in milliseconds required to determine end of speech. - **threshold** (number, optional): VAD sensitivity threshold. Applicable to OpenAI Realtime API only. - **idle_timeout_ms** (integer, optional): Idle timeout in milliseconds. Applicable to OpenAI Realtime API only. - **start_of_speech_sensitivity** (string, optional, possible values: `START_SENSITIVITY_HIGH`, `START_SENSITIVITY_LOW`): Sensitivity for start of speech detection. Applicable to Gemini Live only. - **end_of_speech_sensitivity** (string, optional, possible values: `END_SENSITIVITY_HIGH`, `END_SENSITIVITY_LOW`): Sensitivity for end of speech detection. Applicable to Gemini Live only. - **semantic_vad_config** (object, optional): Configuration for semantic-based turn detection. Applicable when `mode` is `semantic_vad`. Supported by OpenAI Realtime API only. - **eagerness** (string, optional, possible values: `auto`, `low`, `medium`, `high`): Controls how eagerly the model ends its turn. - **avatar** (object, optional): Avatar configuration. - **enable** (boolean, optional, default: `false`): Whether to enable the avatar function for the agent. To enable, set to `true` and configure the `vendor` and `params` fields. - **vendor** (string, optional, possible values: `akool`, `liveavatar`, `anam`, `generic`): Avatar vendor. Supports the following values: - `akool`: [Akool](https://docs-md.agora.io/en/conversational-ai/models/avatar/akool.md) - `liveavatar`: [LiveAvatar](https://docs-md.agora.io/en/conversational-ai/models/avatar/heygen.md) - `anam`: [Anam](https://docs-md.agora.io/en/conversational-ai/models/avatar/anam.md) - `generic`: [Generic](https://docs-md.agora.io/en/conversational-ai/models/avatar/generic.md) - **params** (object, optional): The configuration parameters for the avatar vendor. See [AI Avatar Overview](https://docs-md.agora.io/en/conversational-ai/models/avatar/overview.md) for details. - **turn_detection** (object, optional): Conversation turn detection settings. Controls the logic for voice activity detection and conversation turn determination. The previous version of `turn_detection` is deprecated. Refer to [Deprecated parameters](#deprecated-parameters) for details. Agora recommends switching to the latest parameters.This object has no effect when `mllm.enable` is true. Use [mllm.turn_detection](#properties-mllm-turn-detection) instead.Starting with v2.6, `turn_detection` only handles Start of Speech (SoS) and End of Speech (EoS) detection. Interruption handling strategies, including keyword-based interruption and disabling interruption, have moved to the top-level [`interruption`](#properties-interruption) field.This configuration supports multiple combinations of detection modes: - **Start of Speech (SoS)**: Supports three modes: VAD, Keyword, and Disable. - **End of Speech (EoS)**: Supports VAD and Semantic modes. - **mode** (string, optional, default: `default`, possible values: `default`): Conversation turn detection mode: - `default`: Uses standard conversation turn detection configuration. - **config** (object, optional): Detailed configuration for conversation turn detection. - **speech_threshold** (number, optional, default: `0.5`, possible values: `(0.0`, `1.0)`): Voice activity detection sensitivity. Determines the sound level in the audio signal that is considered voice activity. Lower values make it easier for the agent to detect speech, and higher values ignore weak sounds. - **start_of_speech** (object, optional): Start of Speech (SoS) detection configuration. Determines when a user begins speaking. - **mode** (string, required, possible values: `vad`, `keywords`, `disabled`): Start of speech detection mode: - `vad`: Based on VAD (Voice Activity Detection). Uses audio signal detection. - `keywords`: Deprecated. Use [`interruption.mode = "keywords"`](#properties-interruption-mode) instead. - `disabled`: Deprecated. Use [`interruption.enable = false`](#properties-interruption-enable) with [`interruption.disabled_config.strategy`](#properties-interruption-disabled-config-strategy) to configure the handling strategy. - **{mode}_config** (object, optional): Start of speech detection configuration parameters. The structure and supported fields vary depending on the detection mode.- The configuration type must match `mode`. For example, when `mode` is `vad`, you must provide `vad_config`. - You cannot provide multiple mode configurations simultaneously.**Configuration examples:** - `vad_config` ```json "vad_config": { "interrupt_duration_ms": 160, "speaking_interrupt_duration_ms": 160, "prefix_padding_ms": 800 } ``` - `keywords_config` ```json "keywords_config": { "interrupt_duration_ms": 160, "prefix_padding_ms": 800, "triggered_keywords": ["Are you there", "hello"] } ``` - `disabled_config` ```json "disabled_config": { "strategy": "append" } ``` - **interrupt_duration_ms** (integer, optional, default: `160`, possible values: `[120`, `1200]`): The amount of time in milliseconds that the user's voice must exceed the VAD threshold before an interruption is triggered. - **speaking_interrupt_duration_ms** (integer, optional, default: `160`, possible values: `[120`, `1200]`): Interruption duration in milliseconds while the agent is speaking. - **prefix_padding_ms** (integer, optional, default: `800`, possible values: `[0`, `5000]`): The extra forward padding time in milliseconds before the processing system starts to process the speech input. This padding helps capture the beginning of speech. - **triggered_keywords** (array[string], optional): Specifies the list of keywords that trigger an interruption. When the agent detects any of these keywords in the user's speech, it immediately stops its current interaction and processes the new input. - **strategy** (string, optional, possible values: `append`, `ignore`): Voice processing strategy when the agent is interacting (speaking or thinking): - `append`: Append mode. Human voice does not interrupt the agent. The agent processes the human voice input after the current interaction ends. - `ignore`: Ignore mode. The agent ignores human voice input. If the agent receives human voice while speaking or thinking, the agent discards the input without storing it in context. - **end_of_speech** (object, optional): End of Speech (EoS) detection configuration. Determines when a user ends their speech. - **mode** (string, optional, possible values: `vad`, `semantic`): End of speech detection mode. Possible values: - `vad`: Based on VAD (Voice Activity Detection). Detects silence duration. - `semantic`: Based on semantic triggering. Uses semantic understanding to determine when conversation ends.When `mode` is `semantic`, EOS detection supports English and Chinese only. For unsupported languages, the engine falls back to VAD. - **{mode}_config** (object, optional): End of speech detection configuration parameters. The structure and supported fields vary depending on the detection mode.- The configuration type must match `mode`. For example, when `mode` is `vad`, you must provide `vad_config`. - You cannot provide multiple mode configurations simultaneously.**Configuration examples:** - `vad_config` ```json "vad_config": { "silence_duration_ms": 640 } ``` - `semantic_config` ```json "semantic_config": { "silence_duration_ms": 320, "max_wait_ms": 3000, "pause_state_enabled": true } ``` - **silence_duration_ms** (integer, optional, possible values: `[120`, `2000]`): **Default**: `640` in `vad_config`, `320` in `semantic_config` Silence duration threshold in milliseconds. The minimum silence duration at the end of a speech segment, to ensure that a brief pause does not prematurely end the speech segment. - **max_wait_ms** (integer, optional, default: `3000`, possible values: `[500`, `10000]`): `-1` means forever. Maximum wait time in milliseconds. The maximum time to wait for semantic determination. After timeout, the conversation end is determined based on the current state. - **pause_state_enabled** (boolean, optional, default: `true`): Whether to detect user intent to pause the conversation: - `true`: The agent uses semantic understanding to determine if the user intends to pause the conversation. For example, when the user's input ends with phrases such as "hold on" or "just a moment", the agent waits for further input rather than treating the utterance as complete and sending it to the LLM. - `false`: The agent does not detect intent to pause the conversation. - **interruption** (object, optional): Interruption control configuration. Provides unified management of the agent's behavior when interrupted by the user. - **enable** (boolean, optional, default: `true`): Whether to enable agent interruption: - `true`: Enable interruption. - `false`: Disable interruption. When disabled, the agent cannot be interrupted mid-response. - **mode** (string, optional, default: `start_of_speech`, possible values: `start_of_speech`, `keywords`): The interruption trigger mode: - `start_of_speech`: Trigger interruption when the user starts speaking. - `keywords`: Trigger interruption when the user speaks a specified keyword. Configure the trigger keywords in `keywords_config`. - **keywords_config** (object, optional): Configuration for keyword-based interruption triggering. Applicable only when `mode` is `keywords`. - **trigger_keywords** (array[string], optional): The list of keywords that trigger an interruption. A maximum of 128 keywords is supported. - **disabled_config** (object, optional): Configuration for agent behavior when interruption is disabled. Applicable only when `interruption.enable` is `false`. - **strategy** (string, optional, possible values: `append`, `ignore`): The processing strategy when interruption is disabled: - `append`: User speech does not interrupt the agent. The agent processes the user's input after the current interaction ends. - `ignore`: The agent ignores user speech. If the agent receives user speech while speaking or thinking, it discards the input without storing it in context. - **sal** (object, optional): Selective Attention Locking (SAL) configuration. **(Beta)** - **sal_mode** (string, optional, default: `locking`, possible values: `locking`, `recognition`): Selective attention lock mode. Supports the following options: - `locking`: Speaker Lock Mode. The agent locks onto the speaker, blocking 95% of ambient human voices and noise. You can enable this mode in two ways: - Seamless mode: When a user speaks loudly and clearly at the beginning of a conversation, the intelligent agent automatically recognizes the user as the speaker. - Personalized mode: When creating an agent, a speaker's voiceprint URL is pre-registered through the `sample_urls` field. The agent then locates the speaker based on the pre-registered voiceprint. - `recognition`: Voiceprint recognition mode. You can pre-register only one voiceprint URL using the `sample_urls` field. The agent identifies different speakers and suppresses other background voices and environmental noise. The target speaker is identified through the `vpids` field in the `metadata` field and sent to the LLM. Set `llm.vendor` to "custom" and refer to [Custom LLM](https://docs-md.agora.io/en/conversational-ai/develop/custom-llm.md) for instructions on how to make the LLM process speaker information. - **sample_urls** (object, optional): The registered voiceprint URL as a key-value pair, where the key is the voiceprint name and the value is the download URL for the speaker's voiceprint. Only one voiceprint URL is supported. Example: ```json { "speaker1": "https://example.com/speaker1.pcm" } ```- Do not set the incoming voiceprint name to "unknown"; this is a reserved keyword used to identify unknown speakers. - For a registered voiceprint, ensure that: - Size: The voiceprint file does not exceed 2 MB. - Duration: Contains 10 to 15 seconds of audio, with at least 8 seconds of effective audio excluding silent segments. - Format: 16kHz sampling rate, 16-bit depth, mono PCM audio file. The file name extension must be ".pcm". - **labels** (object, optional): Custom labels in key-value pair format, where the key is the label name and the value is the label value. Enables agents to carry custom business information. These labels are bound to the agent and returned in the `payload` field of all message notification callbacks from the conversational AI engine. Use them to implement custom business logic, such as tagging activity IDs, customer groups, and business scenarios. - **rtc** (object, optional): RTC media encryption configuration. - **encryption_key** (string, optional): The encryption key for RTC media content. The key has no length limit. Agora recommends using a 32-byte key. If no encryption key is set or if the key is empty, built-in encryption is not used. - **encryption_salt** (string, optional): The salt value used for encryption. This is a Base64-encoded string that is 32 bytes long after decoding. This parameter only takes effect when `encryption_mode` is set to `7` (`AES_128_GCM2`) or `8` (`AES_256_GCM2`). Ensure that the salt parameter is not empty for these encryption modes. - **encryption_mode** (integer, optional, possible values: `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`): The built-in encryption mode. - `1`: `AES_128_XTS` - 128-bit AES encryption, XTS mode. - `2`: `AES_128_ECB` - 128-bit AES encryption, ECB mode. - `3`: `AES_256_XTS` - 256-bit AES encryption, XTS mode. - `4`: `SM4_128_ECB` - 128-bit SM4 encryption, ECB mode. - `5`: `AES_128_GCM` - 128-bit AES encryption, GCM mode. - `6`: `AES_256_GCM` - 256-bit AES encryption, GCM mode. - `7`: `AES_128_GCM2` - 128-bit AES encryption, GCM mode. Requires setting `encryption_salt`. - `8`: `AES_256_GCM2` - 256-bit AES encryption, GCM mode. Requires setting `encryption_salt`. Agora recommends using either `7` (`AES_128_GCM2`) or `8` (`AES_256_GCM2`) mode. Both modes support cryptographic salts to enhance security. - **filler_words** (object, optional): Filler word configuration. Plays filler words while waiting for LLM responses to reduce user anxiety and improve conversation flow. Filler word playback follows these rules: - **Playback order**: When multiple filler words or LLM responses are waiting to be played, they are played in the order they arrive. - **Interruption control**: Inherits the interruption mode setting from the [`interruption`](#properties-interruption) field. - **enable** (boolean, optional, default: `false`): Whether to enable filler words: - `true`: Enable filler words. - `false`: Disable filler words. - **trigger** (object, optional): Filler word trigger configuration. Defines when to trigger filler word playback. - **mode** (string, optional, possible values: `fixed_time`): Filler word trigger mode: - `fixed_time`: Fixed time trigger. Triggers filler word playback when LLM response wait time exceeds the threshold. - **{mode}_config** (object, optional): Filler word trigger configuration parameters. The parameter name and structure vary depending on the trigger mode.- The configuration type must match `mode`. For example, when `mode` is `fixed_time`, you must provide `fixed_time_config`. - You cannot provide multiple mode configurations simultaneously.**Configuration example:** ```json "fixed_time_config": { "response_wait_ms": 1500 } ``` - **response_wait_ms** (integer, optional, default: `1500`, possible values: `[100`, `10000]`): LLM response wait threshold in milliseconds. Triggers filler word playback when the LLM waits this duration without generating a response, such as when waiting for RAG retrieval or tool call results. - **content** (object, optional): Filler word content configuration. Defines the source and selection rules for filler words. - **mode** (string, optional, possible values: `static`): Filler word content mode: - `static`: Static filler words. Uses a predefined list of filler words. - **{mode}_config** (object, optional): Filler word content configuration parameters. The parameter name and structure vary depending on the content mode.- The configuration type must match `mode`. For example, when `mode` is `static`, you must provide `static_config`. - You cannot provide multiple mode configurations simultaneously.**Static filler word configuration example:** ```json "static_config": { "phrases": [ "Please wait.", "Okay.", "Uh-huh." ], "selection_rule": "shuffle" } ``` - **phrases** (array[string], required): List of filler word phrases. **Limits:** - Maximum 100 filler words. - Each filler word must not exceed 50 English words. - **selection_rule** (string, optional, possible values: `shuffle`, `round_robin`): Filler word selection rule: - `shuffle`: Random shuffle. Already-used filler words are not repeated until all filler words have been used once. After all filler words are played, they are reshuffled randomly and a new round begins. - `round_robin`: Round-robin. Selects and plays filler words sequentially from the list. After all filler words are played once, a new cycle begins. - **parameters** (object, optional): Agent configuration parameters. - **silence_config** (object, optional): Settings related to agent silence behavior.`silence_config` does not apply when you integrate a `mllm`. - **timeout_ms** (integer, optional, default: `0`, possible values: `0 to 60000`): Specifies the maximum duration (in milliseconds) that the agent can remain silent. After the agent is successfully created and the user joins the channel, any time during which the agent is not listening, thinking, or speaking is considered silent time. When the silent time reaches the specified value, the agent broadcasts a silent reminder message. This feature is useful for prompting users when they become inactive. - `0`: Disables the silent reminder feature. - `(0, 60000]`: Enables the silent reminder. You must also set `content`; otherwise, the configuration is invalid. - **action** (string, optional, default: `speak`): Specifies how the agent behaves when the silent timeout is reached. Valid values: - `speak`: Uses the TTS module to announce the silent prompt (`content`). - `think`: Appends the silent prompt (`content`) to the context and passes it to the LLM. - **content** (string, optional): Specifies the silent prompt message. The message use depends on the value of `action` parameter. - **farewell_config** (object, optional): Graceful hang-up settings for the agent. - **graceful_enabled** (boolean, optional, default: `false`): Enable graceful leave: * `true`: Enabled. When enabled, calling the POST method to stop the agent ensures that the agent is in an `IDLE` state before leaving the channel. * `false`: Disabled. - **graceful_timeout_seconds** (integer, optional, default: `30`, possible values: `[0`, `120]`): Graceful exit timeout (in seconds). Represents the maximum time to wait for the agent to enter an `IDLE` state before exiting the channel. After this time, the agent will exit the channel immediately, even if it is not in an idle state. This field is only effective when `graceful_enabled` is `true`. - **data_channel** (string, optional, default: `datastream`): Agent data transmission channel: - `rtm`: Use RTM transmission. This configuration takes effect only when `advanced_features.enable_rtm` is `true`. - `datastream`: Use RTC data stream transport. - **enable_metrics** (boolean, optional, default: `false`): Whether to receive agent performance data: - `true`: Receive agent performance data. - `false`: Do not receive agent performance data. This setting only takes effect when `advanced_features.enable_rtm` is `true`. See [Listen to agent events](https://docs-md.agora.io/en/conversational-ai/develop/webhooks.md) to learn how to use client components to receive agent performance data. - **enable_error_message** (boolean, optional, default: `false`): Whether to receive agent error events: - `true`: Receive agent error events. - `false`: Do not receive agent error events. This setting only takes effect when `advanced_features.enable_rtm` is `true`. See [Listen to agent events](https://docs-md.agora.io/en/conversational-ai/develop/webhooks.md) to learn how to use client components to receive agent error events. - **audio_scenario** (string, optional, default: `default`): The audio scenario for the RTC channel. - `default`: Maps to `aiserver`. - `chorus`: Real-time chorus scenario, where users have good network conditions and require ultra-low latency. - `aiserver`: Optimized for interactions between the user and the conversational AI agent in terms of latency and network resilience. - **opt_out** (boolean, optional, default: `false`): Whether to disable data retention for the current session. - `false`: (Default) Data retention is enabled. - `true`: Data retention is disabled. When disabled, session interaction text and audio are not retained, which means troubleshooting, performance review, and agent optimization based on session history will not be available.To support service operation, agent optimization, and troubleshooting, session interaction text and audio are temporarily retained for the minimum necessary period by default. If your business has strict security or compliance requirements, enable this setting to prevent session data from being stored. ## Response - If the returned status code is `200`, the request was successful. The response body contains the result of the request. **OK** - **agent_id** (string, optional): Unique id of the agent instance - **create_ts** (integer, optional): Timestamp of when the agent was created - **status** (string, optional, possible values: `IDLE`, `STARTING`, `RUNNING`, `STOPPING`, `STOPPED`, `FAILED`): Current status. - `IDLE` (0): Agent is idle. - `STARTING` (1): The agent is being started. - `RUNNING` (2): The agent is running. - `STOPPING` (3): The agent is stopping. - `STOPPED` (4): The agent has exited. - `FAILED` (6): The agent failed to execute. - If the returned status code is not `200`, the request failed. The response body includes the `detail` and `reason` for failure. Refer to [status codes](https://docs-md.agora.io/en/conversational-ai/rest-api/reference.md) to understand the possible reasons for failure. ### Reference #### Deprecated parameters The following turn detection configuration is deprecated. To create more natural conversations and reduce unintended interruptions, Agora recommends using the latest version of `turn_detection` above. **Turn detection** - **turn_detection** (object, optional): Conversation turn detection settings. - **type** (string, optional, default: `agora_vad`, possible values: `agora_vad`, `server_vad`, `semantic_vad`): Turn detection mechanism. - `agora_vad`: Agora VAD. Compatible with both cascade (ASR/LLM/TTS) and MLLM modes. - `server_vad`: The model detects the start and end of speech based on audio volume and responds at the end of user speech. Only available when `mllm` is enabled and OpenAI Realtime or Gemini Live is selected. The detection behavior is controlled by the LLM provider. - `semantic_vad`: Uses a turn detection model in conjunction with VAD to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability for more natural conversations. Only available when `mllm` is enabled and OpenAI is selected. - **interrupt_mode** (string, optional, default: `interrupt`): Sets the agent's behavior when human voice interrupts the agent while it is interacting (speaking or thinking). Choose from the following values: - `interrupt`: The agent immediately stops the current interaction and processes the human voice input. - `append`: The agent completes the current interaction, then processes the human voice input. - `ignore`: The agent discards the human voice input without processing or storing it in the context. - `keywords`: The agent stops its current interaction after detecting any of the keywords specified in `turn_detection.interrupt_keywords`. - `adaptive`: The agent dynamically increases the voice continuity threshold while speaking to reduce accidental interruptions.Only the `interrupt` mode is supported when you integrate an `mllm`. - **interrupt_duration_ms** (number, optional, default: `160`): The amount of time in milliseconds that the user's voice must exceed the VAD threshold before an interruption is triggered. - **interrupt_keywords** (array[string], optional): Specifies the list of keywords that trigger an interruption when the `turn_detection.interrupt_mode` is set to `"keyword"`. When the agent detects any of these keywords in the user's speech, it immediately stops its current interaction and processes the new input.- Keyword recognition capabilities, such as support for multiple languages or dialects, depend on the ASR provider you choose. - You can configure up to 128 keywords. - **prefix_padding_ms** (integer, optional, default: `800`): The extra forward padding time in milliseconds before the processing system starts to process the speech input. This padding helps capture the beginning of the speech. - **silence_duration_ms** (integer, optional, default: `640`): The duration of audio silence in milliseconds. If no voice activity is detected during this period, the agent assumes that the user has stopped speaking. - **threshold** (number, optional, default: `0.5`, possible values: `(0.0`, `1.0)`): Identification sensitivity determines the level of sound in the audio signal that is considered voice activity. Lower values make it easier for the agent to detect speech, and higher values ignore weak sounds. - **eagerness** (string, optional, default: `auto`, possible values: `auto`, `low`, `high`): The eagerness of the model to respond: - `auto`: Equivalent to medium - `low`: Wait longer for the user to continue speaking - `high`: Respond more quickly Only available in `semantic_vad` mode when using OpenAI Realtime API. ## Authorization This endpoint requires [authentication](https://docs-md.agora.io/en/conversational-ai/rest-api/restful-authentication.md). ## Request examples **curl** ```bash curl --request post \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \ --header 'Authorization: Basic' \ --data ' { "name": "unique_name", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "1001", "remote_rtc_uids": [ "1002" ], "idle_timeout": 120, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "max_history": 32, "greeting_message": "Hello, how can I assist you today?", "failure_message": "Please hold on a second.", "params": { "model": "gpt-4o-mini" } }, "tts": { "vendor": "microsoft", "params": { "key": "", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural" } }, "asr": { "language": "en-US" } } }' ``` **Python** ```python import requests import json url = "https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join" headers = {"Authorization": "Basic"} data = { "name": "unique_name", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "1001", "remote_rtc_uids": ["1002"], "idle_timeout": 120, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "max_history": 32, "greeting_message": "Hello, how can I assist you today?", "failure_message": "Please hold on a second.", "params": { "model": "gpt-4o-mini" } }, "tts": { "vendor": "microsoft", "params": { "key": "", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural" } }, "asr": { "language": "en-US" } } } response = requests.post(url, headers=headers, data=json.dumps(data)) print(response.status_code) print(response.json()) ``` **Node.js** ```js const axios = require("axios"); const url = "https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join"; const headers = { "Authorization": "Basic" }; const data = { name: "unique_name", properties: { channel: "channel_name", token: "token", agent_rtc_uid: "1001", remote_rtc_uids: ["1002"], idle_timeout: 120, llm: { url: "https://api.openai.com/v1/chat/completions", api_key: "", system_messages: [ { role: "system", content: "You are a helpful chatbot." } ], max_history: 32, greeting_message: "Hello, how can I assist you today?", failure_message: "Please hold on a second.", params: { model: "gpt-4o-mini" } }, tts: { vendor: "microsoft", params: { key: "", region: "eastus", voice_name: "en-US-AndrewMultilingualNeural" } }, asr: { language: "en-US" } } }; axios .post(url, data, { headers }) .then(response => { console.log("Status:", response.status); console.log("Response:", response.data); }) .catch(error => { console.error("Error:", error.response ? error.response.data : error.message); }); ``` **curl** ```bash curl -X POST 'https://api.agora.io/api/conversational-ai-agent/v2/projects//join' \ -H 'Content-Type: application/json' \ -H 'Authorization: agora token=' \ -d '{ "name": "", "pipeline_id": "", "properties": { "channel": "", "token": "", "agent_rtc_uid": "1001", "agent_rtm_uid": "", "remote_rtc_uids": ["1002"], "enable_string_uid": false } }' ``` **Python** ```python import requests import json url = "https://api.agora.io/api/conversational-ai-agent/v2/projects//join" headers = { "Content-Type": "application/json", "Authorization": "agora token=" } data = { "name": "", "pipeline_id": "", "properties": { "channel": "", "token": "", "agent_rtc_uid": "1001", "agent_rtm_uid": "", "remote_rtc_uids": ["1002"], "enable_string_uid": False } } response = requests.post(url, headers=headers, data=json.dumps(data)) print(response.json()) ``` **Node.js** ```js const url = "https://api.agora.io/api/conversational-ai-agent/v2/projects//join"; const headers = { "Content-Type": "application/json", "Authorization": "agora token=" }; const data = { name: "", pipeline_id: "", properties: { channel: "", token: "", agent_rtc_uid: "1001", agent_rtm_uid: "", remote_rtc_uids: ["1002"], enable_string_uid: false } }; fetch(url, { method: "POST", headers: headers, body: JSON.stringify(data) }) .then(response => response.json()) .then(json => console.log(json)) .catch(error => console.error("Error:", error)); ``` **curl** ```bash curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \ --header 'Authorization: Basic' \ --data ' { "name": "unique_name", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "1001", "remote_rtc_uids": [ "1002" ], "idle_timeout": 120, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful assistant. User name is {{user_name}}, and their account ID is {{account_id}}." } ], "greeting_message": "Hello {{user_name}}, how can I help you?", "failure_message": "Sorry, {{user_name}}, I cannot answer this question. Please try again later.", "max_history": 32, "params": { "model": "gpt-4o-mini" }, "template_variables": { "user_name": "John", "account_id": "ACC123456", "silent_prompt": "If the user does not respond for an extended period, please inquire whether you require continued service." }, "mcp_servers": [ { "name": "mcpserver", "transport": "streamable_http", "endpoint": "https://registry.run.mcp.com.ai/mcp", "allowed_tools": [ "getV01Servers" ], } ] }, "asr": { "language": "en-US" }, "tts": { "vendor": "minimax", "skip_patterns": [ 1 ], "params": { "group_id": "xxxx", "key": "xxxx", "model": "speech-01-turbo", "voice_setting": { "voice_id": "English_captivating_female1", "speed": 1, "vol": 1, "pitch": 0, "emotion": "happy" }, "audio_setting": { "sample_rate": 16000 } } }, "parameters": { "silence_config": { "timeout_ms": 10000, "action": "think", "content": "{{silent_prompt}}" }, "opt_out": false } } }' ``` **Python** ```python import requests url = "https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join" payload = { "name": "unique_name", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "1001", "remote_rtc_uids": ["1002"], "idle_timeout": 120, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful assistant. User name is {{user_name}}, and their account ID is {{account_id}}." } ], "greeting_message": "Hello {{user_name}}, how can I help you?", "failure_message": "Sorry, {{user_name}}, I cannot answer this question. Please try again later.", "max_history": 32, "params": { "model": "gpt-4o-mini" }, "template_variables": { "user_name": "John", "account_id": "ACC123456", "silent_prompt": "If the user does not respond for an extended period, please inquire whether you require continued service." }, "mcp_servers": [ { "name": "mcpserver", "transport": "streamable_http", "endpoint": "https://registry.run.mcp.com.ai/mcp", "allowed_tools": ["getV01Servers"] } ] }, "asr": { "language": "en-US" }, "tts": { "vendor": "minimax", "skip_patterns": [1], "params": { "group_id": "xxxx", "key": "xxxx", "model": "speech-01-turbo", "voice_setting": { "voice_id": "English_captivating_female1", "speed": 1, "vol": 1, "pitch": 0, "emotion": "happy" }, "audio_setting": { "sample_rate": 16000 } } }, "parameters": { "silence_config": { "timeout_ms": 10000, "action": "think", "content": "{{silent_prompt}}" } } } } headers = {"Authorization": "Basic"} response = requests.post(url, json=payload, headers=headers) print(response.text) ``` **Node.js** ```js const url = 'https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join'; const options = { method: 'POST', headers: {Authorization: 'Basic'}, body: JSON.stringify({ name: 'unique_name', properties: { channel: 'channel_name', token: 'token', agent_rtc_uid: '1001', remote_rtc_uids: ['1002'], idle_timeout: 120, llm: { url: 'https://api.openai.com/v1/chat/completions', api_key: '', system_messages: [ { role: 'system', content: 'You are a helpful assistant. User name is {{user_name}}, and their account ID is {{account_id}}.' } ], greeting_message: 'Hello {{user_name}}, how can I help you?', failure_message: 'Sorry, {{user_name}}, I cannot answer this question. Please try again later.', max_history: 32, params: {model: 'gpt-4o-mini'}, template_variables: { user_name: 'John', account_id: 'ACC123456', silent_prompt: 'If the user does not respond for an extended period, please inquire whether you require continued service.' }, mcp_servers: [ { name: 'mcpserver', transport: 'streamable_http', endpoint: 'https://registry.run.mcp.com.ai/mcp', allowed_tools: ['getV01Servers'] } ] }, asr: {language: 'en-US'}, tts: { vendor: 'minimax', skip_patterns: [1], params: { group_id: 'xxxx', key: 'xxxx', model: 'speech-01-turbo', voice_setting: {voice_id: 'English_captivating_female1', speed: 1, vol: 1, pitch: 0, emotion: 'happy'}, audio_setting: {sample_rate: 16000} } }, parameters: { silence_config: {timeout_ms: 10000, action: 'think', content: '{{silent_prompt}}'} } } }) }; fetch(url, options) .then(res => res.json()) .then(json => console.log(json)) .catch(err => console.error(err)); ``` **curl** ```bash curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \ --header 'Authorization: Basic' \ --data ' { "name": "full_config_agent", "properties": { "channel": "full_channel_name", "token": "full_token_value", "agent_rtc_uid": "12345", "remote_rtc_uids": [ "67890" ], "enable_string_uid": false, "idle_timeout": 180, "advanced_features": { "enable_rtm": true, "enable_sal": true }, "asr": { "language": "en-US", "vendor": "microsoft", "params": { "key": "your_microsoft_key", "region": "eastus", "language": "en-US", "phrase_list": [ "agora", "fengming" ] } }, "tts": { "vendor": "microsoft", "skip_patterns": [ 1, 2 ], "params": { "key": "your_microsoft_key", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural", "speed": 1, "volume": 70, "sample_rate": 24000 } }, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "your_openai_api_key", "system_messages": [ { "role": "system", "content": "You are a professional AI assistant. User name is {{user_name}}." } ], "greeting_message": "Hello {{user_name}}, I am your smart assistant. How can I help you?", "greeting_configs": { "mode": "single_every" }, "failure_message": "Sorry, there was a system problem. Please try again later.", "max_history": 64, "input_modalities": [ "text", "image" ], "output_modalities": [ "text" ], "params": { "model": "gpt-4o", "temperature": 0.8, "max_tokens": 2048, "stream": true }, "template_variables": { "user_name": "John", "company_name": "Agora" } }, "avatar": { "vendor": "heygen", "enable": true, "params": { "api_key": "", "quality": "medium", "agora_uid": "", "agora_token": "", "avatar_id": "", "disable_idle_timeout": false, "activity_idle_timeout": 60 } }, "turn_detection": { "mode": "default", "config": { "speech_threshold": 0.5, "start_of_speech": { "mode": "vad", "vad_config": { "interrupt_duration_ms": 160, "speaking_interrupt_duration_ms": 320, "prefix_padding_ms": 800 } }, "end_of_speech": { "mode": "semantic", "semantic_config": { "silence_duration_ms": 320, "max_wait_ms": 3000 } } } }, "sal": { "sal_mode": "locking", "sample_urls": { "speaker1": "https://example.com/voiceprint1.pcm" } }, "labels": { "campaign_id": "spring_2025", "customer_group": "vip", "region": "eastus" }, "rtc": { "encryption_key": "your_32_byte_encryption_key_here", "encryption_salt": "TsP4fLLyxxxxxxxxxroLA9oE=", "encryption_mode": 8 }, "filler_words": { "enable": true, "trigger": { "mode": "fixed_time", "config": { "response_wait_ms": 1500 } }, "content": { "mode": "static", "config": { "phrases": [ "Please wait a moment.", "OK", "Uh-huh." ], "selection_rule": "shuffle" } } }, "parameters": { "silence_config": { "timeout_ms": 15000, "action": "think", "content": "User has not responded for an extended period. Please inquire whether continued service is needed." }, "farewell_config": { "graceful_enabled": true, "graceful_timeout_seconds": 60 }, "data_channel": "rtm", "enable_metrics": true, "enable_error_message": true, "opt_out": false } } }' ``` **Python** ```python import requests url = "https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join" payload = { "name": "full_config_agent", "properties": { "channel": "full_channel_name", "token": "full_token_value", "agent_rtc_uid": "12345", "remote_rtc_uids": ["67890"], "enable_string_uid": False, "idle_timeout": 180, "advanced_features": { "enable_rtm": True, "enable_sal": True }, "asr": { "language": "en-US", "vendor": "microsoft", "params": { "key": "your_microsoft_key", "region": "eastus", "language": "en-US", "phrase_list": ["agora", "fengming"] } }, "tts": { "vendor": "microsoft", "skip_patterns": [1, 2], "params": { "key": "your_microsoft_key", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural", "speed": 1, "volume": 70, "sample_rate": 24000 } }, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "your_openai_api_key", "system_messages": [ { "role": "system", "content": "You are a professional AI assistant. User name is {{user_name}}." } ], "greeting_message": "Hello {{user_name}}, I am your smart assistant. How can I help you?", "greeting_configs": { "mode": "single_every" }, "failure_message": "Sorry, there was a system problem. Please try again later.", "max_history": 64, "input_modalities": ["text", "image"], "output_modalities": ["text"], "params": { "model": "gpt-4o", "temperature": 0.8, "max_tokens": 2048, "stream": True }, "template_variables": { "user_name": "John", "company_name": "Agora" } }, "avatar": { "vendor": "heygen", "enable": True, "params": { "api_key": "", "quality": "medium", "agora_uid": "", "agora_token": "", "avatar_id": "", "disable_idle_timeout": False, "activity_idle_timeout": 60 } }, "turn_detection": { "mode": "default", "config": { "speech_threshold": 0.5, "start_of_speech": { "mode": "vad", "vad_config": { "interrupt_duration_ms": 160, "speaking_interrupt_duration_ms": 320, "prefix_padding_ms": 800 } }, "end_of_speech": { "mode": "semantic", "semantic_config": { "silence_duration_ms": 320, "max_wait_ms": 3000 } } } }, "sal": { "sal_mode": "locking", "sample_urls": { "speaker1": "https://example.com/voiceprint1.pcm" } }, "labels": { "campaign_id": "spring_2025", "customer_group": "vip", "region": "eastus" }, "rtc": { "encryption_key": "your_32_byte_encryption_key_here", "encryption_salt": "TsP4fLLyxxxxxxxxxroLA9oE=", "encryption_mode": 8 }, "filler_words": { "enable": True, "trigger": { "mode": "fixed_time", "config": { "response_wait_ms": 1500 } }, "content": { "mode": "static", "config": { "phrases": ["Please wait a moment.", "OK", "Uh-huh."], "selection_rule": "shuffle" } } }, "parameters": { "silence_config": { "timeout_ms": 15000, "action": "think", "content": "User has not responded for an extended period. Please inquire whether continued service is needed." }, "farewell_config": { "graceful_enabled": True, "graceful_timeout_seconds": 60 }, "data_channel": "rtm", "enable_metrics": True, "enable_error_message": True } } } headers = {"Authorization": "Basic"} response = requests.post(url, json=payload, headers=headers) print(response.text) ``` **Node.js** ```js const url = 'https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join'; const options = { method: 'POST', headers: {Authorization: 'Basic'}, body: JSON.stringify({ name: 'full_config_agent', properties: { channel: 'full_channel_name', token: 'full_token_value', agent_rtc_uid: '12345', remote_rtc_uids: ['67890'], enable_string_uid: false, idle_timeout: 180, advanced_features: {enable_rtm: true, enable_sal: true}, asr: { language: 'en-US', vendor: 'microsoft', params: { key: 'your_microsoft_key', region: 'eastus', language: 'en-US', phrase_list: ['agora', 'fengming'] } }, tts: { vendor: 'microsoft', skip_patterns: [1, 2], params: { key: 'your_microsoft_key', region: 'eastus', voice_name: 'en-US-AndrewMultilingualNeural', speed: 1, volume: 70, sample_rate: 24000 } }, llm: { url: 'https://api.openai.com/v1/chat/completions', api_key: 'your_openai_api_key', system_messages: [ { role: 'system', content: 'You are a professional AI assistant. User name is {{user_name}}.' } ], greeting_message: 'Hello {{user_name}}, I am your smart assistant. How can I help you?', greeting_configs: {mode: 'single_every'}, failure_message: 'Sorry, there was a system problem. Please try again later.', max_history: 64, input_modalities: ['text', 'image'], output_modalities: ['text'], vendor: 'custom', params: {model: 'gpt-4o', temperature: 0.8, max_tokens: 2048, stream: true}, template_variables: {user_name: 'John', company_name: 'Agora'} }, avatar: { vendor: 'heygen', enable: true, params: { api_key: '', quality: 'medium', agora_uid: '', agora_token: '', avatar_id: '', disable_idle_timeout: false, activity_idle_timeout: 60 } }, turn_detection: { mode: 'default', config: { speech_threshold: 0.5, start_of_speech: { mode: 'vad', vad_config: { interrupt_duration_ms: 160, speaking_interrupt_duration_ms: 320, prefix_padding_ms: 800 } }, end_of_speech: { mode: 'semantic', semantic_config: {silence_duration_ms: 320, max_wait_ms: 3000} } } }, sal: { sal_mode: 'locking', sample_urls: {speaker1: 'https://example.com/voiceprint1.pcm'} }, labels: {campaign_id: 'spring_2025', customer_group: 'vip', region: 'eastus'}, rtc: { encryption_key: 'your_32_byte_encryption_key_here', encryption_salt: 'TsP4fLLyxxxxxxxxxroLA9oE=', encryption_mode: 8 }, filler_words: { enable: true, trigger: {mode: 'fixed_time', config: {response_wait_ms: 1500}}, content: { mode: 'static', config: {phrases: ['Please wait a moment.', 'OK', 'Uh-huh.'], selection_rule: 'shuffle'} } }, parameters: { silence_config: {timeout_ms: 15000, action: 'think', content: 'User has not responded for an extended period. Please inquire whether continued service is needed.'}, farewell_config: {graceful_enabled: true, graceful_timeout_seconds: 60}, data_channel: 'rtm', enable_metrics: true, enable_error_message: true } } }) }; fetch(url, options) .then(res => res.json()) .then(json => console.log(json)) .catch(err => console.error(err)); ``` **curl** ```bash curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \ --header 'Authorization: Basic' \ --data ' { "name": "unique_name", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "friday", "remote_rtc_uids": [ "1002" ], "enable_string_uid": true, "idle_timeout": 120, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "I'm sorry, I can't answer that question.", "max_history": 32, "params": { "model": "gpt-4o-mini" } }, "asr": { "language": "en-US" }, "tts": { "vendor": "microsoft", "params": { "key": "", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural" } } } }' ``` **Python** ```python import requests url = "https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join" payload = { "name": "unique_name", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "friday", "remote_rtc_uids": ["1002"], "enable_string_uid": True, "idle_timeout": 120, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "I'm sorry, I can't answer that question.", "max_history": 32, "params": { "model": "gpt-4o-mini" } }, "asr": { "language": "en-US" }, "tts": { "vendor": "microsoft", "params": { "key": "", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural" } } } } headers = {"Authorization": "Basic"} response = requests.post(url, json=payload, headers=headers) print(response.text) ``` **Node.js** ```js const url = 'https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join'; const options = { method: 'POST', headers: {Authorization: 'Basic'}, body: JSON.stringify({ name: 'unique_name', properties: { channel: 'channel_name', token: 'token', agent_rtc_uid: 'friday', remote_rtc_uids: ['1002'], enable_string_uid: true, idle_timeout: 120, llm: { url: 'https://api.openai.com/v1/chat/completions', api_key: '', system_messages: [{role: 'system', content: 'You are a helpful chatbot.'}], greeting_message: 'Hello, how can I help you?', failure_message: 'I'm sorry, I can't answer that question.', max_history: 32, params: {model: 'gpt-4o-mini'} }, asr: {language: 'en-US'}, tts: { vendor: 'microsoft', params: { key: '', region: 'eastus', voice_name: 'en-US-AndrewMultilingualNeural' } } } }) }; fetch(url, options) .then(res => res.json()) .then(json => console.log(json)) .catch(err => console.error(err)); ``` **curl** ```bash curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \ --header 'Authorization: Basic' \ --data ' { "name": "unique_name", "preset": "deepgram_nova_3,openai_gpt_5_mini,minimax_speech_2_6_turbo", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "1001", "remote_rtc_uids": [ "1002" ], "idle_timeout": 120, "llm": { "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "max_history": 32, "greeting_message": "Hello, how can I assist you today?", "failure_message": "Please hold on a second." }, "tts": { "vendor": "minimax", "params": { "voice_setting": { "voice_id": "English_captivating_female1" } } }, "asr": { "vendor": "deepgram", "params": { "language": "en", "keyterm": "agora" } } } }' ``` **Python** ```python import requests import base64 app_id = "" credentials = "" url = f"https://api.agora.io/api/conversational-ai-agent/v2/projects/{app_id}/join" headers = { "Authorization": f"Basic {credentials}", "Content-Type": "application/json" } payload = { "name": "unique_name", "preset": "deepgram_nova_3,openai_gpt_5_mini,minimax_speech_2_6_turbo", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "1001", "remote_rtc_uids": ["1002"], "idle_timeout": 120, "llm": { "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "max_history": 32, "greeting_message": "Hello, how can I assist you today?", "failure_message": "Please hold on a second." }, "tts": { "vendor": "minimax", "params": { "voice_setting": { "voice_id": "English_captivating_female1" } } }, "asr": { "vendor": "deepgram", "params": { "language": "en", "keyterm": "agora" } } } } response = requests.post(url, headers=headers, json=payload) print(response.status_code) print(response.json()) ``` **Node.js** ```js const appId = ""; const credentials = ""; const url = `https://api.agora.io/api/conversational-ai-agent/v2/projects/${appId}/join`; const payload = { name: "unique_name", preset: "deepgram_nova_3,openai_gpt_5_mini,minimax_speech_2_6_turbo", properties: { channel: "channel_name", token: "token", agent_rtc_uid: "1001", remote_rtc_uids: ["1002"], idle_timeout: 120, llm: { system_messages: [ { role: "system", content: "You are a helpful chatbot." } ], max_history: 32, greeting_message: "Hello, how can I assist you today?", failure_message: "Please hold on a second." }, tts: { vendor: "minimax", params: { voice_setting: { voice_id: "English_captivating_female1" } } }, asr: { vendor: "deepgram", params: { language: "en", keyterm: "agora" } } } }; const response = await fetch(url, { method: "POST", headers: { "Authorization": `Basic ${credentials}`, "Content-Type": "application/json" }, body: JSON.stringify(payload) }); const data = await response.json(); console.log(response.status); console.log(data); ``` ## Response example ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "create_ts": 1737111452, "status": "RUNNING" } ```