Voice agent realtime server events

These events are sent by the service on a voice-agent realtime connection.

These tables describe the event properties and link to their models.

Server events

conversation.item.added

Schema: OpenAI.RealtimeServerEventConversationItemAdded.

Sent by the server when an Item is added to the default Conversation. This can happen in several cases: - When the client sends a conversation.item.create event. - When the input audio buffer is committed. In this case the item will be a user message containing the audio from the buffer. - When the model is generating a Response. In this case the conversation.item.added event will be sent when the model starts generating a specific Item, and thus it will not yet have any content (and status will be in_progress). The event will include the full content of the Item (except when model is generating a Response) except for audio data, which can be retrieved separately with a conversation.item.retrieve event if necessary.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.added" Yes The event type, must be conversation.item.added.
previous_item_id Any of: string; null No
item OpenAI.RealtimeConversationItem Yes

Example event:

{
  "type": "conversation.item.added",
  "event_id": "event_C9G8pjSJCfRNEhMEnYAVy",
  "previous_item_id": null,
  "item": {
    "id": "item_C9G8pGVKYnaZu8PH5YQ9O",
    "type": "message",
    "status": "completed",
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "hi"
      }
    ]
  }
}

conversation.item.created

Schema: OpenAI.RealtimeServerEventConversationItemCreated.

Returned when a conversation item is created. There are several scenarios that produce this event: - The server is generating a Response, which if successful will produce either one or two Items, which will be of type message (role assistant) or type function_call. - The input audio buffer has been committed, either by the client or the server (in server_vad mode). The server will take the content of the input audio buffer and add it to a new user message Item. - The client has sent a conversation.item.create event to add a new Item to the Conversation.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.created" Yes The event type, must be conversation.item.created.
previous_item_id Any of: string; null No
item OpenAI.RealtimeConversationItem Yes

Example event:

{
  "event_id": "event_1920",
  "type": "conversation.item.created",
  "previous_item_id": "msg_002",
  "item": {
    "id": "msg_003",
    "object": "realtime.item",
    "type": "message",
    "status": "completed",
    "role": "user",
    "content": []
  }
}

conversation.item.deleted

Schema: OpenAI.RealtimeServerEventConversationItemDeleted.

Returned when an item in the conversation is deleted by the client with a conversation.item.delete event. This event is used to synchronize the server's understanding of the conversation history with the client's view.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.deleted" Yes The event type, must be conversation.item.deleted.
item_id string Yes The ID of the item that was deleted.

Example event:

{
  "event_id": "event_2728",
  "type": "conversation.item.deleted",
  "item_id": "msg_005"
}

conversation.item.done

Schema: OpenAI.RealtimeServerEventConversationItemDone.

Returned when a conversation item is finalized. The event will include the full content of the Item except for audio data, which can be retrieved separately with a conversation.item.retrieve event if needed.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.done" Yes The event type, must be conversation.item.done.
previous_item_id Any of: string; null No
item OpenAI.RealtimeConversationItem Yes

Example event:

{
  "type": "conversation.item.done",
  "event_id": "event_CCXLgMZPo3qioWCeQa4WH",
  "previous_item_id": "item_CCXLecNJVIVR2HUy3ABLj",
  "item": {
    "id": "item_CCXLfxmM5sXVJVz4mCa2S",
    "type": "message",
    "status": "completed",
    "role": "assistant",
    "content": [
      {
        "type": "output_audio",
        "transcript": "Oh, I can hear you loud and clear! Sounds like we're connected just fine. What can I help you with today?"
      }
    ]
  }
}

conversation.item.input_audio_transcription.completed

Schema: OpenAI.RealtimeServerEventConversationItemInputAudioTranscriptionCompleted.

This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (when VAD is enabled). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events. Realtime API models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model. The transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.input_audio_transcription.completed" Yes The event type, must be conversation.item.input_audio_transcription.completed.
item_id string Yes The ID of the item containing the audio that is being transcribed.
content_index OpenAI.integer Yes The index of the content part containing the audio.
transcript string Yes The transcribed text.
languages Array of OpenAI.TranscriptionLanguage No The languages detected in the audio. Returned by gpt-transcribe. An empty array indicates that no language could be reliably detected.
logprobs Any of: Array of OpenAI.LogProbProperties; null No
usage Exactly one of: OpenAI.TranscriptTextUsageTokens; OpenAI.TranscriptTextUsageDuration Yes Usage statistics for the transcription, this is billed according to the ASR model's pricing rather than the realtime model's pricing.
phrases Any of: Array of VoiceAgentTranscriptionPhrase; null No Phrase-level transcription timing and confidence details.

Example event:

{
  "type": "conversation.item.input_audio_transcription.completed",
  "event_id": "event_CCXGRvtUVrax5SJAnNOWZ",
  "item_id": "item_CCXGQ4e1ht4cOraEYcuR2",
  "content_index": 0,
  "transcript": "Hey, can you hear me?",
  "usage": {
    "type": "tokens",
    "total_tokens": 22,
    "input_tokens": 13,
    "input_token_details": {
      "text_tokens": 0,
      "audio_tokens": 13
    },
    "output_tokens": 9
  }
}

conversation.item.input_audio_transcription.delta

Schema: OpenAI.RealtimeServerEventConversationItemInputAudioTranscriptionDelta.

Returned when the text value of an input audio transcription content part is updated with incremental transcription results.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.input_audio_transcription.delta" Yes The event type, must be conversation.item.input_audio_transcription.delta.
item_id string Yes The ID of the item containing the audio that is being transcribed.
content_index OpenAI.integer No The index of the content part in the item's content array.
delta string No The text delta.
logprobs Any of: Array of OpenAI.LogProbProperties; null No

Example event:

{
  "type": "conversation.item.input_audio_transcription.delta",
  "event_id": "event_CCXGRxsAimPAs8kS2Wc7Z",
  "item_id": "item_CCXGQ4e1ht4cOraEYcuR2",
  "content_index": 0,
  "delta": "Hey",
  "obfuscation": "aLxx0jTEciOGe"
}

conversation.item.input_audio_transcription.failed

Schema: OpenAI.RealtimeServerEventConversationItemInputAudioTranscriptionFailed.

Returned when input audio transcription is configured, and a transcription request for a user message failed. These events are separate from other error events so that the client can identify the related Item.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.input_audio_transcription.failed" Yes The event type, must be conversation.item.input_audio_transcription.failed.
item_id string Yes The ID of the user message item.
content_index OpenAI.integer Yes The index of the content part containing the audio.
error OpenAI.RealtimeServerEventConversationItemInputAudioTranscriptionFailedError Yes Details of the transcription error.

conversation.item.input_audio_transcription.segment

Schema: OpenAI.RealtimeServerEventConversationItemInputAudioTranscriptionSegment.

Returned when an input audio transcription segment is identified for an item.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.input_audio_transcription.segment" Yes The event type, must be conversation.item.input_audio_transcription.segment.
item_id string Yes The ID of the item containing the input audio content.
content_index OpenAI.integer Yes The index of the input audio content part within the item.
text string Yes The text for this segment.
id string Yes The segment identifier.
speaker string Yes The detected speaker label for this segment.
start number Yes Start time of the segment in seconds. format: "double".
end number Yes End time of the segment in seconds. format: "double".

Example event:

{
  "event_id": "event_6501",
  "type": "conversation.item.input_audio_transcription.segment",
  "item_id": "msg_011",
  "content_index": 0,
  "text": "hello",
  "id": "seg_0001",
  "speaker": "spk_1",
  "start": 0,
  "end": 0.4
}

conversation.item.retrieved

Schema: OpenAI.RealtimeServerEventConversationItemRetrieved.

Returned when a conversation item is retrieved with conversation.item.retrieve. This is provided as a way to fetch the server's representation of an item, for example to get access to the post-processed audio data after noise cancellation and VAD. It includes the full content of the Item, including audio data.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.retrieved" Yes The event type, must be conversation.item.retrieved.
item OpenAI.RealtimeConversationItem Yes

Example event:

{
  "type": "conversation.item.retrieved",
  "event_id": "event_CCXGSizgEppa2d4XbKA7K",
  "item": {
    "id": "item_CCXGRxbY0n6WE4EszhF5w",
    "object": "realtime.item",
    "type": "message",
    "status": "completed",
    "role": "assistant",
    "content": [
      {
        "type": "output_audio",
        "transcript": "Yes, I can hear you loud and clear. How can I help you today?",
        "audio": "AAABAP//AAA="
      }
    ]
  }
}

conversation.item.truncated

Schema: OpenAI.RealtimeServerEventConversationItemTruncated.

Returned when an earlier assistant audio message item is truncated by the client with a conversation.item.truncate event. This event is used to synchronize the server's understanding of the audio with the client's playback. This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "conversation.item.truncated" Yes The event type, must be conversation.item.truncated.
item_id string Yes The ID of the assistant message item that was truncated.
content_index OpenAI.integer Yes The index of the content part that was truncated.
audio_end_ms OpenAI.integer Yes The duration up to which the audio was truncated, in milliseconds.
item OpenAI.RealtimeConversationItem No The assistant message after truncation, when the service returns the updated item.

Example event:

{
  "event_id": "event_2526",
  "type": "conversation.item.truncated",
  "item_id": "msg_004",
  "content_index": 0,
  "audio_end_ms": 1500
}

input_audio_buffer.cleared

Schema: OpenAI.RealtimeServerEventInputAudioBufferCleared.

Returned when the input audio buffer is cleared by the client with a input_audio_buffer.clear event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "input_audio_buffer.cleared" Yes The event type, must be input_audio_buffer.cleared.

Example event:

{
  "event_id": "event_1314",
  "type": "input_audio_buffer.cleared"
}

input_audio_buffer.committed

Schema: OpenAI.RealtimeServerEventInputAudioBufferCommitted.

Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "input_audio_buffer.committed" Yes The event type, must be input_audio_buffer.committed.
previous_item_id Any of: string; null No
item_id string Yes The ID of the user message item that will be created.

Example event:

{
  "event_id": "event_1121",
  "type": "input_audio_buffer.committed",
  "previous_item_id": "msg_001",
  "item_id": "msg_002"
}

input_audio_buffer.speech_started

Schema: OpenAI.RealtimeServerEventInputAudioBufferSpeechStarted.

Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user. The client should expect to receive a input_audio_buffer.speech_stopped event when speech stops. The item_id property is the ID of the user message item that will be created when speech stops and will also be included in the input_audio_buffer.speech_stopped event (unless the client manually commits the audio buffer during VAD activation).

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "input_audio_buffer.speech_started" Yes The event type, must be input_audio_buffer.speech_started.
audio_start_ms OpenAI.integer Yes Milliseconds from the start of all audio written to the buffer during the session when speech was first detected. This will correspond to the beginning of audio sent to the model, and thus includes the prefix_padding_ms configured in the Session.
item_id string Yes The ID of the user message item that will be created when speech stops.

Example event:

{
  "event_id": "event_1516",
  "type": "input_audio_buffer.speech_started",
  "audio_start_ms": 1000,
  "item_id": "msg_003"
}

input_audio_buffer.speech_stopped

Schema: OpenAI.RealtimeServerEventInputAudioBufferSpeechStopped.

Returned in server_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "input_audio_buffer.speech_stopped" Yes The event type, must be input_audio_buffer.speech_stopped.
audio_end_ms OpenAI.integer Yes Milliseconds since the session started when speech stopped. This will correspond to the end of audio sent to the model, and thus includes the min_silence_duration_ms configured in the Session.
item_id string Yes The ID of the user message item that will be created.

Example event:

{
  "event_id": "event_1718",
  "type": "input_audio_buffer.speech_stopped",
  "audio_end_ms": 2000,
  "item_id": "msg_003"
}

input_audio_buffer.timeout_triggered

Schema: OpenAI.RealtimeServerEventInputAudioBufferTimeoutTriggered.

Returned when the Server VAD timeout is triggered for the input audio buffer. This is configured with idle_timeout_ms in the turn_detection settings of the session, and it indicates that there hasn't been any speech detected for the configured duration. The audio_start_ms and audio_end_ms fields indicate the segment of audio after the last model response up to the triggering time, as an offset from the beginning of audio written to the input audio buffer. This means it demarcates the segment of audio that was silent and the difference between the start and end values will roughly match the configured timeout. The empty audio will be committed to the conversation as an input_audio item (there will be a input_audio_buffer.committed event) and a model response will be generated. There may be speech that didn't trigger VAD but is still detected by the model, so the model may respond with something relevant to the conversation or a prompt to continue speaking.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "input_audio_buffer.timeout_triggered" Yes The event type, must be input_audio_buffer.timeout_triggered.
audio_start_ms OpenAI.integer Yes Millisecond offset of audio written to the input audio buffer that was after the playback time of the last model response.
audio_end_ms OpenAI.integer Yes Millisecond offset of audio written to the input audio buffer at the time the timeout was triggered.
item_id string Yes The ID of the item associated with this segment.

Example event:

{
  "type": "input_audio_buffer.timeout_triggered",
  "event_id": "event_CEKKrf1KTGvemCPyiJTJ2",
  "audio_start_ms": 13216,
  "audio_end_ms": 19232,
  "item_id": "item_CEKKrWH0GiwN0ET97NUZc"
}

mcp_list_tools.completed

Schema: OpenAI.RealtimeServerEventMCPListToolsCompleted.

Returned when listing MCP tools has completed for an item.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "mcp_list_tools.completed" Yes The event type, must be mcp_list_tools.completed.
item_id string Yes The ID of the MCP list tools item.

Example event:

{
  "event_id": "event_6102",
  "type": "mcp_list_tools.completed",
  "item_id": "mcp_list_tools_001"
}

mcp_list_tools.failed

Schema: OpenAI.RealtimeServerEventMCPListToolsFailed.

Returned when listing MCP tools has failed for an item.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "mcp_list_tools.failed" Yes The event type, must be mcp_list_tools.failed.
item_id string Yes The ID of the MCP list tools item.

Example event:

{
  "event_id": "event_6103",
  "type": "mcp_list_tools.failed",
  "item_id": "mcp_list_tools_001"
}

mcp_list_tools.in_progress

Schema: OpenAI.RealtimeServerEventMCPListToolsInProgress.

Returned when listing MCP tools is in progress for an item.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "mcp_list_tools.in_progress" Yes The event type, must be mcp_list_tools.in_progress.
item_id string Yes The ID of the MCP list tools item.

Example event:

{
  "event_id": "event_6101",
  "type": "mcp_list_tools.in_progress",
  "item_id": "mcp_list_tools_001"
}

output_audio_buffer.cleared

Schema: OpenAI.RealtimeServerEventOutputAudioBufferCleared.

WebRTC/SIP Only: Emitted when the output audio buffer is cleared. This happens either in VAD mode when the user has interrupted (input_audio_buffer.speech_started), or when the client has emitted the output_audio_buffer.clear event to manually cut off the current audio response. Learn more.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "output_audio_buffer.cleared" Yes The event type, must be output_audio_buffer.cleared.
response_id string Yes The unique ID of the response that produced the audio.

Example event:

{
  "event_id": "event_abc123",
  "type": "output_audio_buffer.cleared",
  "response_id": "resp_abc123"
}

rate_limits.updated

Schema: OpenAI.RealtimeServerEventRateLimitsUpdated.

Emitted at the beginning of a Response to indicate the updated rate limits. When a Response is created some tokens will be "reserved" for the output tokens, the rate limits shown here reflect that reservation, which is then adjusted accordingly once the Response is completed.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "rate_limits.updated" Yes The event type, must be rate_limits.updated.
rate_limits Array of OpenAI.RealtimeServerEventRateLimitsUpdatedRateLimits Yes List of rate limit information.

Example event:

{
  "event_id": "event_5758",
  "type": "rate_limits.updated",
  "rate_limits": [
    {
      "name": "requests",
      "limit": 1000,
      "remaining": 999,
      "reset_seconds": 60
    },
    {
      "name": "tokens",
      "limit": 50000,
      "remaining": 49950,
      "reset_seconds": 60
    }
  ]
}

response.output_audio.delta

Schema: OpenAI.RealtimeServerEventResponseAudioDelta.

Returned when the model-generated audio is updated.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.output_audio.delta" Yes The event type, must be response.output_audio.delta.
response_id string Yes The ID of the response.
item_id string Yes The ID of the item.
output_index OpenAI.integer Yes The index of the output item in the response.
content_index OpenAI.integer Yes The index of the content part in the item's content array.
delta string Yes Base64-encoded audio data delta.

Example event:

{
  "event_id": "event_4950",
  "type": "response.output_audio.delta",
  "response_id": "resp_001",
  "item_id": "msg_008",
  "output_index": 0,
  "content_index": 0,
  "delta": "Base64EncodedAudioDelta"
}

response.output_audio.done

Schema: OpenAI.RealtimeServerEventResponseAudioDone.

Returned when the model-generated audio is done. Also emitted when a Response is interrupted, incomplete, or cancelled.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.output_audio.done" Yes The event type, must be response.output_audio.done.
response_id string Yes The ID of the response.
item_id string Yes The ID of the item.
output_index OpenAI.integer Yes The index of the output item in the response.
content_index OpenAI.integer Yes The index of the content part in the item's content array.

Example event:

{
  "event_id": "event_5152",
  "type": "response.output_audio.done",
  "response_id": "resp_001",
  "item_id": "msg_008",
  "output_index": 0,
  "content_index": 0
}

response.output_audio_transcript.delta

Schema: OpenAI.RealtimeServerEventResponseAudioTranscriptDelta.

Returned when the model-generated transcription of audio output is updated.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.output_audio_transcript.delta" Yes The event type, must be response.output_audio_transcript.delta.
response_id string Yes The ID of the response.
item_id string Yes The ID of the item.
output_index OpenAI.integer Yes The index of the output item in the response.
content_index OpenAI.integer Yes The index of the content part in the item's content array.
delta string Yes The transcript delta.

Example event:

{
  "event_id": "event_4546",
  "type": "response.output_audio_transcript.delta",
  "response_id": "resp_001",
  "item_id": "msg_008",
  "output_index": 0,
  "content_index": 0,
  "delta": "Hello, how can I a"
}

response.output_audio_transcript.done

Schema: OpenAI.RealtimeServerEventResponseAudioTranscriptDone.

Returned when the model-generated transcription of audio output is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.output_audio_transcript.done" Yes The event type, must be response.output_audio_transcript.done.
response_id string Yes The ID of the response.
item_id string Yes The ID of the item.
output_index OpenAI.integer Yes The index of the output item in the response.
content_index OpenAI.integer Yes The index of the content part in the item's content array.
transcript string Yes The final transcript of the audio.

Example event:

{
  "event_id": "event_4748",
  "type": "response.output_audio_transcript.done",
  "response_id": "resp_001",
  "item_id": "msg_008",
  "output_index": 0,
  "content_index": 0,
  "transcript": "Hello, how can I assist you today?"
}

response.content_part.added

Schema: OpenAI.RealtimeServerEventResponseContentPartAdded.

Returned when a new content part is added to an assistant message item during response generation.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.content_part.added" Yes The event type, must be response.content_part.added.
response_id string Yes The ID of the response.
item_id string Yes The ID of the item to which the content part was added.
output_index OpenAI.integer Yes The index of the output item in the response.
content_index OpenAI.integer Yes The index of the content part in the item's content array.
part OpenAI.RealtimeServerEventResponseContentPartAddedPart Yes The content part that was added.

Example event:

{
  "event_id": "event_3738",
  "type": "response.content_part.added",
  "response_id": "resp_001",
  "item_id": "msg_007",
  "output_index": 0,
  "content_index": 0,
  "part": {
    "type": "text",
    "text": ""
  }
}

response.content_part.done

Schema: OpenAI.RealtimeServerEventResponseContentPartDone.

Returned when a content part is done streaming in an assistant message item. Also emitted when a Response is interrupted, incomplete, or cancelled.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.content_part.done" Yes The event type, must be response.content_part.done.
response_id string Yes The ID of the response.
item_id string Yes The ID of the item.
output_index OpenAI.integer Yes The index of the output item in the response.
content_index OpenAI.integer Yes The index of the content part in the item's content array.
part OpenAI.RealtimeServerEventResponseContentPartDonePart Yes The content part that is done.

Example event:

{
  "event_id": "event_3940",
  "type": "response.content_part.done",
  "response_id": "resp_001",
  "item_id": "msg_007",
  "output_index": 0,
  "content_index": 0,
  "part": {
    "type": "text",
    "text": "Sure, I can help with that."
  }
}

response.created

Schema: OpenAI.RealtimeServerEventResponseCreated.

Returned when a new Response is created. The first event of response creation, where the response is in an initial state of in_progress.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.created" Yes The event type, must be response.created.
response VoiceAgentRealtimeResponse Yes

response.done

Schema: OpenAI.RealtimeServerEventResponseDone.

Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data. Clients should check the status field of the Response to determine if it was successful (completed) or if there was another outcome: cancelled, failed, or incomplete. A response will contain all output items that were generated during the response, excluding any audio content.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.done" Yes The event type, must be response.done.
response VoiceAgentRealtimeResponse Yes

response.function_call_arguments.delta

Schema: OpenAI.RealtimeServerEventResponseFunctionCallArgumentsDelta.

Returned when the model-generated function call arguments are updated.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.function_call_arguments.delta" Yes The event type, must be response.function_call_arguments.delta.
response_id string Yes The ID of the response.
item_id string Yes The ID of the function call item.
output_index OpenAI.integer Yes The index of the output item in the response.
call_id string Yes The ID of the function call.
delta string Yes The arguments delta as a JSON string.

response.function_call_arguments.done

Schema: OpenAI.RealtimeServerEventResponseFunctionCallArgumentsDone.

Returned when the model-generated function call arguments are done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.function_call_arguments.done" Yes The event type, must be response.function_call_arguments.done.
response_id string Yes The ID of the response.
item_id string Yes The ID of the function call item.
output_index OpenAI.integer Yes The index of the output item in the response.
call_id string Yes The ID of the function call.
name string Yes The name of the function that was called.
arguments string Yes The final arguments as a JSON string.

response.system_tool_arguments.delta

Schema: VoiceAgentServerEventSystemToolArgumentsDelta.

An incremental model-argument fragment for a service-managed system tool.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.system_tool_arguments.delta" Yes The event type. Always response.system_tool_arguments.delta.
event_id string Yes The unique identifier of this server event.
response_id string Yes The response containing the tool invocation.
item_id string Yes The generated tool item's identifier.
output_index integer Yes The item's zero-based position in the response output. format: "int32".
call_id string Yes The invocation identifier.
delta string Yes An incremental JSON fragment of the intent arguments.
obfuscation string No Optional padding metadata with no application semantics.

response.system_tool_arguments.done

Schema: VoiceAgentServerEventSystemToolArgumentsDone.

The completed intent arguments for a system tool, not provider success or a request for a customer function result.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.system_tool_arguments.done" Yes The event type. Always response.system_tool_arguments.done.
event_id string Yes The unique identifier of this server event.
response_id string Yes The response containing the tool invocation.
item_id string Yes The generated tool item's identifier.
output_index integer Yes The item's zero-based position in the response output. format: "int32".
call_id string Yes The invocation identifier.
name VoiceAgentSystemToolName Yes The invoked built-in capability.
arguments string Yes The complete, unexpanded model arguments as a JSON string.

input_collection.status

Schema: VoiceAgentServerEventInputCollectionStatus.

Reports collection or telephony outcomes for a model call.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "input_collection.status" Yes The event type. Always input_collection.status.
event_id string No An optional event identifier.
call_id string Yes The originating system-tool invocation.
collection_id string No The collection operation identifier, when this outcome concerns a collection.
request_id string No The current receiver request identifier, when applicable.
channel VoiceAgentInputCollectionChannel No The collection receiver, omitted for telephony-operation feedback.
status VoiceAgentInputCollectionStatus Yes The reported outcome. Generation completion of a system-tool item is separate.
value string No Completed ASCII digits, including leading zeros, only for secret: false; absent for private collection and telephony-operation feedback.
output_ref VoiceAgentOutputReference No Actual bare variable name from successful private collection, not its value. May be suffixed on collision; reference it with {{name}}.

response.mcp_call_arguments.delta

Schema: OpenAI.RealtimeServerEventResponseMCPCallArgumentsDelta.

Returned when MCP tool call arguments are updated during response generation.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.mcp_call_arguments.delta" Yes The event type, must be response.mcp_call_arguments.delta.
response_id string Yes The ID of the response.
item_id string Yes The ID of the MCP tool call item.
output_index OpenAI.integer Yes The index of the output item in the response.
delta string Yes The JSON-encoded arguments delta.
obfuscation Any of: string; null No

response.mcp_call_arguments.done

Schema: OpenAI.RealtimeServerEventResponseMCPCallArgumentsDone.

Returned when MCP tool call arguments are finalized during response generation.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.mcp_call_arguments.done" Yes The event type, must be response.mcp_call_arguments.done.
response_id string Yes The ID of the response.
item_id string Yes The ID of the MCP tool call item.
output_index OpenAI.integer Yes The index of the output item in the response.
arguments string Yes The final JSON-encoded arguments string.

response.mcp_call.completed

Schema: OpenAI.RealtimeServerEventResponseMCPCallCompleted.

Returned when an MCP tool call has completed successfully.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.mcp_call.completed" Yes The event type, must be response.mcp_call.completed.
output_index OpenAI.integer Yes The index of the output item in the response.
item_id string Yes The ID of the MCP tool call item.

Example event:

{
  "event_id": "event_6302",
  "type": "response.mcp_call.completed",
  "output_index": 0,
  "item_id": "mcp_call_001"
}

response.mcp_call.failed

Schema: OpenAI.RealtimeServerEventResponseMCPCallFailed.

Returned when an MCP tool call has failed.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.mcp_call.failed" Yes The event type, must be response.mcp_call.failed.
output_index OpenAI.integer Yes The index of the output item in the response.
item_id string Yes The ID of the MCP tool call item.

Example event:

{
  "event_id": "event_6303",
  "type": "response.mcp_call.failed",
  "output_index": 0,
  "item_id": "mcp_call_001"
}

response.mcp_call.in_progress

Schema: OpenAI.RealtimeServerEventResponseMCPCallInProgress.

Returned when an MCP tool call has started and is in progress.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.mcp_call.in_progress" Yes The event type, must be response.mcp_call.in_progress.
output_index OpenAI.integer Yes The index of the output item in the response.
item_id string Yes The ID of the MCP tool call item.

Example event:

{
  "event_id": "event_6301",
  "type": "response.mcp_call.in_progress",
  "output_index": 0,
  "item_id": "mcp_call_001"
}

response.output_item.added

Schema: OpenAI.RealtimeServerEventResponseOutputItemAdded.

Returned when a new Item is created during Response generation.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.output_item.added" Yes The event type, must be response.output_item.added.
response_id string Yes The ID of the Response to which the item belongs.
output_index OpenAI.integer Yes The index of the output item in the Response.
item OpenAI.RealtimeConversationItem Yes

Example event:

{
  "event_id": "event_3334",
  "type": "response.output_item.added",
  "response_id": "resp_001",
  "output_index": 0,
  "item": {
    "id": "msg_007",
    "object": "realtime.item",
    "type": "message",
    "status": "in_progress",
    "role": "assistant",
    "content": []
  }
}

response.output_item.done

Schema: OpenAI.RealtimeServerEventResponseOutputItemDone.

Returned when an Item is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.output_item.done" Yes The event type, must be response.output_item.done.
response_id string Yes The ID of the Response to which the item belongs.
output_index OpenAI.integer Yes The index of the output item in the Response.
item OpenAI.RealtimeConversationItem Yes

Example event:

{
  "event_id": "event_3536",
  "type": "response.output_item.done",
  "response_id": "resp_001",
  "output_index": 0,
  "item": {
    "id": "msg_007",
    "object": "realtime.item",
    "type": "message",
    "status": "completed",
    "role": "assistant",
    "content": [
      {
        "type": "output_text",
        "text": "Sure, I can help with that."
      }
    ]
  }
}

response.output_text.delta

Schema: OpenAI.RealtimeServerEventResponseTextDelta.

Returned when the text value of an "output_text" content part is updated.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.output_text.delta" Yes The event type, must be response.output_text.delta.
response_id string Yes The ID of the response.
item_id string Yes The ID of the item.
output_index OpenAI.integer Yes The index of the output item in the response.
content_index OpenAI.integer Yes The index of the content part in the item's content array.
delta string Yes The text delta.

Example event:

{
  "event_id": "event_4142",
  "type": "response.output_text.delta",
  "response_id": "resp_001",
  "item_id": "msg_007",
  "output_index": 0,
  "content_index": 0,
  "delta": "Sure, I can h"
}

response.output_text.done

Schema: OpenAI.RealtimeServerEventResponseTextDone.

Returned when the text value of an "output_text" content part is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "response.output_text.done" Yes The event type, must be response.output_text.done.
response_id string Yes The ID of the response.
item_id string Yes The ID of the item.
output_index OpenAI.integer Yes The index of the output item in the response.
content_index OpenAI.integer Yes The index of the content part in the item's content array.
text string Yes The final text content.

Example event:

{
  "event_id": "event_4344",
  "type": "response.output_text.done",
  "response_id": "resp_001",
  "item_id": "msg_007",
  "output_index": 0,
  "content_index": 0,
  "text": "Sure, I can help with that."
}

session.created

Schema: OpenAI.RealtimeServerEventSessionCreated.

Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "session.created" Yes The event type, must be session.created.
session VoiceAgentSessionResponse Yes The session configuration.
conversation_id string No The session-scoped conversation id. When present, responses attached to the session conversation use the same value in response.created and response.done.

session.subagent.started

Schema: VoiceAgentServerEventSessionSubagentStarted.

The session.subagent.started server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "session.subagent.started" Yes The event type. Always session.subagent.started.
event_id string Yes The server-generated event identifier.
consultation_id string Yes The identifier of the subagent consultation.
call_id string Yes The identifier of the function call that initiated the consultation.
subagent_name string Yes The name of the consulted subagent.

session.subagent.completed

Schema: VoiceAgentServerEventSessionSubagentCompleted.

The session.subagent.completed server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "session.subagent.completed" Yes The event type. Always session.subagent.completed.
event_id string Yes The server-generated event identifier.
consultation_id string Yes The identifier of the subagent consultation.
call_id string Yes The identifier of the function call that initiated the consultation.
subagent_name string Yes The name of the consulted subagent.

session.subagent.aborted

Schema: VoiceAgentServerEventSessionSubagentAborted.

The session.subagent.aborted server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "session.subagent.aborted" Yes The event type. Always session.subagent.aborted.
event_id string Yes The server-generated event identifier.
consultation_id string Yes The identifier of the subagent consultation.
call_id string Yes The identifier of the function call that initiated the consultation.
subagent_name string Yes The name of the consulted subagent.
reason VoiceAgentSubagentAbortReason Yes The reason the consultation was aborted.

session.updated

Schema: OpenAI.RealtimeServerEventSessionUpdated.

Returned when a session is updated with a session.update event, unless there is an error.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "session.updated" Yes The event type, must be session.updated.
session VoiceAgentSessionResponse Yes The session configuration.

error

Schema: OpenAI.RealtimeServerEventError.

Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementers to monitor and log error messages by default.

Property Type Required Description
event_id string Yes The unique ID of the server event.
type "error" Yes The event type, must be error.
error OpenAI.RealtimeServerEventErrorError Yes Details of the error.

Example event:

{
  "event_id": "event_890",
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "code": "invalid_event",
    "message": "The 'type' field is missing.",
    "param": null,
    "event_id": "event_567"
  }
}

warning

Schema: VoiceAgentServerEventWarning.

The warning server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "warning" Yes
event_id string Yes
warning VoiceAgentServerEventWarningDetails Yes

session.avatar.connecting

Schema: VoiceAgentServerEventSessionAvatarConnecting.

The session.avatar.connecting server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "session.avatar.connecting" Yes
event_id string Yes
server_sdp string Yes The server's SDP answer for avatar media negotiation.

rtc.call.sdp.created

Schema: VoiceAgentServerEventRtcCallSdpCreated.

The rtc.call.sdp.created server event: the SDP answer that completes WebRTC negotiation.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "rtc.call.sdp.created" Yes The event type. Always rtc.call.sdp.created.
event_id string Yes The server-generated event identifier.
rtc_call_id string Yes The identifier of the established WebRTC call.
sdp_answer string Yes The server's SDP answer for the WebRTC connection.

rtc.call.error

Schema: VoiceAgentServerEventRtcCallError.

The rtc.call.error server event: a WebRTC signaling failure.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "rtc.call.error" Yes The event type. Always rtc.call.error.
event_id string No An optional server-generated event identifier.
operation string No The signaling operation that failed, when known.
rtc_call_id string No The identifier of the WebRTC call, when known.
error VoiceAgentRtcCallErrorDetails Yes The error detail.

session.avatar.switch_to_speaking

Schema: VoiceAgentServerEventSessionAvatarSwitchToSpeaking.

The session.avatar.switch_to_speaking server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "session.avatar.switch_to_speaking" Yes
event_id string Yes
turn_id string No

session.avatar.switch_to_idle

Schema: VoiceAgentServerEventSessionAvatarSwitchToIdle.

The session.avatar.switch_to_idle server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "session.avatar.switch_to_idle" Yes
event_id string Yes
turn_id string No

response.audio_timestamp.delta

Schema: VoiceAgentServerEventResponseAudioTimestampDelta.

The response.audio_timestamp.delta server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.audio_timestamp.delta" Yes
event_id string Yes
response_id string Yes
item_id string Yes
output_index integer Yes format: "int32".
content_index integer Yes format: "int32".
audio_offset_ms FoundryDurationMilliseconds Yes
audio_duration_ms FoundryDurationMilliseconds Yes
text string Yes
timestamp_type "word" Yes

response.audio_timestamp.done

Schema: VoiceAgentServerEventResponseAudioTimestampDone.

The response.audio_timestamp.done server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.audio_timestamp.done" Yes
event_id string Yes
response_id string Yes
item_id string Yes
output_index integer Yes format: "int32".
content_index integer Yes format: "int32".

response.animation_blendshapes.delta

Schema: VoiceAgentServerEventResponseAnimationBlendshapesDelta.

The response.animation_blendshapes.delta server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.animation_blendshapes.delta" Yes
event_id string Yes
response_id string Yes
item_id string Yes
output_index integer Yes format: "int32".
content_index integer Yes format: "int32".
frames Array of Array of number (format: "float".) Yes Animation frames as numeric blendshape weights.
frame_index integer Yes The index of the first frame in this delta. format: "int32".

response.animation_blendshapes.done

Schema: VoiceAgentServerEventResponseAnimationBlendshapesDone.

The response.animation_blendshapes.done server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.animation_blendshapes.done" Yes
event_id string Yes
response_id string Yes
item_id string Yes
output_index integer Yes format: "int32".

response.animation_viseme.delta

Schema: VoiceAgentServerEventResponseAnimationVisemeDelta.

The response.animation_viseme.delta server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.animation_viseme.delta" Yes
event_id string Yes
response_id string Yes
item_id string Yes
output_index integer Yes format: "int32".
content_index integer Yes format: "int32".
audio_offset_ms FoundryDurationMilliseconds Yes
viseme_id integer Yes format: "int32".

response.animation_viseme.done

Schema: VoiceAgentServerEventResponseAnimationVisemeDone.

The response.animation_viseme.done server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.animation_viseme.done" Yes
event_id string Yes
response_id string Yes
item_id string Yes
output_index integer Yes format: "int32".
content_index integer Yes format: "int32".

response.video.delta

Schema: VoiceAgentServerEventResponseVideoDelta.

The response.video.delta server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
type "response.video.delta" Yes
event_id string Yes
output_index integer Yes format: "int32".
codec string Yes
delta string Yes The base64-encoded video frame data.

session.handoff.started

Schema: VoiceAgentServerEventSessionHandoffStarted.

The session.handoff.started server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The server-generated event identifier.
handoff_id string Yes The server-generated handoff attempt identifier.
edge_id VoiceAgentHandoffId Yes The configured edge selected for this handoff.
from_node_id VoiceAgentHandoffNodeReference Yes The source node identifier or $entrypoint.
to_node_id VoiceAgentHandoffNodeReference Yes The target node identifier or $entrypoint.
from_model string Yes The model active before the handoff.
to_model string Yes The model prepared for the target behavior.
tool_call_id string Yes The function-call identifier that initiated the handoff.
node_generation integer Yes The source generation for started or aborted events, and the committed target generation for completed events. format: "int32". minimum: 1.
type "session.handoff.started" Yes The event type. Always session.handoff.started.

Required preview opt-ins: VoiceAgents=V1Preview.

session.handoff.completed

Schema: VoiceAgentServerEventSessionHandoffCompleted.

The session.handoff.completed server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The server-generated event identifier.
handoff_id string Yes The server-generated handoff attempt identifier.
edge_id VoiceAgentHandoffId Yes The configured edge selected for this handoff.
from_node_id VoiceAgentHandoffNodeReference Yes The source node identifier or $entrypoint.
to_node_id VoiceAgentHandoffNodeReference Yes The target node identifier or $entrypoint.
from_model string Yes The model active before the handoff.
to_model string Yes The model prepared for the target behavior.
tool_call_id string Yes The function-call identifier that initiated the handoff.
node_generation integer Yes The source generation for started or aborted events, and the committed target generation for completed events. format: "int32". minimum: 1.
prepare_duration_ms FoundryDurationMilliseconds Yes The target-preparation duration.
duration_ms FoundryDurationMilliseconds Yes The total handoff duration.
type "session.handoff.completed" Yes The event type. Always session.handoff.completed.

Required preview opt-ins: VoiceAgents=V1Preview.

session.handoff.aborted

Schema: VoiceAgentServerEventSessionHandoffAborted.

The session.handoff.aborted server event.

All of: OpenAI.RealtimeServerEvent.

Property Type Required Description
event_id string Yes The server-generated event identifier.
handoff_id string Yes The server-generated handoff attempt identifier.
edge_id VoiceAgentHandoffId Yes The configured edge selected for this handoff.
from_node_id VoiceAgentHandoffNodeReference Yes The source node identifier or $entrypoint.
to_node_id VoiceAgentHandoffNodeReference Yes The target node identifier or $entrypoint.
from_model string Yes The model active before the handoff.
to_model string Yes The model prepared for the target behavior.
tool_call_id string Yes The function-call identifier that initiated the handoff.
node_generation integer Yes The source generation for started or aborted events, and the committed target generation for completed events. format: "int32". minimum: 1.
reason VoiceAgentHandoffAbortReason Yes Why the handoff ended before target commit.
error Any of: OpenAI.RealtimeServerEventErrorError; null No The handoff failure detail when reason is error.
type "session.handoff.aborted" Yes The event type. Always session.handoff.aborted.

Required preview opt-ins: VoiceAgents=V1Preview.