Audio-Video chat using Open AI (i think GPT 4o)

Question

Audio-Video chat using Open AI (i think GPT 4o)

GenixPRO 181

Hi Team,

We're unable to find any quickstarts or API documentation to implement audio-video chat in Open AI model. Likely GPT 4o as per this video https://www.youtube.com/watch?v=vgYi3Wr7v_g

Appreciate your help with API documentation to implement this.

Thanks.

Saideep Anchuri 9,545 Reputation points Moderator

2024-12-12T07:58:03.99+00:00

Hi GenixPRO

Following up to see if the given response was helpful.

Thank You.
GenixPRO 181 Reputation points

2024-12-12T11:43:33.5166667+00:00

Thanks @saideep anchuri Open AI documentation suggests that GPT 40 or 40-realtime can be used for video as well, and that there's not new API. So we're wondering whether video is currently restricted or should we try to pass video stream and see what happens :) We do need real-time audio/video for our use case. Thanks.
Saideep Anchuri 9,545 Reputation points Moderator

2024-12-13T09:26:08.91+00:00

Hi GenixPRO

There is no direct approach to analyze videos straight away.

but You can split a video into individual frames and analyze those image frames using GPT-4o

kindly refer below Document for reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/gpt-v-quickstart?tabs=command-line%2Cjavascript-keyless%2Ctypescript-keyless&pivots=programming-language-python (using GPT4o turbo vision api)

https://cookbook.openai.com/examples/gpt_with_vision_for_video_understanding

Thank You.
GenixPRO 181 Reputation points

2024-12-13T11:45:44.1766667+00:00

@saideep anchuri our use case is real-time video chat with GPT. Can this be done with any of the existing models? Open AI seems to be releasing many YT videos (https://www.youtube.com/watch?v=vgYi3Wr7v_g) demo'ing this functionality. Which API supports this? Thanks.
Saideep Anchuri 9,545 Reputation points Moderator

2024-12-16T08:02:06.14+00:00

Hi GenixPRO

OpenAI has kindly introduced the Realtime API, which offers robust support for real-time video chat functionality. This API empowers developers to create low-latency, multimodal conversational experiences, including natural speech-to-speech communication. It seamlessly handles both text and audio inputs and outputs, making it ideal for real-time video chat applications.

kindly refer the below Document: https://openai.com/index/introducing-the-realtime-api/

Thank you.
GenixPRO 181 Reputation points

2024-12-19T07:43:31.07+00:00

@Saideep Anchuri are you suggesting that Realtime API can be used for audio AND video communication? We're unable to find documentation for video support. We're aware that Realtime API supports low-latency real-time audio. Is there any documentation for video?
Saideep Anchuri 9,545 Reputation points Moderator

2024-12-19T08:35:43.1666667+00:00

Hi GenixPRO

it seems that real-time audio and video chat is a new feature available through OpenAI, but it is not yet supported by Azure OpenAI, which currently offers only real-time audio.

Please keep an eye on our official documentation for future updates.

Thank You.

1 answer

Your answer

Saideep Anchuri 9,545 Reputation points Moderator

2024-12-12T07:58:03.99+00:00

Hi GenixPRO

Following up to see if the given response was helpful.

Thank You.
GenixPRO 181 Reputation points

2024-12-12T11:43:33.5166667+00:00

Thanks @saideep anchuri Open AI documentation suggests that GPT 40 or 40-realtime can be used for video as well, and that there's not new API. So we're wondering whether video is currently restricted or should we try to pass video stream and see what happens :) We do need real-time audio/video for our use case. Thanks.
Saideep Anchuri 9,545 Reputation points Moderator

2024-12-13T09:26:08.91+00:00

Hi GenixPRO

There is no direct approach to analyze videos straight away.

but You can split a video into individual frames and analyze those image frames using GPT-4o

kindly refer below Document for reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/gpt-v-quickstart?tabs=command-line%2Cjavascript-keyless%2Ctypescript-keyless&pivots=programming-language-python (using GPT4o turbo vision api)

https://cookbook.openai.com/examples/gpt_with_vision_for_video_understanding

Thank You.
GenixPRO 181 Reputation points

2024-12-13T11:45:44.1766667+00:00

@saideep anchuri our use case is real-time video chat with GPT. Can this be done with any of the existing models? Open AI seems to be releasing many YT videos (https://www.youtube.com/watch?v=vgYi3Wr7v_g) demo'ing this functionality. Which API supports this? Thanks.
Saideep Anchuri 9,545 Reputation points Moderator

2024-12-16T08:02:06.14+00:00

Hi GenixPRO

OpenAI has kindly introduced the Realtime API, which offers robust support for real-time video chat functionality. This API empowers developers to create low-latency, multimodal conversational experiences, including natural speech-to-speech communication. It seamlessly handles both text and audio inputs and outputs, making it ideal for real-time video chat applications.

kindly refer the below Document: https://openai.com/index/introducing-the-realtime-api/

Thank you.
GenixPRO 181 Reputation points

2024-12-19T07:43:31.07+00:00

@Saideep Anchuri are you suggesting that Realtime API can be used for audio AND video communication? We're unable to find documentation for video support. We're aware that Realtime API supports low-latency real-time audio. Is there any documentation for video?
Saideep Anchuri 9,545 Reputation points Moderator

2024-12-19T08:35:43.1666667+00:00

Hi GenixPRO

it seems that real-time audio and video chat is a new feature available through OpenAI, but it is not yet supported by Azure OpenAI, which currently offers only real-time audio.

Please keep an eye on our official documentation for future updates.

Thank You.

Answer 1

Hi GenixPRO

Welcome to Microsoft Q&A Forum, thank you for posting your query here!

Currently, audio-video chat is not available in Azure OpenAI. The GPT-4o family of models supports text, audio, and image inputs but does not yet support audio-video chat functionality. There is no ETA for when this feature might be introduced. Microsoft plans to implement this in the future, so we recommend keeping an eye on the OpenAI what's-new for updates.

kindly refer the below document: https://learn.microsoft.com/en-us/azure/ai-services/openai/realtime-audio-quickstart?source=recommendations&pivots=ai-foundry-portal

https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=python-secure%2Cglobal-standard%2Cstandard-chat-completions

Thank You.

Share via

Audio-Video chat using Open AI (i think GPT 4o)

1 answer

Your answer