Prompt Flow of image with text input in Azure AI Foundry

Question

Prompt Flow of image with text input in Azure AI Foundry

Yoke Huan Shum 40

Hi, I'm seeking help with creating a prompt flow in Azure AI Foundry.

I'm trying to create a prompt flow that can handle both image and text inputs simultaneously. I was referring to this link do it and using the GPT-4o model. However, I couldn't make it work.

It's showing this error when I test the prompt flow in the chat session.

User's image

Appreciate if anyone can help me with this. Thanks a lot!

navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-03T04:43:30.0766667+00:00
@Yoke Huan Shum Welcome to Microsoft Q&A Forum, Thank you for posting your query here! .

I have tried following this article to build a chatbot to process images but I am unable to reproduce this error. It worked fine for me:

.

I have not done any modifications or customization to the prompt and kept it very simple.

Questions:

Could you please share the screenshot of your prompt flow ?

Please collect browser HAR trace while reproding this error and share the traces here.

Please confirm if you are using the Azure Open AI GPT4-turbo model (GPT4-Vision).

Do you have any content safety created for this model ?

Instead of using Azure AI Foundry, Could you please use ML Studio and check if that helps ?

Awaiting your reply.
Yoke Huan Shum 40 Reputation points

2024-12-03T05:28:42.27+00:00

@navba-MSFT Thank you for your reply!

I did try the prompt flow without any modification. It's working right now. Thanks!

However, I still couldn't prompt with image and text simultaneously, even though I have changed the input type to 'List' where it should be available for mixed queries.

Do you have any idea where I should modify it to allow both image and text input?

Thanks for the help and looking forward to your reply!
navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-03T09:36:08.3866667+00:00

@Yoke Huan Shum I am able to reproduce this issue at my end. Even after changing the type as list the chat is still not taking the text with image input.

However our documentation says below:

To enable your chatbot to respond with rich text and images, make the chat output list type. The list should consist of strings (for text) and prompt flow Image objects (for images) in custom order.

I am discussing this internally to know if this is a bug or design behavior. I will keep you posted. Apologies for the inconvenience caused due to this.
Farah Diana Masri 40 Reputation points

2024-12-04T01:07:02.5633333+00:00

Hi @navba-MSFT I am interested to know the answer from Yoke Huan's question also about the list type. Hope you can share it once you know the steps. Thank you.
navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-04T04:35:59.0633333+00:00

@Yoke Huan Shum @Farah Diana Masri I am discussing this internally. I have reached out to the Product owners. I will keep you posted.
Farah Diana Masri 40 Reputation points

2024-12-09T07:31:33.2033333+00:00

Hi @navba-MSFT . Do you get a chance to find out about the list type with Product owner? We are still interested to implement this functionality into our custom solution. Thank you.
navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-09T08:39:22.8733333+00:00

@Yoke Huan Shum @Farah Diana Masri Apologies for the late reply. I am yet to hear from the Product Owners. I have reached out to them internally and awaiting their reply. I will get back once I have an update.

Answer accepted by question author

0 additional answers

Your answer

navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-03T04:43:30.0766667+00:00

@Yoke Huan Shum Welcome to Microsoft Q&A Forum, Thank you for posting your query here! .

I have tried following this article to build a chatbot to process images but I am unable to reproduce this error. It worked fine for me:

.

I have not done any modifications or customization to the prompt and kept it very simple.

Questions:

Could you please share the screenshot of your prompt flow ?

Please collect browser HAR trace while reproding this error and share the traces here.

Please confirm if you are using the Azure Open AI GPT4-turbo model (GPT4-Vision).

Do you have any content safety created for this model ?

Instead of using Azure AI Foundry, Could you please use ML Studio and check if that helps ?

Awaiting your reply.
Yoke Huan Shum 40 Reputation points

2024-12-03T05:28:42.27+00:00

@navba-MSFT Thank you for your reply!

I did try the prompt flow without any modification. It's working right now. Thanks!

However, I still couldn't prompt with image and text simultaneously, even though I have changed the input type to 'List' where it should be available for mixed queries.

Do you have any idea where I should modify it to allow both image and text input?

Thanks for the help and looking forward to your reply!
navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-03T09:36:08.3866667+00:00

@Yoke Huan Shum I am able to reproduce this issue at my end. Even after changing the type as list the chat is still not taking the text with image input.

However our documentation says below:

To enable your chatbot to respond with rich text and images, make the chat output list type. The list should consist of strings (for text) and prompt flow Image objects (for images) in custom order.

I am discussing this internally to know if this is a bug or design behavior. I will keep you posted. Apologies for the inconvenience caused due to this.
Farah Diana Masri 40 Reputation points

2024-12-04T01:07:02.5633333+00:00

Hi @navba-MSFT I am interested to know the answer from Yoke Huan's question also about the list type. Hope you can share it once you know the steps. Thank you.
navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-04T04:35:59.0633333+00:00

@Yoke Huan Shum @Farah Diana Masri I am discussing this internally. I have reached out to the Product owners. I will keep you posted.
Farah Diana Masri 40 Reputation points

2024-12-09T07:31:33.2033333+00:00

Hi @navba-MSFT . Do you get a chance to find out about the list type with Product owner? We are still interested to implement this functionality into our custom solution. Thank you.
navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-09T08:39:22.8733333+00:00

@Yoke Huan Shum @Farah Diana Masri Apologies for the late reply. I am yet to hear from the Product Owners. I have reached out to them internally and awaiting their reply. I will get back once I have an update.

Answer 1

navba-MSFT 27,605 Microsoft Employee Moderator

@Yoke Huan Shum @Farah Diana Masri Apologies for the delay.

.

Fix:

I had reached out to the Product Owners in the background. They said the fix will be deployed for this by end of day tomorrow. You can try to test this post that.

.

On a side note:

You should be using the Azure OpenAI GPT-4 Turbo with Vision tool with the GPT-4 (vision-preview version) model.

User's image

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

navba-MSFT 27,605 Reputation points Microsoft Employee Moderator

2024-12-14T09:23:55.9833333+00:00

@Yoke Huan Shum @Farah Diana Masri A quick follow-up to check if you had a chance to look at my above answer. Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Farah Diana Masri 40 Reputation points

2024-12-18T00:42:48.9166667+00:00

Hi @navba-MSFT could you please help us giving insights on how to consume this image prompt flow endpoint? Because we have received this error, can you give any ideas on how to call this endpoint in Postman first? Thank you so much
Rushik Ramani 0 Reputation points

2025-05-08T21:34:18.0633333+00:00

GPT-4T is old model, do we have a concrete answer how to integrate image to text inference in Prompt Flow with model like GPT-4o?

Share via

Prompt Flow of image with text input in Azure AI Foundry

0 additional answers

Your answer