Azure OpenAI Batch Processing Doesn't Work With Structured Response

Question

Azure OpenAI Batch Processing Doesn't Work With Structured Response

Mike 5

I am trying to batch process a large amount of data with the batch API and a structured response schema.

When I upload my batch file it validates and processes but the output file is always empty.

My structured response schema is about 90 properties so close to the limit.

If I run a smaller prompt with a small structured response of 5 properties then batch processing will complete.

If I run a single request with my full prompt and full schema it will complete but it takes a while to complete.

I am guessing here that batches have some sort of timeout value that my request hits but it doesn't get marked as failed it gets marked as completed with an empty file.

I have a large amount of data to process, I really need to use the batch API and my input is completely valid but it seems some internal bug is preventing it from working.

How do I report this bug and get it working?

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-10-16T08:07:09.38+00:00
Mike Greetings!

I understand that it works with small structured responses. Are you seeing any error messages?

Currently only gpt-4o version: 2024-08-06 supports structured outputs.

As mentioned here, there are some limitations with Structured outputs.

Currently structured outputs is not supported on bring your own data scenario.

There is a known issue blocking structured outputs support for global batch.
Mike 5 Reputation points

2024-10-16T20:43:38.6666667+00:00

It turns out Azure batch inputs are completely broken with structured outputs. I thought it was working with the smaller schema I ran last week and it was the fact I was running a larger schema that was the issue but nope, in that time something has broken on Microsoft end. Also the GUI shows there is no error file but the error file can be obtained via API which misled me a bit.

Very frustrating. I really need to run a batch with structured outputs. It was working previously, any idea when it will be fixed?
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-10-18T13:02:18.2866667+00:00

Mike I will check internally and confirm on the same.

1 answer

Your answer

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-10-16T08:07:09.38+00:00

Mike Greetings!

I understand that it works with small structured responses. Are you seeing any error messages?

Currently only gpt-4o version: 2024-08-06 supports structured outputs.

As mentioned here, there are some limitations with Structured outputs.

Currently structured outputs is not supported on bring your own data scenario.

There is a known issue blocking structured outputs support for global batch.
Mike 5 Reputation points

2024-10-16T20:43:38.6666667+00:00

It turns out Azure batch inputs are completely broken with structured outputs. I thought it was working with the smaller schema I ran last week and it was the fact I was running a larger schema that was the issue but nope, in that time something has broken on Microsoft end. Also the GUI shows there is no error file but the error file can be obtained via API which misled me a bit.

Very frustrating. I really need to run a batch with structured outputs. It was working previously, any idea when it will be fixed?
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-10-18T13:02:18.2866667+00:00

Mike I will check internally and confirm on the same.

Answer 1

Mike Apologies for the inconvenience caused.

I have checked further and confirm that this issue is resolved. The doc has also been updated with more details.

Structured outputs is now supported with Global Batch when used in conjunction with API version 2024-08-01-preview or later. Use 2024-10-01-preview for the latest features.

I would request you kindly check now and confirm if you are still seeing the issue.

If the response helped, please do click Accept Answer and Yes for was this answer helpful.

Doing so would help other community members with similar issue identify the solution. I highly appreciate your contribution to the community.

Share via

Azure OpenAI Batch Processing Doesn't Work With Structured Response

1 answer

Your answer