Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
3,914 questions
Azure OpenAI Batch Jobs are getting stuck in Validating stage and some are failing due to enqueued token limit surpassed even though tokens are available
Seena Mary Augusty
0
Reputation points Microsoft Employee
There were so many Azure Open AI batch jobs stuck in validating stage for last couple of days, we had faced this issue starting from March 18, 2025. After 2 days of getting stuck on validating stage, on March 20, 2024 at 4.50 pm PDT it started to work then at 7.50 pm PDT on the same day it started to get stuck in validating stage. Even after 12 hours had passed, its still stuck in validating stage.
Now the issue is ongoing at March 21, 3.54 pm PDT.
Also the batches are submitted every hour and because of this issue, some of them are failing due to enqueued token limit surpassed even though we are dynamically calculating the input tokens before we submit the batch jobs and have not reached enqueued token quota limit. Kindly take a look at this issue as well.
Here are the details of my setup:
- Endpoint: https://live-signals-copilot.openai.azure.com/
- Pricing Tier: Standard S0
- Region: East US
- Resource Configuration:
- Name: Live-Signals-Copilot
- Subscription: Customer Understanding
- Subscription ID: 62e7b93a-1bbe-492b-b06a-3a93f3d269fd
- Resource Group: Clarity
Could you help us resolve this issue as soon as possible. Our production endpoints are reliant on these batch jobs.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 61%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
santoshkc 14,255 Reputation points Microsoft External Staff2025-03-24T10:11:45.28+00:00 It appears that your Azure OpenAI batch jobs are intermittently getting stuck in the Validating stage and, in some cases, failing due to the enqueued token limit surpassed error, despite tokens being available.
There have been recent reports of delays in batch job validation on Azure OpenAI, which could be related to backend processing. If the issue is intermittent, it might be linked to regional capacity constraints or service-side throttling, causing unexpected delays in job processing.
Regarding the enqueued token limit issue, even though your calculations ensure that token limits remain within the quota, Azure may enforce internal queuing policies that impact batch processing. The limit may not always reset immediately, particularly when multiple jobs are queued in a short period, leading to failures despite available tokens.
To mitigate the issue, I recommend reducing the batch job submission frequency if possible. Submitting batch jobs less frequently may help improve validation and reduce queuing conflicts.
I hope you understand! Thank you.
-
Seena Mary Augusty 0 Reputation points Microsoft Employee
2025-03-25T16:07:28.4866667+00:00 Thank you @santoshkc for the response, there are lot of batches stuck for more than 24 hours, shouldn't that be expired. Also we did increase our quota and limited the job submission to be less than quota and the jobs are still in validating stage, Could you kindly help us on how to proceed,
Thank you
-
Seena Mary Augusty 0 Reputation points Microsoft Employee
2025-03-27T01:24:05.83+00:00 Hi @santoshkc we observed that batches submitted with batch deployment of version gpt4o with model version 2024-08-06 was stuck in validating stage for days but when we tried submitting the same batch files with batch deployment of version gpt4o with model version 2024-11-20 started to work, do you know the reason for the same
-
santoshkc 14,255 Reputation points Microsoft External Staff
2025-03-27T14:04:49.6366667+00:00 Glad to hear that your issue was resolved with model version 2024-11-20. The reason could be higher batch quota availability for this version, along with potential optimizations and improvements over the 2024-08-06 version. Since it's a more recent release, it may also have better handling of batch processing and validation.
Thank you.
Sign in to comment
Sign in to answer