Azure OpenAI File Object has deprecated status and status_details fields but no suitable replacement

Question

We are currently using the Azure OpenAI APIs to upload files to Azure and posting a new batch using the newly uploaded file. There are cases in which on posting the batch, we'd be getting something like this error: The input file 'file-28009a6785e5439a8049db25ff33c462' is in 'pending' state. Only files with status 'Processed' are accepted for batch inferencing. Please check the file status and try again

This is happening because the file we just uploaded is still not ready and is still being processed. Therefore, prior to posting the batch, we're fetching the file by ID to check on its status. However, as shown in this documentation: https://platform.openai.com/docs/api-reference/files/object it seems the both status fields are currently deprecated and should not be used anymore. I cannot find any suitable alternative to relying on them and we're therefore stuck relying on deprecated fields to avoid errors.

What should we be doing in this case?

Thanks,
David

Answer 1

Hi David Captur

Thank you for following up.

Below are possible status messages of ingested files and batch job.

validating	The input file is being validated before the batch processing can begin.
validating	The input file is being validated before the batch processing can begin.
failed	The input file has failed the validation process.
in_progress	The input file was successfully validated and the batch is currently running.
finalizing	The batch has completed and the results are being prepared.
completed	The batch has been completed and the results are ready.
expired	The batch wasn't able to be completed within the 24-hour time window.
cancelling	The batch is being cancelled (This may take up to 10 minutes to go into effect.)
cancelled	the batch was cancelled.

You can track batch job progress with below commands.

import time
import datetime 
status = "validating"
while status not in ("completed", "failed", "canceled"):
    time.sleep(60)
    batch_response = client.batches.retrieve(batch_id)
    status = batch_response.status
    print(f"{datetime.datetime.now()} Batch Id: {batch_id},  Status: {status}")
if batch_response.status == "failed":
    for error in batch_response.errors.data:  
        print(f"Error code {error.code} Message {error.message}")

Output

2024-07-31 21:48:32.556488 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: validating
2024-07-31 21:49:39.221560 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: in_progress
2024-07-31 21:50:53.383138 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: in_progress
2024-07-31 21:52:07.274570 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: in_progress
2024-07-31 21:53:21.149501 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: finalizing
2024-07-31 21:54:34.572508 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: finalizing
2024-07-31 21:55:35.304713 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: finalizing
2024-07-31 21:56:36.531816 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: finalizing
2024-07-31 21:57:37.414105 Batch Id: batch_6caaf24d-54a5-46be-b1b7-518884fcbdde,  Status: completed

Hope it answers your query.

Reference : Batch job progress

Thank you.