Unexpected Ingestion error in Azure OpenAI Studio

Question

Unexpected Ingestion error in Azure OpenAI Studio

Nitin Prakash 0

In Azure OpenAI studio chat playground, I used to be able to add a datasource, which is my azure blob storage. This process would create an azure cognitive services index, which is used by the chatbot.

However, since the last few days, when I try to add a datasource in Azure OpenAI studio playground, I get the following error:

We couldn't connect your data
Indexing failed on openai-dev-index-two-indexer-chunk. Details: message: Unexpected ingestion error occurred. details: The request body is too large and exceeds the maximum permissible limit. RequestId:0f9b2df4-f003-0089-0602-d5705c000000 Time:2023-08-22T14:12:06.3664871Z ErrorCode:RequestBodyTooLarge maxlimit:131584 Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit. RequestId:0f9b2df4-f003-0089-0602-d5705c000000 Time:2023-08-22T14:12:06.3664871Z</Message><MaxLimit>131584</MaxLimit></Error>

User's image

I files in Azure blob storage are now much smaller than what they were before (i.e. when the indexing process used to work), however now this process does not work anymore.

Any idea why this is happening? What can be done to over come this issue? I am a little confused because this exact setup used to work until a few days ago. What has changed?

Thank you.

VenkateshDodda-MSFT 24,671 Reputation points Microsoft Employee

2023-08-23T10:35:59.5466667+00:00
@Nitin Prakash Thanks for reaching out to Microsoft Q&A, apologize for any inconvenience caused on this.

Based on the shared information, I have understood that you are using Azure Blob Storage as source in Add your Data (Preview) under chat playground in Azure Open AI studio and the index creation is failing with the above error RequestBodyTooLarge maxlimit.
Azure Blob storage indexer has a limit of importing maximum 16MB size of Blob in Free and Basic tiers of cognitive search and this limit varies based on the tier of your cognitive search instance is running on.

Here is the documentation, for more information about indexer limits in azure cognitive search.

If you want to index large data-sets to cognitive search you need to use either of these two methods

Pushing data into an Index using either of these components like Batching documents or Managing threads

Pilling in data from supported data source using a Search index.

For more information, go through this documentation about index large data sets in cognitive search and its implementation.

If you are using Upload files as data source then As mentioned in the documentation here, Max size of all files per upload (Azure OpenAI on your data)16MB.

If your importing blob size is less than 16MB and you are unable to import the data to chat playground in Azure Open AI I would suggest you help us with cognitive Search instance Name, Issue time stamp (UTC) and OpenAI service name to check and assist you further on this.

Feel free to reach back to me if you have any further questions on this.
Nitin Prakash 0 Reputation points

2023-08-23T14:12:36.78+00:00

@VenkateshDodda-MSFT

Thank you for your response. Here are my org's subscription details:

Cognitive Search service subscription ID = e1-**********************************-09ff

Cognitive Search instance Name = iiba-*************************************-v

Issue Time stamp = 2023-08-23T14:08:14.2421923Z

Open AI Service name = IIBA-*************************************-SEAST

Open AI Service sub id = e1-**********************************-09ff

We are on Standard pricing tier.

We have 459 files in total. The total size is under 7 MB, which is much smaller than the limit of 16 MB.

I tried the index creation process via the Upload File option in the Azure OpenAI Playground, and this works fine. I see an index being created. However, I want to use the Azure Blob Storage option. This used to work until a few days ago, so not sure what has changed.

I would appreciate if you could take a look based on the information I have furnished above. Let me know if you require anything more from me.
Nitin Prakash 0 Reputation points

2023-08-23T14:40:45.7133333+00:00

@VenkateshDodda-MSFT

Thanks for your response.

We have 459 files, which total to under 7 MB, which is much smaller than the limit of 16 mb.

I tried the Upload File option in Azure OpenAI playground with all 549 files, and this seems to work. I can see an index being created.

However, I would like to use the Azure Blob storage option instead, with the same number of files. The index creation process used to work until a few days ago, so I am not sure what changed.

I would appreciate it if you could look into this issue for me. Here are the details you require:

We are on Standard pricing tier.

Cognitive Service name = iiba-cs-useast-v2

Cognitive Service id = e1-**********************************-09ff

Azure OpenAI service name =IIBA-*************************************-SEAST

OpenAI service sub id =e1-**********************************-09ff

Time = 2023-08-22T14:12:06.3664871Z

Index creation request id = 0f9b2df4-f003-0089-0602-d5705c000000
VenkateshDodda-MSFT 24,671 Reputation points Microsoft Employee

2023-08-24T07:29:17.35+00:00

@Nitin Prakash Thanks for sharing requested information.

Since this is public forum, I would encourage not to share any PII details like Subscription ID etc., I will go ahead and mask those details in your earlier comments.

I have looked at the backend logs, for your cognitive service at the given timestamp (2023-08-22 14:12:07) I can couple of delete Indexers operations on the indexer openai-dev-index-two-indexer-chunk and it got succeeded.

Also, we can see that you have recreated same indexer on 23th August during the indexer creation when see the below when the cognitive search service is trying to check whether the storage account type is ADLS Gen2 or not.

This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint

Reference documentation, for the above error message.

If you want to understand more or investigate deeper, I suggest you raise a support ticket to check and assist you further on this.

Your answer

VenkateshDodda-MSFT 24,671 Reputation points Microsoft Employee

2023-08-23T10:35:59.5466667+00:00

@Nitin Prakash Thanks for reaching out to Microsoft Q&A, apologize for any inconvenience caused on this.

Based on the shared information, I have understood that you are using Azure Blob Storage as source in Add your Data (Preview) under chat playground in Azure Open AI studio and the index creation is failing with the above error RequestBodyTooLarge maxlimit.
Azure Blob storage indexer has a limit of importing maximum 16MB size of Blob in Free and Basic tiers of cognitive search and this limit varies based on the tier of your cognitive search instance is running on.

Here is the documentation, for more information about indexer limits in azure cognitive search.

If you want to index large data-sets to cognitive search you need to use either of these two methods

Pushing data into an Index using either of these components like Batching documents or Managing threads

Pilling in data from supported data source using a Search index.

For more information, go through this documentation about index large data sets in cognitive search and its implementation.

If you are using Upload files as data source then As mentioned in the documentation here, Max size of all files per upload (Azure OpenAI on your data)16MB.

If your importing blob size is less than 16MB and you are unable to import the data to chat playground in Azure Open AI I would suggest you help us with cognitive Search instance Name, Issue time stamp (UTC) and OpenAI service name to check and assist you further on this.

Feel free to reach back to me if you have any further questions on this.
Nitin Prakash 0 Reputation points

2023-08-23T14:12:36.78+00:00

@VenkateshDodda-MSFT

Thank you for your response. Here are my org's subscription details:

Cognitive Search service subscription ID = e1-**********************************-09ff

Cognitive Search instance Name = iiba-*************************************-v

Issue Time stamp = 2023-08-23T14:08:14.2421923Z

Open AI Service name = IIBA-*************************************-SEAST

Open AI Service sub id = e1-**********************************-09ff

We are on Standard pricing tier.

We have 459 files in total. The total size is under 7 MB, which is much smaller than the limit of 16 MB.

I tried the index creation process via the Upload File option in the Azure OpenAI Playground, and this works fine. I see an index being created. However, I want to use the Azure Blob Storage option. This used to work until a few days ago, so not sure what has changed.

I would appreciate if you could take a look based on the information I have furnished above. Let me know if you require anything more from me.
Nitin Prakash 0 Reputation points

2023-08-23T14:40:45.7133333+00:00

@VenkateshDodda-MSFT

Thanks for your response.

We have 459 files, which total to under 7 MB, which is much smaller than the limit of 16 mb.

I tried the Upload File option in Azure OpenAI playground with all 549 files, and this seems to work. I can see an index being created.

However, I would like to use the Azure Blob storage option instead, with the same number of files. The index creation process used to work until a few days ago, so I am not sure what changed.

I would appreciate it if you could look into this issue for me. Here are the details you require:

We are on Standard pricing tier.

Cognitive Service name = iiba-cs-useast-v2

Cognitive Service id = e1-**********************************-09ff

Azure OpenAI service name =IIBA-*************************************-SEAST

OpenAI service sub id =e1-**********************************-09ff

Time = 2023-08-22T14:12:06.3664871Z

Index creation request id = 0f9b2df4-f003-0089-0602-d5705c000000
VenkateshDodda-MSFT 24,671 Reputation points Microsoft Employee

2023-08-24T07:29:17.35+00:00

@Nitin Prakash Thanks for sharing requested information.

Since this is public forum, I would encourage not to share any PII details like Subscription ID etc., I will go ahead and mask those details in your earlier comments.

I have looked at the backend logs, for your cognitive service at the given timestamp (2023-08-22 14:12:07) I can couple of delete Indexers operations on the indexer openai-dev-index-two-indexer-chunk and it got succeeded.

Also, we can see that you have recreated same indexer on 23th August during the indexer creation when see the below when the cognitive search service is trying to check whether the storage account type is ADLS Gen2 or not.

This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint

Reference documentation, for the above error message.

If you want to understand more or investigate deeper, I suggest you raise a support ticket to check and assist you further on this.

Share via

Unexpected Ingestion error in Azure OpenAI Studio

Your answer