Azure Open Ai Insert own data get error "Cracking and chunking - Data ingestion failed"

Question

Azure Open Ai Insert own data get error "Cracking and chunking - Data ingestion failed"

Kelvin Shee 95

hi Expert,

I am using Azure AI Foundry and have created a project and hub. The model I am using is GPT-4o along with text-embedding-ada-002 on the Chat Playground. I added my data from a storage account using Azure AI Services and want it to detect my storage files and respond accordingly.

Below is the error I encountered when adding data to Azure OpenAI. error 1

crack_chunk_embed_log.txt

may i know how i can fix this issue?

thanks.

Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-02-27T10:55:42.6533333+00:00
Hi Kelvin Shee

It looks like you're encountering a UnicodeDecodeError while trying to process data in Azure OpenAI.

Here are some steps:

Verify that the data you are attempting to decode is encoded in gb2312. If the data includes characters beyond the gb2312 encoding, consider using an alternative encoding like GB18030, which encompasses gb2312 and supports additional characters.

If you think the data may include characters not supported by the gb2312 encoding, consider switching to GB18030

with open('yourfile.txt', 'r', encoding='GB18030') as file: content = file.read()

Azure OpenAI On Your Data supports the following file types:

.txt

.md

.html

.docx

.pptx

.pdf ,

Kindly refer below Link: https://github.com/microsoft/sample-app-aoai-chatGPT/tree/main/scripts#optional-crack-pdfs-to-text For preprocessing longer text or mixed datatype.

supported-data-sources

Thank You.
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-02-28T06:01:01.3233333+00:00

Hi Kelvin Shee

Following up to see if the above response was helpful.

Thank Yu.
Kelvin Shee 95 Reputation points

2025-03-03T02:33:04.0566667+00:00

hi @saideep anchuri

as the storage all are the .html , so is that i need decode is encoded in gb2312?
Kelvin Shee 95 Reputation points

2025-03-03T02:34:27.5466667+00:00

hi Saideep Anchuri

as the files all are the .html files , so is that i need verify that the data you are attempting to decode is encoded in gb2312?
Kelvin Shee 95 Reputation points

2025-03-03T03:24:52.86+00:00

i check on my html files the type are MacRoman

Saideep Anchuri 9,425 Microsoft External Staff Moderator

Hi Kelvin Shee

Issue is coming from few html files because of un-supported encoding. Please re-save all the file into uniform encoding. Attached sample script to save them into UTF-8 file.

import chardet
import os
 
def convert_to_utf8(file_path):
    # Detect the file's current encoding
    with open(file_path, 'rb') as file:
        raw_data = file.read()
        result = chardet.detect(raw_data)
        encoding = result['encoding']
        print(f"Detected encoding for {file_path}: {encoding}")
 
    # Read the file with the detected encoding and write it as UTF-8
    with open(file_path, 'r', encoding=encoding, errors='ignore') as file:
        file_text = file.read()
 
    # Write the content back to the file in UTF-8 encoding
    with open(file_path, 'w', encoding='utf-8') as file:
        file.write(file_text)
 
def convert_files_in_directory(directory):
    for root, dirs, files in os.walk(directory):
        for file in files:
            file_path = os.path.join(root, file)
            convert_to_utf8(file_path)
 
# Specify the directory containing your files
directory_path = '/path/to/your/files'
convert_files_in_directory(directory_path)

Thank You.

Kelvin Shee 95 Reputation points

2025-03-03T05:59:50.93+00:00

hi Saideep Anchuri

now i hit another error on step 2

here is the logs

error step2.txt

could help on this error issue ?

thanks.
Kelvin Shee 95 Reputation points

2025-03-03T06:18:08.8833333+00:00

may i know is have any solution to solve. thanks.
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-03-03T07:25:03.44+00:00

Hi Kelvin Shee

Thank you for the update. azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Forbidden' - Getting a new failed ingestion error now with chat playground. - indicates that your index creation or update has failed because of insufficient permission over AI search. Please assign the AI service as AI search contributor over your AI search resource and make sure you are already using Azure AI developer role on the AI resource. If it is managed identity, please update the roles accordingly.

Thank You.
Kelvin Shee 95 Reputation points

2025-03-04T01:56:56.4266667+00:00

hi Saideep Anchuri

i do have below two role permission ,

Contributor & Azure AI Inference Deployment Operator role

is that still need add AI search contributor permission?
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-03-04T05:58:29.8133333+00:00

Hi Kelvin Shee

There are multiple scenarios mentioned here in which It requires granual access on AI search if we are trying to index on AI search. Please add Search Service Contributor for the AI studio hub. If it is secured workspace scenario, please make sure that you have gone through instruction from Azure managed virtual network documentation

Thank You.
Kelvin Shee 95 Reputation points

2025-03-05T05:42:49.05+00:00

hi Saideep Anchuri

after added the AI Search Contributor , it keep loading and already ran for 5hrs and the storage container it just only 4 files (pdf and html)

even i try Remove Data Source and refresh the page ( it still show loading ) ?
Kelvin Shee 95 Reputation points

2025-03-05T06:41:51.2+00:00

hi Saideep Anchuri
My current role in the resource group is 'Owner,' which is inherited for all services under the resource group. In the Azure AI Services, I also have 'Owner,' 'Azure AI Administrator,' and 'Cognitive Services OpenAI Contributor' permissions. However, I am still facing the same error.

Here is the list of available roles, but I did not see the 'Search Service Contributor' role in the list."

thanks.
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-05T08:31:35.4066667+00:00

Hi @Kelvin Shee ,

Actually there some permissions you need to give for AzureOpenai and Search service managed identity.

Check this documentation, here Assignee means the managed identity of that particular resource where you need to give Role on mentioned Resource.

Please verify all of them whether you given or not, along with that please add details if any services network configuration is disabled or selected for particular Vnet.

Do let me know if you any further query.

Thank you
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-05T12:20:07.8733333+00:00

Hi @Kelvin Shee ,

First make sure you have assigned Search Service Contributor and Search Index Data Reader to AI hub managed identity/ OpenAI managed identity. You should assign in search service there it is visible.

It seems like you disabled the network access in your search services.

Either make it public and try.

OR

Under firewall and virtual networking tab, choose "Selected networks", and then select Allow Azure services on the trusted services list to access this storage account and click Save.

Use Azure OpenAi resource with system managed identity as authentication type.

After trying above if you any query let me know.

Thank you
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-06T03:42:45.0566667+00:00

Hi @Kelvin Shee ,

Did you get a chance to try above steps? do let me know if you have any query.

Thank you
Kelvin Shee 95 Reputation points

2025-03-06T05:57:12.6166667+00:00

hi JAYA SHANKAR G S and Saideep Anchuri

as below the pic , my hub are facing public

This loading it take me 4hrs and still keep loading?
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-06T06:11:06.4666667+00:00

Hi @Kelvin Shee ,

Check network settings in search service and regarding your loading issue refresh the page and check in Data and indexes tab where you can see recently created indexes.

Thank you
Kelvin Shee 95 Reputation points

2025-03-06T06:42:43.3733333+00:00

hi

on the indexes it do have the documents. so is that i can texting on the chat playground that asking base on the documents index?

i just try and it not able to get the content?
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-06T06:53:20.7633333+00:00

@Kelvin Shee

Please give below details.

Here, what is the network configuration?

and

Here, you see the status if it's continuously loading.

It's failing because the network configuration is disabled in search service.
Kelvin Shee 95 Reputation points

2025-03-07T09:19:00.7766667+00:00
hi JAYA SHANKAR G S

now look good on the search services

I have an another question

how I can build to web base with the data index ?

during I setup the Data Index I do setup schedule updates by One time indexing , how I can change to Hourly ? should I re-create again or can i modify the data index that is ready ?

thanks.

Accepted answer

0 additional answers

Your answer

Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-02-27T10:55:42.6533333+00:00

Hi Kelvin Shee

It looks like you're encountering a UnicodeDecodeError while trying to process data in Azure OpenAI.

Here are some steps:

Verify that the data you are attempting to decode is encoded in gb2312. If the data includes characters beyond the gb2312 encoding, consider using an alternative encoding like GB18030, which encompasses gb2312 and supports additional characters.

If you think the data may include characters not supported by the gb2312 encoding, consider switching to GB18030

with open('yourfile.txt', 'r', encoding='GB18030') as file: content = file.read()

Azure OpenAI On Your Data supports the following file types:

.txt

.md

.html

.docx

.pptx

.pdf ,

Kindly refer below Link: https://github.com/microsoft/sample-app-aoai-chatGPT/tree/main/scripts#optional-crack-pdfs-to-text For preprocessing longer text or mixed datatype.

supported-data-sources

Thank You.
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-02-28T06:01:01.3233333+00:00

Hi Kelvin Shee

Following up to see if the above response was helpful.

Thank Yu.
Kelvin Shee 95 Reputation points

2025-03-03T02:33:04.0566667+00:00

hi @saideep anchuri

as the storage all are the .html , so is that i need decode is encoded in gb2312?
Kelvin Shee 95 Reputation points

2025-03-03T02:34:27.5466667+00:00

hi Saideep Anchuri

as the files all are the .html files , so is that i need verify that the data you are attempting to decode is encoded in gb2312?
Kelvin Shee 95 Reputation points

2025-03-03T03:24:52.86+00:00

i check on my html files the type are MacRoman
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-03-03T04:14:51.6666667+00:00

Hi Kelvin Shee

Issue is coming from few html files because of un-supported encoding. Please re-save all the file into uniform encoding. Attached sample script to save them into UTF-8 file.

import chardet import os def convert_to_utf8(file_path): # Detect the file's current encoding with open(file_path, 'rb') as file: raw_data = file.read() result = chardet.detect(raw_data) encoding = result['encoding'] print(f"Detected encoding for {file_path}: {encoding}") # Read the file with the detected encoding and write it as UTF-8 with open(file_path, 'r', encoding=encoding, errors='ignore') as file: file_text = file.read() # Write the content back to the file in UTF-8 encoding with open(file_path, 'w', encoding='utf-8') as file: file.write(file_text) def convert_files_in_directory(directory): for root, dirs, files in os.walk(directory): for file in files: file_path = os.path.join(root, file) convert_to_utf8(file_path) # Specify the directory containing your files directory_path = '/path/to/your/files' convert_files_in_directory(directory_path)

Thank You.
Kelvin Shee 95 Reputation points

2025-03-03T05:59:50.93+00:00

hi Saideep Anchuri

now i hit another error on step 2

here is the logs

error step2.txt

could help on this error issue ?

thanks.
Kelvin Shee 95 Reputation points

2025-03-03T06:18:08.8833333+00:00

may i know is have any solution to solve. thanks.
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-03-03T07:25:03.44+00:00

Hi Kelvin Shee

Thank you for the update. azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Forbidden' - Getting a new failed ingestion error now with chat playground. - indicates that your index creation or update has failed because of insufficient permission over AI search. Please assign the AI service as AI search contributor over your AI search resource and make sure you are already using Azure AI developer role on the AI resource. If it is managed identity, please update the roles accordingly.

Thank You.
Kelvin Shee 95 Reputation points

2025-03-04T01:56:56.4266667+00:00

hi Saideep Anchuri

i do have below two role permission ,

Contributor & Azure AI Inference Deployment Operator role

is that still need add AI search contributor permission?
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator

2025-03-04T05:58:29.8133333+00:00

Hi Kelvin Shee

There are multiple scenarios mentioned here in which It requires granual access on AI search if we are trying to index on AI search. Please add Search Service Contributor for the AI studio hub. If it is secured workspace scenario, please make sure that you have gone through instruction from Azure managed virtual network documentation

Thank You.
Kelvin Shee 95 Reputation points

2025-03-05T05:42:49.05+00:00

hi Saideep Anchuri

after added the AI Search Contributor , it keep loading and already ran for 5hrs and the storage container it just only 4 files (pdf and html)

even i try Remove Data Source and refresh the page ( it still show loading ) ?
Kelvin Shee 95 Reputation points

2025-03-05T06:41:51.2+00:00

hi Saideep Anchuri
My current role in the resource group is 'Owner,' which is inherited for all services under the resource group. In the Azure AI Services, I also have 'Owner,' 'Azure AI Administrator,' and 'Cognitive Services OpenAI Contributor' permissions. However, I am still facing the same error.

Here is the list of available roles, but I did not see the 'Search Service Contributor' role in the list."

thanks.
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-05T08:31:35.4066667+00:00

Hi @Kelvin Shee ,

Actually there some permissions you need to give for AzureOpenai and Search service managed identity.

Check this documentation, here Assignee means the managed identity of that particular resource where you need to give Role on mentioned Resource.

Please verify all of them whether you given or not, along with that please add details if any services network configuration is disabled or selected for particular Vnet.

Do let me know if you any further query.

Thank you
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-05T12:20:07.8733333+00:00

Hi @Kelvin Shee ,

First make sure you have assigned Search Service Contributor and Search Index Data Reader to AI hub managed identity/ OpenAI managed identity. You should assign in search service there it is visible.

It seems like you disabled the network access in your search services.

Either make it public and try.

OR

Under firewall and virtual networking tab, choose "Selected networks", and then select Allow Azure services on the trusted services list to access this storage account and click Save.

Use Azure OpenAi resource with system managed identity as authentication type.

After trying above if you any query let me know.

Thank you
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-06T03:42:45.0566667+00:00

Hi @Kelvin Shee ,

Did you get a chance to try above steps? do let me know if you have any query.

Thank you
Kelvin Shee 95 Reputation points

2025-03-06T05:57:12.6166667+00:00

hi JAYA SHANKAR G S and Saideep Anchuri

as below the pic , my hub are facing public

This loading it take me 4hrs and still keep loading?
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-06T06:11:06.4666667+00:00

Hi @Kelvin Shee ,

Check network settings in search service and regarding your loading issue refresh the page and check in Data and indexes tab where you can see recently created indexes.

Thank you
Kelvin Shee 95 Reputation points

2025-03-06T06:42:43.3733333+00:00

hi

on the indexes it do have the documents. so is that i can texting on the chat playground that asking base on the documents index?

i just try and it not able to get the content?
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-06T06:53:20.7633333+00:00

@Kelvin Shee

Please give below details.

Here, what is the network configuration?

and

Here, you see the status if it's continuously loading.

It's failing because the network configuration is disabled in search service.
Kelvin Shee 95 Reputation points

2025-03-07T09:19:00.7766667+00:00

hi JAYA SHANKAR G S

now look good on the search services

I have an another question

how I can build to web base with the data index ?

during I setup the Data Index I do setup schedule updates by One time indexing , how I can change to Hourly ? should I re-create again or can i modify the data index that is ready ?

thanks.

Answer 1

JAYA SHANKAR G S 4,035 Microsoft External Staff Moderator

Hi @Kelvin Shee ,

I believe by making search service public you were able to add data to chat playground

and to your further questions.

1. how I can build to web base with the data index ? Do you mean Web app? If yes you can directly deploy it like shown below.

If you want to make any changes like data source, skillset etc to ai search you do directly in portal or sdk and no need to create the index unnecessarily every time you add to chat playground.

enter image description here

And refer this documentation for more information.

during I setup the Data Index I do setup schedule updates by One time indexing , how I can change to Hourly ? should I re-create again or can i modify the data index that is ready ? You no need to create new index, use the current index with schedule(Hourly).

UPDATE

Create a managed identity and assign appropriate roles should resolve your issue. Refer this documentation to assign roles.

Next, you provide the query to chat playground which more specific to your documents added, then you will not get information not available response from model.

If you have any further query let me know, please do accept the answer and give feedback by clicking on Yes.

Thank you

Kelvin Shee 95 Reputation points

2025-03-10T03:13:26.48+00:00

hi JAYA SHANKAR G S

i already completed build the web app , but when I try search the data index content and it hit error.
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-10T03:44:02.27+00:00

Hi @Kelvin Shee ,

Seems it's similar kind of issue mentioned in this case .

Please check the above case and let me know if you are still facing the same issue.

Thank you
Kelvin Shee 95 Reputation points

2025-03-10T05:55:29.39+00:00

hi JAYA SHANKAR G S

on azure ai services , i already enable System assigned and using Microsoft Managed Keys.

but unlucky that is have error as below.
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-10T05:58:59.0133333+00:00

Hi @Kelvin Shee ,

Please share the details asked in private chat so that we can connect offline, also have you checked the steps provided by Will in the above given case? please check it.

Thank you
Kelvin Shee 95 Reputation points

2025-03-10T07:05:30.0833333+00:00

hi JAYA SHANKAR G S

how i can send the details in private chat ?
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-10T07:15:02.79+00:00

@Kelvin Shee Under your question you can see Private messages option, click on it to chat.
JAYA SHANKAR G S 4,035 Reputation points Microsoft External Staff Moderator

2025-03-11T04:35:51.5666667+00:00

Hi @Kelvin Shee ,

I have added an update please check it and let me know if you have any query.

If that helps please do accept the solution and give feedback clicking yes.

Thank you

Share via

Azure Open Ai Insert own data get error "Cracking and chunking - Data ingestion failed"

0 additional answers

Your answer