Creating a vector index using an excel file in Azure ML Studio

Srivallabh Narayan 5

I want to use the the Open AI Chat playground to develop a Q&A BOT on my large excel file . For this I am looking to create a vector index using this excel file . The vector index creation job for Excel fails at the data chunking stage every time .

I have tried converting the excel into csv and then .txt and then creating a vector index . Although I am able to create a vector index this way , the results seem heavily unreliable. Any help on creating an Azure Cognitive Search Index using ML Studio would be great.

romungi-MSFT 42,791 Reputation points Microsoft Employee

2023-09-02T14:58:38.31+00:00

@Srivallabh Narayan Did you try to use the Azure OpenAI bring your own data feature to upload your file directly, this should use a search index and provide you an option to deploy it as a webapp.
Srivallabh Narayan 5 Reputation points

2023-09-05T14:22:04.1033333+00:00
@romungi-MSFT . Thank you for the prompt response. I have tried this . The bring your own data feature on AI studio does not let you upload an excel or a csv format file .

As a result I was trying to create an index on Azure ML studio prompt flow for the excel file . This seems to fail in the data chunking stage giving the following error:

Any help on this would be appreciated .

Thanks

ValueError: No chunked documents found in /mnt/azureml
Min Lei 30 Reputation points

2023-09-26T21:17:16.6866667+00:00

Hi romungi-MSFT,

I got the same issue. I was trying to use vector index prompt flow with excel file, and found that "Although I am able to create a vector index this way , the results seem heavily unreliable." About 50% of the QnA pairs in my Excel file cannot be retrieved - the LLMs response says they not in the content I provided.

In my case, I need to use both Excel and Word files for the same prompt flow. When I created the Vector Index in Prompt flow, I uploaded 1 excel and 1 docx file, it created 1 index with these 2 files. The content in the docx seems have no issue in retrieving, but the content in excel only got about 50% can be found by the retrieval in the index. I am guessing the Chunk Size 1024 maybe too big for Excel type of content that one row is for 1 pair of QnA which in my case is much smaller than 1024 token. Do you know whether their is a way we can edit the chunk size when creating the vector index in Prompt flow. I did not find anywhere I can make any changes.

What is your suggestion to solve this issue?

1 answer

Jay 86 Reputation points

2024-01-16T07:05:23.5366667+00:00

Hi @Srivallabh Narayan , Looks like there is a new feature in Azure ML called Prompt-Flow which is currently in preview that can help you to solve your problem as this can support the file formats like

Supported file types for creating a vector index job: .txt, .md, .html, .htm, .py, .pdf, .ppt, .pptx, .doc, .docx, .xls, .xlsx. Any other file types will be ignored during creation.

Check out here
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Creating a vector index using an excel file in Azure ML Studio

1 answer