How to change the field structure of an index created by data import and vectorization using Azure AI Search to look like the original dataset.

Question

I am looking to use Azure AI Search to import and vectorize data. The dataset is specified as csv and I want to create fields in Azure AI Search as well as columns in the csv. Currently I have performed the data import and vectorization and an index has been created. However, the field structure is not like the csv, but rather [chunk_id, parent_id, chunk, title, vector]. Originally, we would like to create fields with the following structure. [product, department, category, question, answer]. Is it possible to achieve what I want to do? Also, what is the best way to achieve what I want to do? If anyone knows anything about my question, could you please let me know? Thanks.

Answer 1

@PA 松村優
Importing and vectorizing data via the portal UX currently does not accommodate delimited text scenarios such as CSV files. Users will have to construct their indexer and skillset manually, referring to the documentation- Search over CSV blobs - Azure AI Search | Microsoft Learn

You should follow the instructions in the documentation to update your indexer and adjust the skillset definition. Specifically, you need to modify the input for the SplitSkill to "/document/columnName," where "columnName" represents the desired column to be chunked and vectorized.

Additionally, you must update your index projections to include any additional fields you wish to incorporate into each document.

Let us know.

Answer 2

I'm trying to do the same thing. The UI's "Import and Vectorize" doesn't really work even though you can specify a "CSV" delimitedText parsing mode.

Has anyone got this to work properly? if so, can you post a sample json of the indexer and skillset for parsing a CSV document with this sample layout:

EmployeeID, FirstName, LastName, Biography

1, Ron, Hubert, "a long biography that can't fit on one page."