I have a question about the assistant playground in Azure OpenAI.

Question

I'm using the Assistance playground in Azure OpenAI. I have added 20 JSON data files to one vector store and am testing by connecting the assistant and the vector store in the assistant playground.

The 20 JSON data files have a total of 34000 pieces of data, with 170 keys per JSON object. The reason for splitting them into 20 is that if the JSON files are large, the vector store will fail to parse them.

The instructions also tell the search to check the 20 JSON files in the vector store and only answer data that matches 100% of the criteria you provide.

With this background structure, the Assistant Playground will continue to return incorrect information if you ask, “Get the 10 most recent data by date of creation.” We've checked the dataset several times, and it's fine.

Also, even when it returns matching JSON data as an answer, certain values in the JSON data are returned differently than the values in the dataset.

Also, when you ask “total number of JSON data in vector storage”, it gives you a different answer than 34000.

Is this a limitation of the LLM? I'm curious about the underlying reason.

My guess is that it is not currently checking all the data in the vector store, and I am very curious about the reason for this.

Is there a workaround, and if so, please let me know, and if not, what is the underlying reason why it shouldn't be? Thank you.

Answer 1

@홍재희 Hello, I am a teammate of Yutong and will assist you further with this matter.

There are a few items that come to mind after reading your post that could be causing issues for you. Splitting your data into smaller JSON files was a good move to avoid parsing issues. However, there might still be challenges with how the data is indexed and retrieved. If the vector store isn't indexing all of the data correctly, it might not be able to retrieve the most recent data accurately. I also have concerns about the instruction to match 100% of the criteria might be too strict, leading to incomplete or incorrect results if the criteria are not perfectly met. You are correct in that the LLM might have limitations with understanding complex queries, especially if the queries get into sorting or aggregating data. Lastly, data retrieval could be only getting partial data or encountering inconsistent data values.

With those risks called out, I can also think of a few workarounds for you to try. Can you try to reindex your data stored? This would ensure that all data is correctly indexed. It would also give you another opportunity to observe the indexing logs and ensure there are no errors with the process.

Next, can you relax your query criteria? See if allowing partial matches that still meet the essential criteria.

Lastly, if the above does not help, you will need to implement debugging and logging to track how the queries are processed and where they might be failing. This will help you and us understand if the issue is with the query execution or the data retrieval..

We look forward to your reply.