How to Index Json files in azure cognitive search having more than 40 complex fields.

Question

I have 1GB of Json files which I am trying to index using Azure Cognitive Search. On the very last step while creating an indexer I am getting an error saying, "The request is invalid. Details : Invalid index: The index contains 54 complex collection fields. An index can have at most 40 complex collection fields."

Microsoft support also mentioned that this is a product limitation.

I have around 60 complex collection fields in my json files and all of them are required.

Please suggest me the workaround.

Answer 1

Apologies for all the inconvenience, frustration, and delayed response! I can see you have reached out to Microsoft Support and Stack Overflow forum. As mentioned by Bertrand Le Roy from our product engineering team, that indexes are limited to 40 complex collection fields. Adding it here:

“Support is correct that indexes are limited to 40 complex collection fields. It's usually possible however to work around this limitation by excluding what doesn't strictly need to be in the index, or by modifying how the index fields are built and aggregated from data in the document: the portal wizard that suggests an index structure is an automated process that tries its best by sampling data in your datasource, but it doesn't understand what's important and what's not. An additional manual step over its suggestions is often necessary and is an opportunity to optimize them to your specific requirements.
This doc article may also provide some useful suggestions: How to model complex data types in Azure Cognitive Search “

There is internal discussion on this, and we have informed our product engineering team on this feedback. Also the best way to share your feedback would be to create a User Voice feedback item and upvote it. The product group monitors this site for feedback. This is the best way to ensure you are heard and you may receive a response depending on how much they information they can currently share.