How to search 16MB or more of data in cognitive search

Question

How to search 16MB or more of data in cognitive search

Maruyama, Sota/丸山颯太 0

Cognitive Search has a maximum document size of 16MB.

Using the "Microsoft.Skills.Text.SplitSkill" skill set,

Is it possible to search over 16MB of data?

VenkateshDodda-MSFT 24,591 Reputation points Microsoft Employee

2023-08-31T05:21:21.9933333+00:00

@Maruyama, Sota/丸山颯太 Thanks for reaching out to Microsoft Q&A, apologize for any inconvenience caused on this.

Can please confirm based on my understanding that you want to search more than 16MB of data per single request? If yes, then as mentioned in the documentation here.

In Azure Cognitive Search, the body of a request is subject to an upper limit of 16 MB, imposing a practical limit on the contents of individual fields or collections that aren't otherwise constrained by theoretical limits (see Supported data types for more information about field composition and restrictions).

Limits on query size and composition exist because unbounded queries can destabilize your search service. Typically, such queries are created programmatically. If your application generates search queries programmatically, we recommend designing it in such a way that it doesn't generate queries of unbounded size.
Maruyama, Sota/丸山颯太 0 Reputation points

2023-08-31T06:11:42.51+00:00

I understand that since the API request limit is 16MB, 20MB cannot be sent in a single request. Can I use the text split cognitive skill to split the data and send it as two API requests to process the data within the limit?
Maruyama, Sota/丸山颯太 0 Reputation points

2023-08-31T06:12:38.53+00:00

I understand that since the API request limit is 16MB, 20MB cannot be sent in a single request. Can I use the text split cognitive skill to split the data and send it as two API requests to process the data within the limit?
VenkateshDodda-MSFT 24,591 Reputation points Microsoft Employee

2023-08-31T09:09:21.3233333+00:00

@Maruyama, Sota/丸山颯太 Thanks for your response. I am checking with internal team and will update you here once I hear back from them.
Maruyama, Sota/丸山颯太 0 Reputation points

2023-08-31T23:18:13.79+00:00

@VenkateshDodda-MSFT Thank you and best regards.
VenkateshDodda-MSFT 24,591 Reputation points Microsoft Employee

2023-09-06T04:22:38.78+00:00

Hello @Maruyama, Sota/丸山颯太 ,

I hope the answer helps. Please let me know if you have any further queries.

Please accept as Yes if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

1 answer

Your answer

VenkateshDodda-MSFT 24,591 Reputation points Microsoft Employee

2023-08-31T05:21:21.9933333+00:00

@Maruyama, Sota/丸山颯太 Thanks for reaching out to Microsoft Q&A, apologize for any inconvenience caused on this.

Can please confirm based on my understanding that you want to search more than 16MB of data per single request? If yes, then as mentioned in the documentation here.

In Azure Cognitive Search, the body of a request is subject to an upper limit of 16 MB, imposing a practical limit on the contents of individual fields or collections that aren't otherwise constrained by theoretical limits (see Supported data types for more information about field composition and restrictions).

Limits on query size and composition exist because unbounded queries can destabilize your search service. Typically, such queries are created programmatically. If your application generates search queries programmatically, we recommend designing it in such a way that it doesn't generate queries of unbounded size.
Maruyama, Sota/丸山颯太 0 Reputation points

2023-08-31T06:11:42.51+00:00

I understand that since the API request limit is 16MB, 20MB cannot be sent in a single request. Can I use the text split cognitive skill to split the data and send it as two API requests to process the data within the limit?
Maruyama, Sota/丸山颯太 0 Reputation points

2023-08-31T06:12:38.53+00:00

I understand that since the API request limit is 16MB, 20MB cannot be sent in a single request. Can I use the text split cognitive skill to split the data and send it as two API requests to process the data within the limit?
VenkateshDodda-MSFT 24,591 Reputation points Microsoft Employee

2023-08-31T09:09:21.3233333+00:00

@Maruyama, Sota/丸山颯太 Thanks for your response. I am checking with internal team and will update you here once I hear back from them.
Maruyama, Sota/丸山颯太 0 Reputation points

2023-08-31T23:18:13.79+00:00

@VenkateshDodda-MSFT Thank you and best regards.
VenkateshDodda-MSFT 24,591 Reputation points Microsoft Employee

2023-09-06T04:22:38.78+00:00

Hello @Maruyama, Sota/丸山颯太 ,

I hope the answer helps. Please let me know if you have any further queries.

Please accept as Yes if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

Answer 1

VenkateshDodda-MSFT 24,591 Microsoft Employee

@Maruyama, Sota/丸山颯太 Thanks for your patience on this.
I have checked with Azure cognitive search engineering team on this and sharing the below.
Regarding the 16MB limit, as you are trying to use the indexer, the limit is related to the search service SKU

S1 and up support the largest size of doc and also if you want to use smaller SKUs such as basic or free then you need split the docs before ingestion.

Maruyama, Sota/丸山颯太 0 Reputation points

2023-09-06T04:28:22.2266667+00:00

@VenkateshDodda-MSFT
Sorry for the late reply.

When I tried it on S1 earlier, I was able to process more than 16MB of data.

Thank you!
VenkateshDodda-MSFT 24,591 Reputation points Microsoft Employee

2023-09-06T04:54:47.9266667+00:00

Maruyama, Sota/丸山颯太 Thanks for your response. I have converted my comment to answer.

Please accept as Yes if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

Share via

How to search 16MB or more of data in cognitive search

1 answer

Your answer