Application Insights REST API not returning full result set in Azure Data factory Copy Activity

Question

Hello There,

I have a copy activity which is having the source as Application insights REST API, I am trying to query for the last one month data which would be GB in volume, however I am getting around 136MB of data only when run the pipeline, in the Data Preview i am getting the below error.

Is there any limit in the Azure side, how can we remove that limitation

Please help me to get this resolved

error": {
"code": "PartialError",
"message": "There were some errors when processing your query.",
"details": [
{
"code": "EngineError",
"message": "Something went wrong processing your query on the server.",
"innererror": {
"code": "-2133196797",
"message": "Query result set has exceeded the internal data size limit 64000000 (E_QUERY_RESULT_SET_TOO_LARGE; see https://aka.ms/kustoquerylimits)",
"severity": 2,
"severityName": "Error"
}
}
]
}
}strong text

Answer 1

Hi @Muralikrishna Pirla ,

As per my analysis, this is a limitation from ADX. Result truncation is a limit set by default on the result set returned by the query. Kusto limits the number of records returned to the client to 500,000, and the overall data size for those records to 64 MB. When either of these limits is exceeded, the query fails with a "partial query failure". Because of which you are seeing the error message when you are trying to preview the data.

You can disable result truncation by using the notruncation request option. We recommend that some form of limitation is still put in place.

For example:

set notruncation;  
MyTable | take 1000000  

It's also possible to have more refined control over result truncation by setting the value of truncationmaxsize (maximum data size in bytes, defaults to 64 MB) and truncationmaxrecords (maximum number of records, defaults to 500,000). For example, the following query sets result truncation to happen at either 1,105 records or 1 MB, whichever is exceeded.

set truncationmaxsize=1048576;  
set truncationmaxrecords=1105;  
MyTable | where User=="UserId1"  

The following apply when using set statements, and/or when specifying flags in client request properties.

• If notruncation is set, and any of truncationmaxsize, truncationmaxrecords, or query_take_max_records are also set - notruncation is ignored.
• If truncationmaxsize, truncationmaxrecords and/or query_take_max_records are set multiple times - the lower value for each property applies.

For more info please refer to this document and let us know if you have further query: Limit on result set size (result truncation)

Hope this info helps.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.