Data load to ADX (Kusto) fails with error - UserErrorKustoWriteFailed

Question

We are loading data from OData to Kusto table. In OData source, a query is given to pull specific columns data to Kusto.

If the query has $top=10000 clause, ADF loads the data to Kusto table but when it is removed or 10001 or more is given, the pipeline fails.

I don't think there is any limit as such to load data to Kusto. Please help.

Error message -

Failure happened on 'Source' side. ErrorCode=UserErrorKustoWriteFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write to Kusto failed with following error: An error occurred for source: 'DataReader'. Error: 'One or more errors occurred.'.,Source=Microsoft.DataTransfer.Runtime.KustoConnector,''Type=Kusto.Ingest.Exceptions.IngestClientException,Message=An error occurred for source: 'DataReader'. Error: 'One or more errors occurred.',Source=Kusto.Ingest,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Requesting response from path WorkItems and query https://xxx.dev.azure.com/xx/_odata/Tble?$select=col1,col2,col3&$top=11000 failed with status code NotFound and message .,Source=Microsoft.DataTransfer.Runtime.ODataConnector,'

Answer 1

The Source Azure DevOps OData table has 500+ columns and I was trying to pull specific 150 columns by using OData query option. This used to fail after loading 10K rows. looks like there is some limitation as mentioned in below doc

https://learn.microsoft.com/en-us/azure/devops/report/extend-analytics/wit-analytics?view=azure-devops#enforce-server-side-paging

Workaround

Instead of query I selected table option and in the mapping tab of copy activity, I mapped only required column

Another workaround was to use query without select columns clause but just the main URL and filter clause if needed..