Data load to ADX (Kusto) fails with error - UserErrorKustoWriteFailed

Question

Data load to ADX (Kusto) fails with error - UserErrorKustoWriteFailed

We are loading data from OData to Kusto table. In OData source, a query is given to pull specific columns data to Kusto.

If the query has $top=10000 clause, ADF loads the data to Kusto table but when it is removed or 10001 or more is given, the pipeline fails.

I don't think there is any limit as such to load data to Kusto. Please help.

Error message -

Failure happened on 'Source' side. ErrorCode=UserErrorKustoWriteFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write to Kusto failed with following error: An error occurred for source: 'DataReader'. Error: 'One or more errors occurred.'.,Source=Microsoft.DataTransfer.Runtime.KustoConnector,''Type=Kusto.Ingest.Exceptions.IngestClientException,Message=An error occurred for source: 'DataReader'. Error: 'One or more errors occurred.',Source=Kusto.Ingest,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Requesting response from path WorkItems and query https://xxx.dev.azure.com/xx/_odata/Tble?$select=col1,col2,col3&$top=11000 failed with status code NotFound and message .,Source=Microsoft.DataTransfer.Runtime.ODataConnector,'

HimanshuSinha-msft 18,531 Reputation points Microsoft Employee

2020-12-10T23:01:03.073+00:00

Hello @Vaibhav Chaudhari ,
Good to see you asking a question rather then answering . I must thank you for all your involvement on Q&A community ,

Going back to the ask . The break point here is 10000 thats what i understood . I have a hypothesis , as per the doc [here][1] , there is a limitation of 1 GB and that may be something which is kicking in . Can I request you to please query only one col and see what is breaking point ? Basically update the OData query from $select=col1,col2,col3&$top=11000
to $select=col1&$top=11000 .

Let me kow how it goes .

Thanks
Himanshu
HimanshuSinha-msft 18,531 Reputation points Microsoft Employee

2020-12-17T22:48:28.807+00:00

Hello ,
We have not heard back from you on this and was just following up .
Incase if you have resolution , request you to share the same here , so that other community members can benefit from that .
Thanks
Himanshu

HimanshuSinha-msft 18,531 Reputation points Microsoft Employee

2020-12-10T23:01:03.073+00:00

Hello @Vaibhav Chaudhari ,
Good to see you asking a question rather then answering . I must thank you for all your involvement on Q&A community ,

Going back to the ask . The break point here is 10000 thats what i understood . I have a hypothesis , as per the doc [here][1] , there is a limitation of 1 GB and that may be something which is kicking in . Can I request you to please query only one col and see what is breaking point ? Basically update the OData query from $select=col1,col2,col3&$top=11000
to $select=col1&$top=11000 .

Let me kow how it goes .

Thanks
Himanshu
HimanshuSinha-msft 18,531 Reputation points Microsoft Employee

2020-12-17T22:48:28.807+00:00

Hello ,
We have not heard back from you on this and was just following up .
Incase if you have resolution , request you to share the same here , so that other community members can benefit from that .
Thanks
Himanshu

Answer 1

The Source Azure DevOps OData table has 500+ columns and I was trying to pull specific 150 columns by using OData query option. This used to fail after loading 10K rows. looks like there is some limitation as mentioned in below doc

https://learn.microsoft.com/en-us/azure/devops/report/extend-analytics/wit-analytics?view=azure-devops#enforce-server-side-paging

Workaround

Instead of query I selected table option and in the mapping tab of copy activity, I mapped only required column

Another workaround was to use query without select columns clause but just the main URL and filter clause if needed..

Data load to ADX (Kusto) fails with error - UserErrorKustoWriteFailed

0 additional answers

Activity