Data load to ADX (Kusto) fails with error - UserErrorKustoWriteFailed

Vaibhav_Chaudhari 21 Reputation points
2020-12-09T08:26:11.817+00:00

We are loading data from OData to Kusto table. In OData source, a query is given to pull specific columns data to Kusto.

If the query has $top=10000 clause, ADF loads the data to Kusto table but when it is removed or 10001 or more is given, the pipeline fails.

I don't think there is any limit as such to load data to Kusto. Please help.

Error message -

Failure happened on 'Source' side. ErrorCode=UserErrorKustoWriteFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write to Kusto failed with following error: An error occurred for source: 'DataReader'. Error: 'One or more errors occurred.'.,Source=Microsoft.DataTransfer.Runtime.KustoConnector,''Type=Kusto.Ingest.Exceptions.IngestClientException,Message=An error occurred for source: 'DataReader'. Error: 'One or more errors occurred.',Source=Kusto.Ingest,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Requesting response from path WorkItems and query https://xxx.dev.azure.com/xx/_odata/Tble?$select=col1,col2,col3&$top=11000 failed with status code NotFound and message .,Source=Microsoft.DataTransfer.Runtime.ODataConnector,'
Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
495 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,898 questions
{count} votes

Accepted answer
  1. Vaibhav Chaudhari 38,671 Reputation points
    2020-12-18T10:24:20.17+00:00

    The Source Azure DevOps OData table has 500+ columns and I was trying to pull specific 150 columns by using OData query option. This used to fail after loading 10K rows. looks like there is some limitation as mentioned in below doc

    https://learn.microsoft.com/en-us/azure/devops/report/extend-analytics/wit-analytics?view=azure-devops#enforce-server-side-paging

    Workaround

    Instead of query I selected table option and in the mapping tab of copy activity, I mapped only required column

    Another workaround was to use query without select columns clause but just the main URL and filter clause if needed..

    0 comments No comments

0 additional answers

Sort by: Most helpful