Azure Data Factory Copy Data to CSV not accurate

Question

The pipeline was, I created a link servicer with ServiceNow Rest API, created a dataset, and used Copy Data activity; those data sink to the storage as a CSV file.

The Rest API call was "approved and processed" data. when I checked with the postman, the data count was equal to the ServiceNow Application, but when I processed through the Data factory, the count was less.

This was worked previously and now the data is not accurate

Eg :

1. I have changed the API to mention specific engineer data, then the data is accurate in the data factory, but if I take the whole data for all engineers, then the data is not accurate.

This was a weird situation and looking for support.

Answer 1

Hello @Gayan Udayakantha !

Check the query used to retrieve the data: Ensure that the same query is used in Postman and Azure Data Factory to retrieve the data. If there are any differences in the query, it could result in differences in the number of records retrieved.

Check the authentication method used: Ensure that the same authentication method is used in Postman and Azure Data Factory. If there are any differences in the authentication method, it could result in differences in the number of records retrieved.

Check for any filters or limitations: Check if there are any filters or limitations applied to the data retrieval in either Postman or Azure Data Factory. If there are any differences in the filters or limitations, it could result in differences in the number of records retrieved.

Check for any throttling or rate limiting: Check if there are any throttling or rate limiting settings applied to the ServiceNow Rest API. If there are any differences in the throttling or rate limiting settings, it could result in differences in the number of records retrieved.

Check the data processing in Azure Data Factory: Check if there are any data transformations or processing steps applied to the data in Azure Data Factory. If there are any differences in the data processing, it could result in differences in the number of records retrieved.

Check for any errors or warnings: Check if there are any errors or warnings logged in Azure Data Factory. If there are any errors or warnings, it could indicate issues with the data retrieval or processing.

Kindly mark this answer as Accepted in case it helped or post your feedback !

Regards