Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
Data Factory Time Discrepancy: Seeking Explanation
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 40%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
EmaLom
0
Reputation points
In Data Factory, I have a total transfer time of 23 minutes, but the sum of reading and writing time is much lower. My reading time is 4:48 and writing time is 7:12. Can someone help me understand this discrepancy ?
Azure Data Lake Storage
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator2024-06-26T17:10:22.29+00:00 Hello EmaLom,
Welcome to the Microsoft Q&A forum.
The total transfer time in Azure Data Factory includes not only the time spent reading and writing data, but also the time spent on other activities such as data transformation, data validation, and network latency.
In your case, the sum of reading and writing time is much lower than the total transfer time, which suggests that there may be other activities that are contributing to the overall transfer time.
To identify the discrepancy, you can try the following steps:
- Check the activity logs in Azure Data Factory to see if there are any warnings or errors that could be causing delays.
- Check the network latency between the source and destination systems to see if there are any issues with the network that could be causing delays.
- Check the data transformation activities in your pipeline to see if there are any complex transformations that could be causing delays.
- Check the performance of the source and destination systems to see if there are any bottlenecks that could be causing delays.
I hope this helps! Let me know if you have any further questions.
-
EmaLom 0 Reputation points
2024-06-26T20:25:51.97+00:00 @Bhargava-MSFT Could you explain better this point "Check the activity logs in Azure Data Factory to see if there are any warnings or errors that could be causing delays. ?"
-
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
2024-06-26T20:54:54.77+00:00 Please check out this document: https://learn.microsoft.com/en-us/azure/data-factory/monitor-data-factory#azure-activity-log
-
EmaLom 0 Reputation points
2024-06-26T20:58:24.3233333+00:00 @Bhargava-MSFT Please review our conversation and explain how your responses have been helpful to me.
-
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
2024-06-27T17:38:11.5866667+00:00 Sorry, I didn't get you. Please let me know if you have any questions on the document that I provided.
-
EmaLom 0 Reputation points
2024-06-29T06:13:46.0866667+00:00 I regret to inform you that the document in question does not appear to contain the information I need.
-
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
2024-07-01T17:43:10.0533333+00:00 Hello EmaLom,
Sorry, since I do not have access to your data, I can't troubleshoot the issue directly. The above-mentioned steps are general guidance.
If you need further assistance with live troubleshooting, please log a support case.
Sign in to comment
Sign in to answer