Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,374 questions
Copy data for very small file takes way too long
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 93%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPF%3C/text%3E%3C/svg%3E)
Pegoraro, Francesco
40
Reputation points
I encountered a timeout problem related to a copy data activity from ADSL Gen 2 to ADSL gen 2 trying to copy a 0 byte csv file.
How is it possible that the activity takes 28 minutes to copy a 0 byte file?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga 4,250 Reputation points Microsoft Vendor2024-04-05T06:40:25.93+00:00 Hi @Francesco Pegoraro
Welcome to Microsoft Q&A platform and thanks for posting your question here.There could be several reasons why a copy activity takes longer than expected to copy a 0-byte file. One possible reason is that the copy activity is experiencing a bottleneck due to the Azure Data Factory service or the data store.
To troubleshoot this issue, here are some possible causes and resolutions that may help:
- Cause: The copy activity is being executed on a self-hosted integration runtime (SHIR) and the machine is not optimized for performance.
- Resolution: Use a dedicated machine to host the SHIR and start with default values for parallel copy settings. You can also scale up the machine or scale out to multiple machines to improve performance.
- Cause: The copy activity is being executed on an Azure integration runtime and the Data Integration Units (DIUs) or parallel copy settings are not optimized.
- Resolution: Start with default values for DIUs and parallel copy settings. You can also increase the number of DIUs or parallel copies to improve performance.
Cause: The copy activity is experiencing a transient issue such as a network issue, credential failure, or service outage.
- Resolution: Rerun the pipeline or activity as described in the Azure documentation on "Transient Issues". If the run is stuck for more than an hour, create a support case.
I checked for outages in Azure services, and there are currently no reported outages.
Reference
https://azure.status.microsoft/en-gb/status
https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage?tabs=data-factoryI hope this information helps you. Let me know if you have any further questions or concerns.
-
Harishga 4,250 Reputation points Microsoft Vendor
2024-04-08T02:40:08.18+00:00 Hi @Francesco Pegoraro
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help. -
Pegoraro, Francesco 40 Reputation points
2024-04-10T07:05:05.8233333+00:00 Hi @Harishga
unfortunately none of your suggestions helped me figure out the problem we had. The pipeline that usually takes 1 h to complete kept running for 4 days costing us a lot of money, is there any way we can ask for a refund from microsoft?
Francesco
-
Deleted
This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Sign in to comment