question

PhilipMabon-0247 avatar image
0 Votes"
PhilipMabon-0247 asked MartinJaffer-MSFT answered

Self hosted Integration run time performance for copy activity on premise files

We want to use a copy activity where are source is multiple large file on premise (10GB+) to sink destination that is on the same on premise file system but a different location.

Is the Self hosted integration run time smart enough to avoid transferring file to the ADF and then back down?
Want to avoid Ingress cost and have great transfer speed.







azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MartinJaffer-MSFT avatar image
1 Vote"
MartinJaffer-MSFT answered

Hello @PhilipMabon-0247 and welcome to Microsoft Q&A.

I think the SHIR should be smart enough to avoid the transfer, given you select it for that dataset pair. To be sure I propose the following test. Tell me whether you think this test is good enough.

I will install SHIR on my computer, and create 2 datasets for different locations on my computer file system. I make a file of a significant, known size.
I will schedule a run, then turn off all my other applications and watch the traffic via the Task Manager or similar tool.
I expect a small bit of traffic to fetch instructions for the task. The traffic should be much less than the file size.

If the traffic is less than file size, and copy succeeds, then SHIR is smart enough not to upload / download.
If traffic is close to file size, and copy succeeds, then SHIR is not smart enough.

Sound good?

Update: I have confirmed the outbound network b/s is less than the disk write b/s to the best of my ability.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.