Is there a disk speed cap after 100 GB of copied data?

Question

Is there a disk speed cap after 100 GB of copied data?

Peter Björklund 0

Is there some kind of speed cap when I copy data in Azure, if so how do I extend it? If there is no speed cap, why is it acting like this?

2023-03-29_161331

The source server have a 12 000 GiB Standard HDD LRS with 2000 Max IOPS and 500 Mbps thoughput. The source server isn't doing anything else at the moment than serving the file to the destination server.

Recource monitor of source server

2023-03-29_161000

The destination server have a 8192 GiB Premium SSD LRS with 16000 IOPS and 500 Mbps max throughput. The destination server is a development database server but is not under heavy load-

Recource monitor of destination server

2023-03-29_161429

With best regards, Peter Björklund

Gary Nebbett 6,206 Reputation points

2023-03-31T08:27:47.6733333+00:00

Hello Peter,

One thing that I have noticed when creating large files is the performance impact of continually "extending" the target file; if the final size of the target could be predicted, it was much better to create the file and set its size at the start of the operation and then write the data.

We don't know which copy command you are using, so we can't check how that program behaves in this respect.

The first image in your message is almost "beautiful" - three almost straight lines joined by nicely rounded curves and definitely does suggest that some external "intelligence" (algorithm) is guiding it.

My initial idea would be to make four short traces (perhaps one second duration) with "Windows Performance Recorder" (WPR): one trace during the initial fast phase on both the source and target systems and one trace during the slow phase (again on both systems).

I would be happy to help you analyse the captured data. The "out-of-the-box" WPR command to start a trace that I would suggest is wpr -start FileIO -start Network -filemode; this command could be improved by using a custom trace profile (.wprp file) to eliminate events from the trace that are not needed for this problem - let me know if you are interested in pursuing this course of action.

Gary

2 answers

Your answer

Gary Nebbett 6,206 Reputation points

2023-03-31T08:27:47.6733333+00:00

Hello Peter,

One thing that I have noticed when creating large files is the performance impact of continually "extending" the target file; if the final size of the target could be predicted, it was much better to create the file and set its size at the start of the operation and then write the data.

We don't know which copy command you are using, so we can't check how that program behaves in this respect.

The first image in your message is almost "beautiful" - three almost straight lines joined by nicely rounded curves and definitely does suggest that some external "intelligence" (algorithm) is guiding it.

My initial idea would be to make four short traces (perhaps one second duration) with "Windows Performance Recorder" (WPR): one trace during the initial fast phase on both the source and target systems and one trace during the slow phase (again on both systems).

I would be happy to help you analyse the captured data. The "out-of-the-box" WPR command to start a trace that I would suggest is wpr -start FileIO -start Network -filemode; this command could be improved by using a custom trace profile (.wprp file) to eliminate events from the trace that are not needed for this problem - let me know if you are interested in pursuing this course of action.

Gary

Answer 1

Limitless Technology 44,711

Hello

Thank you for your question and reaching out. I can understand you are having query\issues related to Azure slow copy speed.

You might see slow performance when you try to transfer files to the Azure Files service. If you don't have a specific minimum I/O size requirement, we recommend that you use 1 MiB as the I/O size for optimal performance.

Use the right copy method:

Use AzCopy for any transfer between two file shares.

Use Robocopy between file shares on an on-premises computer.

Reference :

https://learn.microsoft.com/en-us/azure/storage/files/files-troubleshoot-performance?tabs=windows

--If the reply is helpful, please Upvote and Accept as answer--

Peter Björklund 0 Reputation points

2023-03-30T14:01:33.3733333+00:00

How do I set the IO size and how would that change the fact that I have 100MB/s in transfer speed during the first 100GB of data then it drops to 20MB/s? I figure that if the IO size was the issue I would experience the speed drop long before 100GB is transfered.

Answer 2

Peter Björklund 0

How do I set the IO size and how would that change the fact that I have 100MB/s in transfer speed during the first 100GB of data then it drops to 20MB/s? I figure that if the IO size was the issue I would experience the speed drop long before 100GB is transfered.

Share via

Is there a disk speed cap after 100 GB of copied data?

2 answers

Your answer