Share via

azcopy copy failure between storage accounts with >10MB blob size

David Sullivan 10 Reputation points
2024-10-16T20:29:51.65+00:00

Hi Team,

Any ideas how to resolve this?... I am experiencing a problem when trying to copy a blob from one storage account to another when the blob is ~10MB or more. A ~1MB blob works fine.

I consistently have success with ~1MB blob size.

I consistently have failure with ~10MB blob size.

Included a section of Failure log below.

Included a section of Success log below.

Both storage accounts are private endpoint.

Command I'm using:

azcopy copy "https://<source-server>.blob.core.windows.net/123456/456789/731f7b0c?sv=2024-11-04&se=2024-10-31T13%3A57%3A31Z&sr=b&sp=r&sig=KL6h3o%2BDOsEkYxMJApVnEl9mrZ6o395NuhVBDnoZUV8%3D" "https://<destination-server>.blob.core.windows.net/123456/456789/731f7b0c?sv=2024-11-04&se=2024-10-31T13%3A57%3A31Z&sr=b&sp=c&sig=vGcoZyIeyisYtMR7S2IcqjD%2B5SehZil%2BYfhBMoYUMuw%3D"

Failure log for ~10MB blob size...

2024/10/16 13:36:51 Final job part has been scheduled

2024/10/16 13:36:51 INFO: [P#0-T#0] Starting transfer: Source "https://<source-server>.blob.core.windows.net/123456/456789/731f7b0c" Destination "https://<destination-server>.blob.core.windows.net/123456/456789/731f7b0c". Specified chunk size 8388608

2024/10/16 13:36:53 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 2, E: 0, T: 2, GRs: 32

2024/10/16 13:36:53 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 43.2961

2024/10/16 13:36:55 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 2, E: 0, T: 2, GRs: 32

2024/10/16 13:36:55 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,

2024/10/16 13:36:57 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 2, E: 0, T: 2, GRs: 32

2024/10/16 13:36:57 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,

2024/10/16 13:36:59 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 2, E: 0, T: 2, GRs: 32

2024/10/16 13:36:59 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,

2024/10/16 13:37:00 ==> REQUEST/RESPONSE (Try=1/69.051752ms, OpTime=8.262589975s) -- RESPONSE STATUS CODE ERROR

PUT https://<destination-server>.blob.core.windows.net/123456/456789%2F731f7b0c?blockid=MDAwMDADRZyL3jVIVVvyV4vy%2F57nMDAwMDAwMDAwMDAwMDAx&comp=block&se=2024-10-31T13%3A35%3A42Z&sig=-REDACTED-&sp=c&sr=b&sv=2024-11-04

Accept: application/xml

Content-Length: 0

User-Agent: AzCopy/10.26.0 azsdk-go-azblob/v1.4.0 (go1.22.5; linux)

X-Ms-Client-Request-Id: e4ab9c09-b8da-42ee-5e11-4ea8b2858600

x-ms-copy-source: https://<source-server>.blob.core.windows.net/123456/456789%2F731f7b0c?se=2024-10-31T13%3A35%3A42Z&sig=-REDACTED-&sp=r&sr=b&sv=2024-11-04

x-ms-source-range: bytes=8388608-10829442

x-ms-version: 2023-08-03

--------------------------------------------------------------------------------

RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.

Content-Length: 279

Content-Type: application/xml

Date: Wed, 16 Oct 2024 13:37:00 GMT

Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

X-Ms-Client-Request-Id: e4ab9c09-b8da-42ee-5e11-4ea8b2858600

X-Ms-Error-Code: AuthorizationPermissionMismatch

X-Ms-Request-Id: 2beedd1e-d01e-005f-1bd0-1fe9cf000000

X-Ms-Version: 2023-08-03

Response Details: <Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission. </Message>

2024/10/16 13:37:00 ERR: [P#0-T#0] COPYFAILED: https://<source-server>.blob.core.windows.net/123456/456789/731f7b0c : 403 : 403 This request is not authorized to perform this operation using this permission.. When Staging block from URL. X-Ms-Request-Id: 2beedd1e-d01e-005f-1bd0-1fe9cf000000

Dst: https://<destination-server>.blob.core.windows.net/123456/456789/731f7b0c

2024/10/16 13:37:00 ==> REQUEST/RESPONSE (Try=1/67.071312ms, OpTime=8.26257104s) -- RESPONSE STATUS CODE ERROR

PUT https://<destination-server>.blob.core.windows.net/123456/456789%2F731f7b0c?blockid=MDAwMDADRZyL3jVIVVvyV4vy%2F57nMDAwMDAwMDAwMDAwMDAw&comp=block&se=2024-10-31T13%3A35%3A42Z&sig=-REDACTED-&sp=c&sr=b&sv=2024-11-04

Accept: application/xml

Content-Length: 0

User-Agent: AzCopy/10.26.0 azsdk-go-azblob/v1.4.0 (go1.22.5; linux)

X-Ms-Client-Request-Id: 75fce8dc-54a8-4123-48ea-f9304ae66420

x-ms-copy-source: https://<source-server>.blob.core.windows.net/123456/456789%2F731f7b0c?se=2024-10-31T13%3A35%3A42Z&sig=-REDACTED-&sp=r&sr=b&sv=2024-11-04

x-ms-source-range: bytes=0-8388607

x-ms-version: 2023-08-03

--------------------------------------------------------------------------------

RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.

Content-Length: 279

Content-Type: application/xml

Date: Wed, 16 Oct 2024 13:37:00 GMT

Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

X-Ms-Client-Request-Id: 75fce8dc-54a8-4123-48ea-f9304ae66420

X-Ms-Error-Code: AuthorizationPermissionMismatch

X-Ms-Request-Id: a648e3b1-401e-00a9-0ed0-1f9c81000000

X-Ms-Version: 2023-08-03

Response Details: <Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission. </Message>

2024/10/16 13:37:00 JobID=8b9c4503-35de-5548-5bf2-578bf2ff9ee7 canceled

2024/10/16 13:37:00 all parts of entire Job 8b9c4503-35de-5548-5bf2-578bf2ff9ee7 successfully completed, cancelled or paused

2024/10/16 13:37:00 all parts of entire Job 8b9c4503-35de-5548-5bf2-578bf2ff9ee7 successfully cancelled

2024/10/16 13:37:00 is part of Job which 1 total number of parts done

2024/10/16 13:37:01 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 0, E: 1, T: 1, GRs: 32

2024/10/16 13:37:01 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,

2024/10/16 13:37:01

Diagnostic stats:

IOPS: 0

End-to-end ms per request: 8262

Network Errors: 0.00%

Server Busy: 0.00%

Job 8b9c4503-35de-5548-5bf2-578bf2ff9ee7 summary

Elapsed Time (Minutes): 0.1667

Number of File Transfers: 1

Number of Folder Property Transfers: 0

Number of Symlink Transfers: 0

Total Number of Transfers: 1

Number of File Transfers Completed: 0

Number of Folder Transfers Completed: 0

Number of File Transfers Failed: 0

Number of Folder Transfers Failed: 0

Number of File Transfers Skipped: 0

Number of Folder Transfers Skipped: 0

Total Number of Bytes Transferred: 0

Final Job Status: Cancelled

2024/10/16 13:37:01 Closing Log

Success log for ~1MB blob size...

2024/10/16 13:58:44 Final job part has been scheduled

2024/10/16 13:58:44 INFO: [P#0-T#0] Starting transfer: Source "https://<source-server>.blob.core.windows.net/123456/456789/731f7b0c" Destination "https://<destination-server>.blob.core.windows.net/123456/456789/731f7b0c". Specified chunk size 8388608

2024/10/16 13:58:46 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 1, E: 0, T: 1, GRs: 32

2024/10/16 13:58:46 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,

2024/10/16 13:58:48 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 1, E: 0, T: 1, GRs: 32

2024/10/16 13:58:48 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,

2024/10/16 13:58:50 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 1, E: 0, T: 1, GRs: 32

2024/10/16 13:58:50 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,

2024/10/16 13:58:52 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 1, E: 0, T: 1, GRs: 32

2024/10/16 13:58:52 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total,

2024/10/16 13:58:53 ==> REQUEST/RESPONSE (Try=1/60.735674ms, OpTime=8.282378661s) -- RESPONSE SUCCESSFULLY RECEIVED

GET https://<destination-server>.blob.core.windows.net/123456/456789%2F731f7b0c?comp=properties&restype=account&se=2024-10-31T13%3A57%3A31Z&sig=-REDACTED-&sp=c&sr=b&sv=2024-11-04

X-Ms-Request-Id: [2c9edc81-e01e-008e-4fd3-1fe227000000]

2024/10/16 13:58:53 ==> REQUEST/RESPONSE (Try=1/144.060498ms, OpTime=144.108074ms) -- RESPONSE SUCCESSFULLY RECEIVED

PUT https://<destination-server>.blob.core.windows.net/123456/456789%2F731f7b0c?se=2024-10-31T13%3A57%3A31Z&sig=-REDACTED-&sp=c&sr=b&sv=2024-11-04

X-Ms-Request-Id: [2c9edcb2-e01e-008e-79d3-1fe227000000]

2024/10/16 13:58:53 ==> REQUEST/RESPONSE (Try=1/50.111282ms, OpTime=50.148197ms) -- RESPONSE STATUS CODE ERROR

HEAD https://<destination-server>.blob.core.windows.net/123456/456789%2F731f7b0c?se=2024-10-31T13%3A57%3A31Z&sig=-REDACTED-&sp=c&sr=b&sv=2024-11-04

Accept: application/xml

User-Agent: AzCopy/10.26.0 azsdk-go-azblob/v1.4.0 (go1.22.5; linux)

X-Ms-Client-Request-Id: 0afc46c9-c14e-4a0a-5e27-1972905165b4

x-ms-version: 2023-08-03

--------------------------------------------------------------------------------

RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.

Date: Wed, 16 Oct 2024 13:58:52 GMT

Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

X-Ms-Client-Request-Id: 0afc46c9-c14e-4a0a-5e27-1972905165b4

X-Ms-Error-Code: AuthorizationPermissionMismatch

X-Ms-Request-Id: 2c9edd2f-e01e-008e-6ed3-1fe227000000

X-Ms-Version: 2023-08-03

Response Details:

2024/10/16 13:58:53 ERR: [P#0-T#0] Could not read destination length. If the destination is write-only, use --check-length=false on the command line.

2024/10/16 13:58:53 INFO: [P#0-T#0] COPYSUCCESSFUL: https://<destination-server>.blob.core.windows.net/123456/456789/731f7b0c

2024/10/16 13:58:53 JobID=a2351294-6679-9942-707c-16c037f84b8a, Part#=0, TransfersDone=1 of 1

2024/10/16 13:58:53 all parts of entire Job a2351294-6679-9942-707c-16c037f84b8a successfully completed, cancelled or paused

2024/10/16 13:58:53 is part of Job which 1 total number of parts done

2024/10/16 13:58:54 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 0, E: 0, T: 0, GRs: 32

2024/10/16 13:58:54 100.0 %, 1 Done, 0 Failed, 0 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 4.3231

2024/10/16 13:58:54

Diagnostic stats:

IOPS: 0

End-to-end ms per request: 2825

Network Errors: 0.00%

Server Busy: 0.00%

Job a2351294-6679-9942-707c-16c037f84b8a summary

Elapsed Time (Minutes): 0.1667

Number of File Transfers: 1

Number of Folder Property Transfers: 0

Number of Symlink Transfers: 0

Total Number of Transfers: 1

Number of File Transfers Completed: 1

Number of Folder Transfers Completed: 0

Number of File Transfers Failed: 0

Number of Folder Transfers Failed: 0

Number of File Transfers Skipped: 0

Number of Folder Transfers Skipped: 0

Total Number of Bytes Transferred: 1080983

Final Job Status: Completed

2024/10/16 13:58:54 Closing Log

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,183 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,876 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Keshavulu Dasari 765 Reputation points Microsoft Vendor
    2024-10-16T20:59:59.31+00:00

    Hi David Sullivan,
    Welcome to Microsoft Q&A Forum, thank you for posting our query here!

    With AzCopy when copying blobs larger than 10MB between storage accounts you can try below to resolve the problem

    1. Increase Concurrency: Adjust the AZCOPY_CONCURRENCY_VALUE environment variable to increase the number of concurrent operations. This can help improve performance and handle larger blobs more efficiently.
    2. Network Configuration: Ensure that both storage accounts are properly configured for network access, especially since they are using private endpoints. Verify that the necessary firewall and virtual network settings are correctly set up. Run a performance benchmark test using AzCopy to identify any bottlenecks. This can help you understand if there are specific performance issues related to your setup.

    Check for Errors in Logs: Review the failure logs for any specific error messages that might indicate the cause of the issue. Common errors could be related to network timeouts, insufficient permissions, or other configuration problems. Make sure your AzCopy commands are optimized for large file transfers.
    For example, you might need to adjust parameters like --s2s-preserve-access-tier if you’re copying to a premium block blob storage account.
    For more information:
    https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-optimize https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-blobs-copy

    If you can share the specific error messages from your failure logs, I might be able to provide more advice,
    Please let us know if you have any further queries. I’m happy to assist you further

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.