Share via

copy files using azcopy with SAS fails with http 403

Abdur Rab Omar Sahaf 1 Reputation point
2020-12-29T15:02:52.963+00:00

I am trying to copy files using azcopy and SAS to azure blob storage. copy fails from the command prompt with following error but same SAS key works fine in Storage Explorer. I am not sure what is issue.

2020/12/29 14:44:01 AzcopyVersion 10.8.0
2020/12/29 14:44:01 OS-Environment windows
2020/12/29 14:44:01 OS-Architecture amd64
2020/12/29 14:44:01 Log times are in UTC. Local time is 29 Dec 2020 09:44:01
2020/12/29 14:44:01 Job-Command copy H:\Backup_Cloud\Full* https://sttxxxxxxxxxx.blob.core.windows.net/full/?se=xxxxxsig=-REDACTED-xxxxxxxxxxxxxxxxx2 --overwrite=prompt --blob-type Detect --recursive
2020/12/29 14:44:01 Number of CPUs: 8
2020/12/29 14:44:01 Max file buffer RAM 4.000 GB
2020/12/29 14:44:01 Max concurrent network operations: 128 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2020/12/29 14:44:01 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2020/12/29 14:44:01 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2020/12/29 14:44:01 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2020/12/29 14:44:01 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2020/12/29 14:44:01 Max open files when downloading: 2147483119 (auto-computed)
2020/12/29 14:44:01 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2020-12-29T14:43:56Z or --include-after=2020-12-29T14:43:56Z
2020/12/29 14:44:01 Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2020/12/29 14:44:01 Final job part has been created
2020/12/29 14:44:01 JobID=1a731be7-52fc-9841-76c8-ac2643c322f8, credential type: Anonymous
2020/12/29 14:44:01 PERF: primary performance constraint is Unknown. States: X: 0, O: 0, M: 0, L: 0, R: 0, D: 0, W: 0, F: 0, B: 0, E: 0, T: 0, GRs: 12
2020/12/29 14:44:01 scheduling JobID=1a731be7-52fc-9841-76c8-ac2643c322f8, Part#=0, Transfer#=0, priority=0
2020/12/29 14:44:01 0.0 %, 0 Done, 0 Failed, 6 Pending, 0 Skipped, 6 Total,
2020/12/29 14:44:01 scheduling JobID=1a731be7-52fc-9841-76c8-ac2643c322f8, Part#=0, Transfer#=1, priority=0
2020/12/29 14:44:01 scheduling JobID=1a731be7-52fc-9841-76c8-ac2643c322f8, Part#=0, Transfer#=2, priority=0
2020/12/29 14:44:01 scheduling JobID=1a731be7-52fc-9841-76c8-ac2643c322f8, Part#=0, Transfer#=3, priority=0
2020/12/29 14:44:01 scheduling JobID=1a731be7-52fc-9841-76c8-ac2643c322f8, Part#=0, Transfer#=4, priority=0
2020/12/29 14:44:01 scheduling JobID=1a731be7-52fc-9841-76c8-ac2643c322f8, Part#=0, Transfer#=5, priority=0
2020/12/29 14:44:01 Final job part has been scheduled
2020/12/29 14:44:01 INFO: [P#0-T#0] has worker 4 which is processing TRANSFER
2020/12/29 14:44:01 INFO: [P#0-T#2] has worker 25 which is processing TRANSFER
2020/12/29 14:44:01 INFO: [P#0-T#1] has worker 2 which is processing TRANSFER
2020/12/29 14:44:01 INFO: [P#0-T#4] has worker 46 which is processing TRANSFER
2020/12/29 14:44:01 INFO: [P#0-T#5] has worker 5 which is processing TRANSFER
2020/12/29 14:44:01 INFO: [P#0-T#3] has worker 35 which is processing TRANSFER
2020/12/29 14:44:01 INFO: [P#0-T#0] Starting transfer: Source "%5C%5C?\H:\Backup_Cloud\Full\Tfs_OACCAC_backup_2020_12_27_200001_8935540.bak" Destination "https://sttfxxxxxx.blob.core.windows.net/full/Tfs_OACCAC_backup_2020_12_27_200001_8935540.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12". Specified chunk size 8388608
2020/12/29 14:44:01 INFO: [P#0-T#4] Starting transfer: Source "%5C%5C?\H:\Backup_Cloud\Full\Tfs_Warehouse_backup_2020_12_27_200001_9248841.bak" Destination "https://sttxxxxxx.blob.core.windows.net/full/Tfs_Warehouse_backup_2020_12_27_200001_9248841.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12". Specified chunk size 8388608
2020/12/29 14:44:01 INFO: [P#0-T#2] Starting transfer: Source "%5C%5C?\H:\Backup_Cloud\Full\Tfs_Sandbox_backup_2020_12_27_200001_8935540.bak" Destination "https://sttxxxxxxxxx.blob.core.windows.net/full/Tfs_Sandbox_backup_2020_12_27_200001_8935540.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12". Specified chunk size 8388608
2020/12/29 14:44:01 INFO: [P#0-T#3] Starting transfer: Source "%5C%5C?\H:\Backup_Cloud\Full\Tfs_Archive_backup_2020_12_27_200001_8779317.bak" Destination "https://sttfxxxxxx.blob.core.windows.net/full/Tfs_Archive_backup_2020_12_27_200001_8779317.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12". Specified chunk size 8388608
2020/12/29 14:44:01 INFO: [P#0-T#1] Starting transfer: Source "%5C%5C?\H:\Backup_Cloud\Full\Tfs_Configuration_backup_2020_12_27_200001_9096218.bak" Destination "https://sttfxxxxxx.blob.core.windows.net/full/Tfs_Configuration_backup_2020_12_27_200001_9096218.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12". Specified chunk size 8388608
2020/12/29 14:44:01 INFO: [P#0-T#5] Starting transfer: Source "%5C%5C?\H:\Backup_Cloud\Full\TFS_WSS_Content_backup_2020_12_27_200001_8935540.bak" Destination "https://sttxxxxxxxx.blob.core.windows.net/full/TFS_WSS_Content_backup_2020_12_27_200001_8935540.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12". Specified chunk size 8388608
2020/12/29 14:44:01 ==> REQUEST/RESPONSE (Try=1/7.0053ms, OpTime=70.0056ms) -- RESPONSE STATUS CODE ERROR
HEAD https://sttfxxxxxxx.blob.core.windows.net/full/TFS_WSS_Content_backup_2020_12_27_200001_8935540.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12&timeout=901
User-Agent: [AzCopy/10.8.0 Azure-Storage/0.10 (go1.13; Windows_NT)]
X-Ms-Client-Request-Id: [6b7dd5e1-36dd-4c22-58dc-1e446311fa8e]
X-Ms-Version: [2019-12-12]

RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Date: [Tue, 29 Dec 2020 14:44:00 GMT]
Server: [Microsoft-HTTPAPI/2.0]
X-Ms-Error-Code: [AuthenticationFailed]
X-Ms-Request-Id: [4e4d5ebc-001e-0040-5df1-dd6a99000000]
Response Details:

2020/12/29 14:44:01 ==> REQUEST/RESPONSE (Try=1/7.0053ms, OpTime=70.0056ms) -- RESPONSE STATUS CODE ERROR
HEAD https://stxxxxxxxxxxx.blob.core.windows.net/full/Tfs_Configuration_backup_2020_12_27_200001_9096218.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12&timeout=901
User-Agent: [AzCopy/10.8.0 Azure-Storage/0.10 (go1.13; Windows_NT)]
X-Ms-Client-Request-Id: [a8fcdaad-73a7-49e3-6caf-cc2ecd75c252]
X-Ms-Version: [2019-12-12]

RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Date: [Tue, 29 Dec 2020 14:44:00 GMT]
Server: [Microsoft-HTTPAPI/2.0]
X-Ms-Error-Code: [AuthenticationFailed]
X-Ms-Request-Id: [e1895697-901e-0039-51f1-ddeb8a000000]
Response Details:

2020/12/29 14:44:01 ==> REQUEST/RESPONSE (Try=1/8.0005ms, OpTime=71.0008ms) -- RESPONSE STATUS CODE ERROR
HEAD https://stxxxxxxxxx.blob.core.windows.net/full/Tfs_Warehouse_backup_2020_12_27_200001_9248841.bak?sig=-REDACTED-&sp=racwdl&sr=c&sv=2019-12-12&timeout=901
User-Agent: [AzCopy/10.8.0 Azure-Storage/0.10 (go1.13; Windows_NT)]
X-Ms-Client-Request-Id: [3c94aeda-d762-45c1-5371-1c9d4dbefca9]
X-Ms-Version: [2019-12-12]

RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Date: [Tue, 29 Dec 2020 14:44:00 GMT]
Server: [Microsoft-HTTPAPI/2.0]
X-Ms-Error-Code: [AuthenticationFailed]
X-Ms-Request-Id: [a27a6c72-d01e-0001-55f1-dd928a000000]

Diagnostic stats:
IOPS: 2
End-to-end ms per request: 73
Network Errors: 0.00%
Server Busy: 0.00%

Job 1a731be7-52fc-9841-76c8-ac2643c322f8 summary
Elapsed Time (Minutes): 0.0337
Number of File Transfers: 6
Number of Folder Property Transfers: 0
Total Number of Transfers: 6
Number of Transfers Completed: 0
Number of Transfers Failed: 6
Number of Transfers Skipped: 0
TotalBytesTransferred: 0
Final Job Status: Failed

Azure Storage Explorer
Azure Storage Explorer
An Azure tool that is used to manage cloud storage resources on Windows, macOS, and Linux.
242 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,912 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. JoyDutt 826 Reputation points
    2020-12-29T16:09:57.51+00:00

    Azcopy seems to need the seconds in start and expiry in order for it to work, however, this is unfortunately not documented anywhere.

    EXPIRE=$(date -u -d "1 day" '+%Y-%m-%dT%H:%M:%SZ') START=$(date -u -d "-1 day" '+%Y-%m-%dT%H:%M:%SZ') ACCOUNT="some_account" KEY="some_key" CONTAINER="some_container" BLOB="test.jpg" SAS=$(az storage account generate-sas --account-name "$ACCOUNT" --account-key "$KEY" --start "$START" --expiry "$EXPIRE" --https-only --permissions acdlpruw --resource-types sco --services bfqt | sed 's/%3A/:/g;s/"//g')

    azcopy copy "https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB}?${SAS}" "$HOME/test.jpg"

    (If above does not work then You can re-try after removing - the --account-key parameter in7th line above).

    Example:

    • task: Bash@3 displayName: Install azcopy inputs: targetType: 'inline' script: | mkdir $(Agent.ToolsDirectory)/azcopy && cd "$_" wget -O azcopy_v10.tar.gz https://aka.ms/downloadazcopy-v10-linux tar -xf azcopy_v10.tar.gz --strip-components=1
    • task: AzureCLI@2 displayName: Download using azcopy inputs: azureSubscription: my-vmssagents-service-connection scriptType: bash scriptLocation: inlineScript inlineScript: | export STORE_NAME="data" export CONTAINER_NAME="data" 
      NOW=`date +"%Y-%m-%dT%H:%M:00Z"` \
EXPIRY=`date -d "$NOW + 1 day" +"%Y-%m-%dT%H:%M:00Z"` \
&& export SAS_TOKEN=$( az storage container generate-sas \
    --account-name $STORE_NAME \
    --name $CONTAINER_NAME \
    --start $NOW \
    --expiry $EXPIRY \
    --permissions acdlrw \
    --output tsv )

$(Agent.ToolsDirectory)/azcopy/azcopy copy \
    &#34;https://${STORE_NAME}.blob.core.windows.net/${CONTAINER_NAME}/${<!-- -->{ parameters.folder }}/?${SAS_TOKEN}&#34; \
    &#34;.&#34; --recursive --include-pattern &#34;*c_*b.nc;left.nc;right.nc&#34; # &lt;-- my specific pattern

    Ref: https://github.com/Azure/azure-storage-blob-go/blob/master/azblob/zc_sas_query_params.go#L139

    (Please don’t forget to **"Accept the answer" & “up-vote” **, this can be beneficial to other members. Thank You)

    Regards, ** J.D. **

    0 comments