Azure 'azcopy sync' issue in syncing across containers using sas token for a multitenant app

Question

I am trying to sync data across Azure containers in different accounts using a MultiTenant App and azcopy tool.

The syncing happens via "azcopy sync" and using separate SAS tokens for both source storage account and destination storage account.

I am generating short lived sas tokens using the Java SDK following the user delegation key method.

Here is the scenario:

• Account1 (destination) has App1 registered. i.e. Account1 is home tenant for App1. Account1 has StorageAccount1 and Container1 configured.
  App1 is given "Storage Blob Data Contributor" role on StorageAccount1
• Account2 (source) has StorageAccount2 and Container2 configured. It is the data source for us. Here, App1 is added as a ServicePrincipal via:

az ad sp create --id client-id-of-App1-in-Account1

In Account2, for this SP, we also gave the Storage Blob Data Reader Role as:

az role assignment create \
--assignee-object-id <object-id-for-this-sp> \
--role 2a2b9908-6ea1-4ae2-8e65-a410df84e7d1 \
--scope /subscriptions/<subsid-account2>/resourceGroups/<resgrpname>/providers/Microsoft.Storage/storageAccounts/<storagename>

This completes the setup.

---

Now using Java SDK, I generated a user delegation key for both source and destination. The snippet looks something like below.

genSasToken(String storageAccountName, String containerName,
                             String tenantId,
                             String azureAppClientId,
                             String azureAppClientSecret,
                             boolean isDestinationAccount) {
    BlobContainerSasPermission blobContainerSasPermission =
        new BlobContainerSasPermission().setReadPermission(true).setListPermission(true);
    if (isDestinationAccount) {
      blobContainerSasPermission.setCreatePermission(true)
              .setAddPermission(true)
              .setWritePermission(true)
              .setExecutePermission(true);
    }
    BlobServiceSasSignatureValues builder =
        new BlobServiceSasSignatureValues(OffsetDateTime.now().plusHours(1), blobContainerSasPermission)
            .setProtocol(SasProtocol.HTTPS_ONLY);
    // Create a BlobServiceClient object which will be used to create a container client
    String endpoint = String.format(Locale.ROOT, "https://%s.blob.core.windows.net",
            storageAccountName);
    ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
                                                        .clientId(azureAppClientId)
                                                        .clientSecret(azureAppClientSecret)
                                                        .tenantId(tenantId)
                                                        .build();

    BlobServiceClient blobServiceClient =
        new BlobServiceClientBuilder().endpoint(endpoint).credential(clientSecretCredential).buildClient();
    BlobContainerClient blobContainerClient =
        blobServiceClient.getBlobContainerClient(containerName);
    // Get a user delegation key for the Blob service that's valid for one hour.
    // You can use the key to generate any number of shared access signatures over the lifetime of the key.
    OffsetDateTime keyStart = OffsetDateTime.now();
    OffsetDateTime keyExpiry = OffsetDateTime.now().plusHours(1);
    UserDelegationKey userDelegationKey = blobServiceClient.getUserDelegationKey(keyStart, keyExpiry);

    String sas = blobContainerClient.generateUserDelegationSas(builder, userDelegationKey);
    return sas;
  }

Above method is called for both source and destination and gives us SAS tokens generated programmatically.

Interesting thing happening is this:

azcopy sync https://storageaccount2/container2/?sas-token-for2 https://storageaccount1/container1/?sas-token-for1

above sync errors out as

INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /Users/runner/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.10.1-0.20201022074806-8d8fc11be726/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) =====
Description=This request is not authorized to perform this operation.
RequestId:xxx
Time:2021-01-27T10:26:34.9282634Z, Details:
Code: AuthorizationFailure
GET https://storageaccount1.blob.core.windows.net/container1/?comp=properties&restype=account&se=2021-01-27t11%3A10%3A12z&sig=-REDACTED-&ske=2021-01-27t11%3A10%3A12z&skoid=xxx&sks=b&skt=2021-01-27t10%3A10%3A12z&sktid=xxx&skv=2020-02-10&sp=racwle&spr=https&sr=c&sv=2020-02-10&timeout=901
User-Agent: [AzCopy/10.8.0 Azure-Storage/0.10 (go1.13; darwin)]
X-Ms-Client-Request-Id: [xxx]
X-Ms-Version: [2019-12-12]
--------------------------------------------------------------------------------
RESPONSE Status: 403 This request is not authorized to perform this operation.
Content-Length: [246]
Content-Type: [application/xml]
Date: [Wed, 27 Jan 2021 10:26:34 GMT]
Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
X-Ms-Client-Request-Id: [xxx]
X-Ms-Error-Code: [AuthorizationFailure]
X-Ms-Request-Id: [xxx]
X-Ms-Version: [2019-12-12]

But, when I try to copy from source to localhost using same sas token 2, it works.

azcopy sync https://storageaccount2/container2/sas-token-for2 /tmp

and

when I try to copy a localhost folder to destination using same sas token it also works.

azcopy sync /tmp https://storageaccount1/container1/sas-token-for1

So the tokens work individually like above.

But

azcopy sync https://storageaccount2/container2/sas-token-for2 https://storageaccount1/container1/sas-token-for1

Fails.

Any pointers what might be the issue here?

Also, generating a sas from azure portal for destination storage account and using it in azcopy works fine. i.e.

azcopy sync storageaccount2/container2/sas-token-for2 storageaccount1/container1/sas-token-generated-from-portal

This works fine

I also posted the same query on SO here