Generating SAS Token for Azure Datalake does not work properly

Question

I'm having trouble accessing my Azure Data Lake container using a SAS token generated by Python. While a manually generated SAS token works, the Python-generated one leads to an authentication error (see picture):.

Now this is not the best solution, so I tried generating the token based on my Accountname, accountkey and the container with python. I also set the starting point 5 minutes prior to the current timestamp. Here is the code:

class ADLImporter:
    def __init__(self):
        self.logger_inst = logger.create_logger("INFO")
        sas_token = self._generate_sas_token(account_name=ADL.ACCOUNTNAME, 
                                            account_key=ADL.ACCOUNTKEY, 
                                            container_name=ADL.CONTAINER_NAME,
                                            expiry_time=helpers.get_timestamp_plus_5_hours(),
                                            start_time=helpers.get_timestamp_minus_5_minutes())
        print(sas_token)
        # TODO remove sas token from env and check why it still does not work.
        sas_url = f"{ADL.ACCOUNTNAME}/{ADL.CONTAINER_NAME}?{sas_token}"
        self.container_client = ContainerClient.from_container_url(sas_url)
        self.logger_inst.info("Connected to container.")


    def _generate_sas_token(self, account_name, account_key, container_name, expiry_time, start_time):
        """
        Generate a SAS token for a blob container.

        Args:
          account_name (str): The storage account name.
          account_key (str): The storage account key.
          container_name (str): The name of the blob container.
          expiry_time (datetime): The time at which the SAS token expires.

        Returns:
          str: The SAS token.
        """
        sas_token = generate_container_sas(
            account_name=account_name,
            account_key=account_key,
            container_name=container_name,
            permission=ContainerSasPermissions(read=True, list=True),
            expiry=expiry_time,
            protocol='https', 
            version='2023-11-08'
        )  

        self.logger_inst.info("Created SAS Token.")

        return sas_token

However, if I replace the manually generated token with the one I generate with python, I get an authorization error:

......exec("raise error from None") # pylint: disable=exec-used # nosec File "<string>", line 1, in <module> azure.core.exceptions.ClientAuthenticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:2b4c23f5-601e-002d-41b0-112e45000000 Time:2023-11-07T19:30:45.5693910Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rl....

I do not know what I am doing wrong. At the end I end up with these tokens:

manually: sp=rl&st=2023-11-07T20:00:00Z&se=2023-11-08T03:00:00Z&spr=https&sv=2022-11-02&sr=c&sig=<sig...>

with python: se=2023-11-08T00%3A26%3A51Z&sp=rl&spr=https&sv=2023-08-03&sr=c&sig=<sig...>

Any ideas what could be wrong?

Answer 1

@Oweys - Thanks for the question and uisng MS Q&A platform.

The issue you are facing seems to be related to the format of the SAS token that you are generating with Python. The SAS token format is crucial, and even a small difference in formatting can lead to authentication errors. Let's go through your code and identify the issue.

Here's the manually generated SAS token for reference:

sp=rl&st=2023-11-07T20:00:00Z&se=2023-11-08T03:00:00Z&spr=https&sv=2022-11-02&sr=c&sig=<sig...>

And here's the Python-generated SAS token:

se=2023-11-08T00%3A26%3A51Z&sp=rl&spr=https&sv=2023-08-03&sr=c&sig=<sig...>

The differences between the two tokens are the date/time format, the version (sv), and possibly other factors. Let's address these issues one by one.

Date/Time Format: The date and time format in the manually generated token is in a specific ISO 8601 format. In the Python-generated token, you are encoding the time with URL encoding (e.g., %3A for :). You should format the datetime correctly in ISO 8601 format.

Version (sv): The manually generated token specifies sv=2022-11-02, which is the SAS token version. In your Python code, you're using sv=2023-08-03. Make sure to use the correct SAS token version. You should use the same version as the one that works, which seems to be sv=2022-11-02.

Check Other Parameters: Make sure that all other parameters, such as sp (permissions), spr (protocol), and sr (resource type), are consistent with the manually generated token.

Here's how you can modify your _generate_sas_token function to ensure the correct formatting:

from azure.storage.blob import generate_container_sas, ContainerSasPermissions

def _generate_sas_token(self, account_name, account_key, container_name, expiry_time, start_time):
    permissions = ContainerSasPermissions(read=True, list=True)
    sas_token = generate_container_sas(
        account_name=account_name,
        account_key=account_key,
        container_name=container_name,
        permission=permissions,
        expiry=expiry_time,
        start=start_time,
        protocol='https',
        version='2022-11-02'  # Use the correct version
    )

    self.logger_inst.info("Created SAS Token.")
    return sas_token

Ensure that you pass the correct version ('2022-11-02') as shown above and format the start_time and expiry_time as ISO 8601 datetime strings.

After making these changes, your Python-generated SAS token should match the manually generated one, and it should work correctly for accessing your Azure Data Lake container.

I hope this information helps. If you have any further questions or need more clarification, please let me know.