how to upload files from AWS lambda to Azure Storage ADLS Gen 2

Question

how to upload files from AWS lambda to Azure Storage ADLS Gen 2

Amit Mehta 30

I have to upload files coming in AWS S3 bucket to azure BLOB Storage in specific folder. So I have written a lambda function in python that gets invoked when a file is put in the S3 and the lambda code tries to upload to azure blob. But currently is fails with error -

Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.

What do I need to do on the Azure BLOB container side to allow this lambda upload to go through?

Here is the python code that I am using to upload:

		response = s3.get_object(Bucket=bucket, Key=key)
        #read object content
        content = response['Body'].read().decode('utf-8')
        #print(f"Content: {content}")
        #connect to Azure Blob Storage
        connection_string = os.environ['AZURE_STORAGE_CONNECTION_STRING']
        container_name = os.environ['container_name']
        folder_path = os.environ['folder_path']  # Specify the folder path within the container
        blob_service_client = BlobServiceClient.from_connection_string(connection_string)
        container_client = blob_service_client.get_container_client(container_name)
        file_name = os.path.basename(key)
        blob_name = folder_path + file_name
        print("BLOB NAME: " + blob_name)
        blob_client = container_client.get_blob_client(blob_name)
        #upload file to Azure Blob Storage
        blob_client.upload_blob(content, overwrite=True)

Keshavulu Dasari 4,480 Reputation points Microsoft External Staff

2025-03-21T18:18:16.83+00:00
Hi Amit Mehta,

I understand the issue is related to authorization. To resolve this, you need to ensure that your Azure Blob Storage is properly configured.

The best practice is to use Azure AD for authentication. This involves registering an application in Azure AD and assigning it the necessary roles.

Register an Application in Azure AD:

Go to the Azure portal and navigate to Azure Active Directory.

Select App registrations and click on New registration.

Provide a name for your application and register it.

Assign Roles to the Application:

After registering the application, go to your storage account in the Azure portal.

Navigate to Access control and click on Add role assignment.

Assign the Storage Blob Data Contributor role to your registered application.

Obtain Client ID and Secret:

In the Azure AD, go to your registered application and navigate to Certificates & secrets.

Create a new client secret and note down the Client ID and Client Secret.

Update your Lambda function to use the Azure AD credentials for authentication. You can use
the DefaultAzureCredential from the azure-identity library to handle the authentication.

Note: We can provide guidance on the Azure side, but Microsoft Q&A does not support AWS.

For more information:
https://learn.microsoft.com/en-us/entra/architecture/auth-sync-overview

https://stackoverflow.com/questions/63206596/authenticate-to-blobserviceclient-using-clientsecretcredential-in-a-native-app

If you have any other questions or further issues regarding this, please let me know in the comments.
Amit Mehta 30 Reputation points

2025-03-21T18:29:37.5033333+00:00

I see option to create a Private endpoint connections under the Security + Networking page on the specific Storage account. Should i go for this option to create a specific endpoint for my AWS Lambda function to send files to Storage container.

I also have a .Net Core application running at specific time daily from a On Premise server that needs to send data files and thinking of using this Private endpoint connections option there also.

Thanks

Amit Mehta
Amit Mehta 30 Reputation points

2025-03-21T19:27:44.03+00:00

Thanks for the information
Keshavulu Dasari 4,480 Reputation points Microsoft External Staff

2025-03-21T19:44:21.0666667+00:00

Hi Amit Mehta,

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be glad to assist you.

Accepted answer

0 additional answers

Your answer

Keshavulu Dasari 4,480 Reputation points Microsoft External Staff

2025-03-21T18:18:16.83+00:00

Hi Amit Mehta,

I understand the issue is related to authorization. To resolve this, you need to ensure that your Azure Blob Storage is properly configured.

The best practice is to use Azure AD for authentication. This involves registering an application in Azure AD and assigning it the necessary roles.

Register an Application in Azure AD:

Go to the Azure portal and navigate to Azure Active Directory.

Select App registrations and click on New registration.

Provide a name for your application and register it.

Assign Roles to the Application:

After registering the application, go to your storage account in the Azure portal.

Navigate to Access control and click on Add role assignment.

Assign the Storage Blob Data Contributor role to your registered application.

Obtain Client ID and Secret:

In the Azure AD, go to your registered application and navigate to Certificates & secrets.

Create a new client secret and note down the Client ID and Client Secret.

Update your Lambda function to use the Azure AD credentials for authentication. You can use
the DefaultAzureCredential from the azure-identity library to handle the authentication.

Note: We can provide guidance on the Azure side, but Microsoft Q&A does not support AWS.

For more information:
https://learn.microsoft.com/en-us/entra/architecture/auth-sync-overview

https://stackoverflow.com/questions/63206596/authenticate-to-blobserviceclient-using-clientsecretcredential-in-a-native-app

If you have any other questions or further issues regarding this, please let me know in the comments.
Amit Mehta 30 Reputation points

2025-03-21T18:29:37.5033333+00:00

I see option to create a Private endpoint connections under the Security + Networking page on the specific Storage account. Should i go for this option to create a specific endpoint for my AWS Lambda function to send files to Storage container.

I also have a .Net Core application running at specific time daily from a On Premise server that needs to send data files and thinking of using this Private endpoint connections option there also.

Thanks

Amit Mehta
Amit Mehta 30 Reputation points

2025-03-21T19:27:44.03+00:00

Thanks for the information
Keshavulu Dasari 4,480 Reputation points Microsoft External Staff

2025-03-21T19:44:21.0666667+00:00

Hi Amit Mehta,

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be glad to assist you.

Answer 1

@ Amit Mehta,

Using Private Endpoint connections can be a good option for securely connecting your AWS Lambda function and .Net Core application to Azure Blob Storage.

For Lambda Private Endpoints allow you to connect to Azure services over a private link, which means the traffic between your AWS Lambda function and Azure Blob Storage will be routed through a secure, private network.

You will need to set up a Site-to-Site VPN or an AWS Direct Connect to establish a private network connection between AWS and Azure.

Ensure that your AWS Lambda function is configured to use a VPC endpoint that can route traffic to the Azure Private Endpoint.

Update your Lambda function's network settings to use the private IP address of the Azure Private Endpoint.

For .Net Core Application, using a Private Endpoint for your .Net Core application running on an on-premises server can enhance security by ensuring that data transfers to Azure Blob Storage occur over a private network.

You will need to set up a VPN connection from your on-premises network to Azure to access the Private Endpoint.

Configure your on-premises network to route traffic to the Azure Private Endpoint.

Update your .Net Core application to use the private IP address of the Azure Private Endpoint for connecting to Azure Blob Storage.Using Private Endpoint connections can be a good option for securely connecting your AWS Lambda function and .Net Core application to Azure Blob Storage.

Benefits, Traffic is routed through a private network, reducing exposure to the public internet, Private connections can offer more consistent and potentially faster data transfer rates.

For more information:

https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-aws-bgp

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members
User's image

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be glad to assist you

Share via

how to upload files from AWS lambda to Azure Storage ADLS Gen 2

0 additional answers

Your answer