Azure Data Factory : How to set the bucket region of an Amazon S3 Compatible data source

Question

Hi there,

I'm trying to connect my Supabase storage, an Amazon S3 compatible storage to my Azure Data Factory for a copy flow. I'm able to connect to it with tools like Cyberduck, but it does not work with ADF. I guess the 403 error is related to the missing of the region information, which I can reproduce with Cyberduck by omitting it and getting a SignatureDoesNotMatch error.
Supabase does not allow to add the region as a dns prefix like "region.project.supabase.co". I've tried to add a string parameter named "region" with the "eu-central-1" value, but I think it's ignored.

Answer 1

To resolve the issue of setting the bucket region for an Amazon S3 compatible data source in ADF, follow these steps:

When setting up the Linked Service for your S3-compatible storage in Azure Data Factory, you need to ensure that you are using the correct authentication mechanism, such as Access Key and Secret Key, which you are likely already doing.

Since ADF does not provide a direct field to specify the region, try explicitly adding the region information in the connection string format (if supported by the Linked Service). This could look like:

s3.amazonaws.com/{bucket-name}?region={region}

Alternatively, when creating a Linked Service for S3-compatible storage, check for any option in the "Advanced" settings where you might input a custom endpoint and add the region there.

Ensure that the region is specified correctly in the Supabase configuration for your S3 storage. Even though Supabase doesn’t allow specifying the region in the DNS prefix, there might be an endpoint configuration you can use in ADF, such as:

https://{bucket-name}.s3.{region}.amazonaws.com

If Azure Data Factory allows for custom endpoints in the Linked Service configuration, try adding Supabase S3-compatible custom endpoint and the region. For example, if you're using a specific region like eu-central-1, ensure this is reflected in the endpoint URL.

In some cases, incorrect region names or case mismatches (like eu-central-1 versus EU-CENTRAL-1) can cause authentication errors. Verify the region's exact spelling in your Supabase storage.

If possible, see if there's a way to inject custom headers into the request (e.g., through a Web activity or API call) to include the region in the request headers for S3 authentication.

If you still encounter issues after trying these steps, check the ADF logs for any additional details that might help fine-tune the region settings or other parameters related to your connection.

Answer 2

Hi @Aurélien L

Apologies for the inconvenience.

The error code 9006 and the message indicating a Forbidden status code suggest that there is an issue with the credentials or permissions when trying to connect to the Amazon S3 bucket from Azure Data Factory. Here are some troubleshooting steps that might help you.

Check Permissions: Ensure that the IAM role or user associated with your S3 bucket has the necessary permissions. To copy data from Amazon S3, make sure you've been granted the following permissions for Amazon S3 object operations: s3:GetObject and s3:GetObjectVersion.

ListAllMyBuckets Permission: If you are getting the error during the “Test Connection” step, you might need the s3:ListAllMyBuckets permission. This permission is required for browsing the bucket structure in ADF.

For additional information please refer: Required permissions.

Re-enter Credentials: Sometimes, re-entering the credentials in the ADF linked service configuration can resolve the issue. Ensure that the access key and secret key are correct and have not expired.

I hope this information helps. Please do let us know if you have any further queries.

Thank you.