Getting error 9006 when creating a Data Factory linked Service from Amazon S3

Question

Getting error 9006 when creating a Data Factory linked Service from Amazon S3

LD McElrath 0

Hello,

I am getting error 9006 when creating a Data Factory linked Service from Amazon S3,

The details returned are -
"Credential is incorrect or do not have the permission to access. Access Denied The remote server returned an error: (403) Forbidden. Activity ID: ff2ddf6c-38fa-4c23-b1c2-a42e3b4ba20e.:

I also get an error when trying to use the azcopy copy command instead of using ADF.

Here are the details of the error -

INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.15.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=CannotVerifyCopySource) =====
Description=Forbidden
RequestId:19d2978e-701e-0007-6011-516d8f000000
Time:2023-03-07T16:24:34.7681897Z, Details: 
   Code: CannotVerifyCopySource

...


   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Forbidden
   Content-Length: [200]
   Content-Type: [application/xml]
   Date: [Tue, 07 Mar 2023 16:24:33 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [8fa81503-02d9-416d-432e-629064934632]
   X-Ms-Error-Code: [CannotVerifyCopySource]
   X-Ms-Request-Id: [19d2978e-701e-0007-6011-516d8f000000]
   X-Ms-Version: [2020-10-02]

My feeling is that it's related to our account permissions on the S3 site, as we are able to perform this task against another company without issue.

Our permissions with the site throwing the error is as follows -

"Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:GetObjectTagging",
                "s3:GetObjectVersion",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],

Note that 's3:ListAllMyBuckets' is not included, which is a permission we have with the site that works.

Any help or assistance would be appreciated.

Thanks.

MartinJaffer-MSFT 26,236 Reputation points

2023-03-08T16:26:46.8266667+00:00

@LD McElrath Hello and welcome to Microsoft Q&A.

Please correct me if my understanding is incorrect. In Azure Data Factory, you are trying to create a linked service pointing to your S3 Bucket. During this process you are getting a "Forbidden" error. You haven't specified at which point the error appears, but I suspect it is either when doing "Test Connection" in Linked Service, or, when trying to browse the file path in the dataset. That would require ListAllMyBuckets.

To use "Test Connection" without that permission, use "Test Connection -> To Linked File Path" as shown in pictures below.

Details on how permissions effect the UI.

Please let me know if this helps, or if I got something wrong.
LD McElrath 0 Reputation points

2023-03-09T14:46:20.41+00:00

Thanks Martin - yes, that's correct. I get the error when testing the connection. I went ahead and tried again while using "to file path" and providing a bucket name, but still get the error.

<Edited - I attached the wrong error>
MartinJaffer-MSFT 26,236 Reputation points

2023-03-09T20:52:54.58+00:00
@LD McElrath Hmm. So it doesn't work with either AzCopy or Data Factory. Try re-entering the credentials before testing connection?

Do you have any services which are connecting sucessfully? If not, then the problem may lie on the S3 side.

If all else fails, try adding the last permission and test again. This is on the outside chance there is a bug or doc is wrong. If adding the last permission makes no difference, then I'm thinking the credentials got changed. This is S3, and not S3 compatible, right?

From what I can see in your screenshot, you have not parameterized anything in the linked service, so that cannot be a complication.

Well lets try with these troubleshooting steps first.

Use Amazon S3 CLI to connect with same credentials you put into ADF

do aws s3 ls to try listing buckets, or do the specific bucket.

Just in case the test connection is a false negative, try doing "preview data" using the dataset.
LD McElrath 0 Reputation points

2023-03-09T21:59:24.1233333+00:00

Hi Martin,

Yes, I am using S3 and not S3 compatible.

For the troubleshooting steps, I am able to connect through the cli. When I run "aws s3 ls <bucket>", it works fine and gives me a list of the files. When I run it without the bucket, "aws s3 ls", I get the error below -

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

I have not yet been able to add the permission for "ListAllMyBuckets", so the error sounds consistent with that.
MartinJaffer-MSFT 26,236 Reputation points

2023-03-13T16:01:54.23+00:00

@LD McElrath I agree with your assessment of error and permission matching.

The doc says your should be able todo without it. I can consider this contradiction a bug.

However there is good news!

You don't need "ListAllMyBuckets" to do the actual copy. ListAllMyBuckets only kicks in for browsing and similar actions. If the filepath is fully defined, there is no need to browse. That's my take anyway (90% confidence).

So, I can either give you a support ticket, or you can just skip over the test connection step. The support ticket is so engineers can do a deeper investigation, and have a starting point for fixing this.

One thing I noticed from your screenshot, you did provide a bucket but no directory. I wonder if that makes a difference. It is the weird kind of thing I've come to expect.
LD McElrath 0 Reputation points

2023-03-17T15:37:01.9966667+00:00

Thanks Martin, I've reached out to the vendor using S3 and will follow up.
MartinJaffer-MSFT 26,236 Reputation points

2023-03-21T19:02:34.09+00:00

@LD McElrath Have you heard back? Are you any closer to a resolution?
HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2023-03-31T16:58:20.8833333+00:00

Hello @LD McElrath , We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet . In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help . Thanks Himanshu
LD McElrath 0 Reputation points

2023-04-12T20:47:34.3933333+00:00

Hi Martin and Himanshu, My apologies for the delayed response. Would it be possible to post a ticket for this? Thanks

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2023-03-08T16:26:46.8266667+00:00

@LD McElrath Hello and welcome to Microsoft Q&A.

Please correct me if my understanding is incorrect. In Azure Data Factory, you are trying to create a linked service pointing to your S3 Bucket. During this process you are getting a "Forbidden" error. You haven't specified at which point the error appears, but I suspect it is either when doing "Test Connection" in Linked Service, or, when trying to browse the file path in the dataset. That would require ListAllMyBuckets.

To use "Test Connection" without that permission, use "Test Connection -> To Linked File Path" as shown in pictures below.

Details on how permissions effect the UI.

Please let me know if this helps, or if I got something wrong.
LD McElrath 0 Reputation points

2023-03-09T14:46:20.41+00:00

Thanks Martin - yes, that's correct. I get the error when testing the connection. I went ahead and tried again while using "to file path" and providing a bucket name, but still get the error.

<Edited - I attached the wrong error>
MartinJaffer-MSFT 26,236 Reputation points

2023-03-09T20:52:54.58+00:00

@LD McElrath Hmm. So it doesn't work with either AzCopy or Data Factory. Try re-entering the credentials before testing connection?

Do you have any services which are connecting sucessfully? If not, then the problem may lie on the S3 side.

If all else fails, try adding the last permission and test again. This is on the outside chance there is a bug or doc is wrong. If adding the last permission makes no difference, then I'm thinking the credentials got changed. This is S3, and not S3 compatible, right?

From what I can see in your screenshot, you have not parameterized anything in the linked service, so that cannot be a complication.

Well lets try with these troubleshooting steps first.

Use Amazon S3 CLI to connect with same credentials you put into ADF

do aws s3 ls to try listing buckets, or do the specific bucket.

Just in case the test connection is a false negative, try doing "preview data" using the dataset.
LD McElrath 0 Reputation points

2023-03-09T21:59:24.1233333+00:00

Hi Martin,

Yes, I am using S3 and not S3 compatible.

For the troubleshooting steps, I am able to connect through the cli. When I run "aws s3 ls <bucket>", it works fine and gives me a list of the files. When I run it without the bucket, "aws s3 ls", I get the error below -

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

I have not yet been able to add the permission for "ListAllMyBuckets", so the error sounds consistent with that.
MartinJaffer-MSFT 26,236 Reputation points

2023-03-13T16:01:54.23+00:00

@LD McElrath I agree with your assessment of error and permission matching.

The doc says your should be able todo without it. I can consider this contradiction a bug.

However there is good news!

You don't need "ListAllMyBuckets" to do the actual copy. ListAllMyBuckets only kicks in for browsing and similar actions. If the filepath is fully defined, there is no need to browse. That's my take anyway (90% confidence).

So, I can either give you a support ticket, or you can just skip over the test connection step. The support ticket is so engineers can do a deeper investigation, and have a starting point for fixing this.

One thing I noticed from your screenshot, you did provide a bucket but no directory. I wonder if that makes a difference. It is the weird kind of thing I've come to expect.
LD McElrath 0 Reputation points

2023-03-17T15:37:01.9966667+00:00

Thanks Martin, I've reached out to the vendor using S3 and will follow up.
MartinJaffer-MSFT 26,236 Reputation points

2023-03-21T19:02:34.09+00:00

@LD McElrath Have you heard back? Are you any closer to a resolution?
HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2023-03-31T16:58:20.8833333+00:00

Hello @LD McElrath , We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet . In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help . Thanks Himanshu
LD McElrath 0 Reputation points

2023-04-12T20:47:34.3933333+00:00

Hi Martin and Himanshu, My apologies for the delayed response. Would it be possible to post a ticket for this? Thanks

Share via

Getting error 9006 when creating a Data Factory linked Service from Amazon S3

Your answer