AWS S3 file cannot be found using ADF

Question

AWS S3 file cannot be found using ADF

Shuxin Yang (杨书新) 21

I'm using ADF to import a csv file from AWS S3 bucket, the folder was named a single forward slash, this odd folder name seems not allowed when using ADF, it forced to remove the slash when sending a request, and of course the response is 404 not found.

I wonder is it the only that I have to rename the folder, or is there anything I can do to make ADF accept the folder name?

Any assistance you can provide would be greatly appreciated.

Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2023-05-30T06:00:41.3866667+00:00

Hi,

Thanks for reaching out to Microsoft Q&A.

I dont think you can have only / as a source address, its better to make sure your files are in a folder instead of directly into root folder.

https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-simple-storage-service?tabs=data-factory#amazon-s3-as-a-source-type

https://bh3r1th.medium.com/adf-pipeline-to-copy-a-file-from-aws-s3-to-adls-gen2-aa696eea0ae7

Please Upvote and Accept as answer if the reply was helpful, this will be benefitting the other community members who go through the same issue.
Shuxin Yang (杨书新) 21 Reputation points

2023-05-30T06:55:03.1966667+00:00

Hi,

Thank you so much for replying.

I actually have read quite a lot of ADF documents, none of them ever mentioned the folder name restrictions of AWS S3, but I am pretty sure that the folder "/" works on AWS S3, and due to historical reasons I cannot change the folder name, does this mean I cannot make the requirement work on ADF?

Thanks again for your help!
Shuxin Yang (杨书新) 21 Reputation points

2023-05-30T07:00:07.5766667+00:00

Hi,

Thank you so much for replying.

I have read a lot of ADF related documents, have not seen any of them mentioned the folder name restrictions about AWS folder name. I'm pretty sure that the folder name "/" does work in AWS S3 buckets, due to historical reasons I cannot change the folder name, does this mean I cannot make this requirement work using ADF?

Thanks again for your help!

Your answer

Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2023-05-30T06:00:41.3866667+00:00

Hi,

Thanks for reaching out to Microsoft Q&A.

I dont think you can have only / as a source address, its better to make sure your files are in a folder instead of directly into root folder.

https://learn.microsoft.com/en-us/azure/data-factory/connector-amazon-simple-storage-service?tabs=data-factory#amazon-s3-as-a-source-type

https://bh3r1th.medium.com/adf-pipeline-to-copy-a-file-from-aws-s3-to-adls-gen2-aa696eea0ae7

Please Upvote and Accept as answer if the reply was helpful, this will be benefitting the other community members who go through the same issue.
Shuxin Yang (杨书新) 21 Reputation points

2023-05-30T06:55:03.1966667+00:00

Hi,

Thank you so much for replying.

I actually have read quite a lot of ADF documents, none of them ever mentioned the folder name restrictions of AWS S3, but I am pretty sure that the folder "/" works on AWS S3, and due to historical reasons I cannot change the folder name, does this mean I cannot make the requirement work on ADF?

Thanks again for your help!
Shuxin Yang (杨书新) 21 Reputation points

2023-05-30T07:00:07.5766667+00:00

Hi,

Thank you so much for replying.

I have read a lot of ADF related documents, have not seen any of them mentioned the folder name restrictions about AWS folder name. I'm pretty sure that the folder name "/" does work in AWS S3 buckets, due to historical reasons I cannot change the folder name, does this mean I cannot make this requirement work using ADF?

Thanks again for your help!

Share via

AWS S3 file cannot be found using ADF

Your answer