amazon s3 access points data factory transfer

B K 41 Reputation points
2021-09-20T22:05:26.58+00:00

I'm trying to transfer data over from AmazonS3 to my Azure Data Lake. I'm currently given access to the s3 bucket via an access point.

https://aws.amazon.com/s3/features/access-points/

When I write python it looks like this:

AWSAccessKeyId='AKIAJQSDEGOZWTVPUETQ'
AWSSecretKey='xxxxx'

Bucket="arn:aws:s3:us-west-2:xxxxx:accesspoint/datafeed", 
Prefix="dplp-aea10a31/"

client = boto3.client('s3', aws_access_key_id=AWSAccessKeyId, aws_secret_access_key=AWSSecretKey, region_name='us-east-1' )

dict_objects = client.list_objects_v2(
  Bucket="arn:aws:s3:us-west-2:xxxxx:accesspoint/datafeed", 
  Prefix="dplp-aea10a31/"
)

The problem I have now is I have to download the file and read it. I'm looking to transfer it to my data lake to make it easier to deal with.

I don't see any way to do this thru Azure Synapse / Data Lake at first look.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,933 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 83,301 Reputation points Microsoft Employee
    2021-09-21T11:13:48.667+00:00

    Hello @B K ,

    Thanks for the question and using MS Q&A platform.

    There are couple of ways to access Amazon Simple Storage Service in Azure.


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators