Copy last added file to a GCS bucket into Azure Blob storage

Question

Hi all,
I'm VERY new to Azure Data Factory, so pardon me for my stupid or obvious question.

I want to schedule the copy files stored in a GCS bucket in Azure Blob Storage once a day. Until now, I managed to copy (both manually and by scheduling the activity of the pipeline) file from the bucket in GCS where I'm uploading the file manually.

In the near future, the upload will happen automatically once a day at a given time,
presumably during the night. My goal is to schedule the copy of just the last added file and avoid copying every time all the file,
overwriting the existing ones.

It's something that requires writing some python script? Is there some parameter to set?

Thank you all in advance for the replies.

Answer 1

Hi @Gabriele Sciurti ,

Thanks for posting your question in Microsoft Q&A portal and for using Azure Services.

As per my understanding, you want to Load last modified file from Google Cloud Storage bucket to Azure Blob Storage once a day.
Please correct me if my understanding is not complying to your ask.

In Azure Data Factory, if we want to accomplish this task, we should first connect to GCS by creating Linked service to Google Cloud Storage (S3 API) connector. Refer this page for detailed settings.

Next, we need to perform load using ADF activities: Get Metadata, ForEach, Copy, Set Variable.
For detailed overview, please refer to this page.

After creation of Pipeline, we will schedule it to run everyday using Schedule Trigger:

Detailed description of creation of Schedule Trigger is present at this link.

Hope the above information will help. Please let us know if you are able to execute all steps successfully.
In case of any blocker, please share your ask we are happy to assist.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Similar thread :
https://stackoverflow.com/questions/73502790/copy-last-added-file-to-a-gcs-bucket-into-azure-blob-storage/73505550#73505550