Automation for Copying Files

Question

Hi,

I have an SFTP server running on AZ VM. I have a File Shares storage account that is used for that SFTP server. There are several SFTP partners that will push their files to that SFTP. Each sftp partner has its own subfolder to push their files.

Now,

• I need to copy those files from all those partner to another folder (in file File Shares) that is used to store all those files and run some workloads on that folder.
• I also need to move all the files from sftp partners to BLOB storage account that will be used as sftp files repository (Cool tier) and will have policy to move old files to Archive tier.

What Azure tool could be used for copying the files between folders / storage accounts? I've just came across DATA FACTORY that seems to be suitable - so will be playing with it these days. Is DATA FACTORY a way to go? ... or is there any better way to achieve the above?

Answer 1

Hi @michal ,
Thankyou for using Microsoft Q&A platform and posting your query .

As I went through your ask, it seems you want to know which is the best platform to use for the requirement of copying data from source like SFTP to storage account or copy data within the storage account. Please let me know if my understanding is incorrect.

You can use the Copy activity to copy data among data stores located on-premises and in the cloud. It is available in Azure Data Factory Pipelines as well as in Synapse pipelines. You can use anyone of these tools to meet your requirement.

Azure synapse just have additional capabilities of bring ADF pipeline feature (where you can create data ingestion pipelines and make use of data flow to perform data transformation) along with spark technology features within itself as there you can create notebooks and write custom codes in pyspark/scala/.Net spark/Spark SQL. You can make use of datawarehouse features using dedicated and serverless sql pools , you can integrate it with PowerBI to create reports.

To know more about Azure synapse analytics , please visit Azure syanpse overview

Coming back to your query, please visit the following resources to know what can be achieved using Copy activity in ADF or Synapse pieplines:

Copy activity in Azure Data Factory and Azure Synapse Analytics
Copy Data tool in Azure Data Factory and Synapse Analytics
https://www.youtube.com/watch?v=XOJeyRBXBos&t=509s
https://www.youtube.com/watch?v=cEfDGwHvzJE&t=258s

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Hello @michal

Copy Activity in Azure Data Factory can definitely be used to copy the files

You can set Triggers in a Data Factory pipeline which would trigger the job automatically and perform the necessary copy of the files