Automation for Copying Files

michal 186 Reputation points
2022-04-24T20:00:46.067+00:00

Hi,

I have an SFTP server running on AZ VM. I have a File Shares storage account that is used for that SFTP server. There are several SFTP partners that will push their files to that SFTP. Each sftp partner has its own subfolder to push their files.

Now,

  • I need to copy those files from all those partner to another folder (in file File Shares) that is used to store all those files and run some workloads on that folder.
  • I also need to move all the files from sftp partners to BLOB storage account that will be used as sftp files repository (Cool tier) and will have policy to move old files to Archive tier.

What Azure tool could be used for copying the files between folders / storage accounts? I've just came across DATA FACTORY that seems to be suitable - so will be playing with it these days. Is DATA FACTORY a way to go? ... or is there any better way to achieve the above?

Azure Files
Azure Files
An Azure service that offers file shares in the cloud.
1,156 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,661 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,412 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,471 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. AnnuKumari-MSFT 30,361 Reputation points Microsoft Employee
    2022-04-26T06:46:21.137+00:00

    Hi @michal ,
    Thankyou for using Microsoft Q&A platform and posting your query .

    As I went through your ask, it seems you want to know which is the best platform to use for the requirement of copying data from source like SFTP to storage account or copy data within the storage account. Please let me know if my understanding is incorrect.

    You can use the Copy activity to copy data among data stores located on-premises and in the cloud. It is available in Azure Data Factory Pipelines as well as in Synapse pipelines. You can use anyone of these tools to meet your requirement.

    Azure synapse just have additional capabilities of bring ADF pipeline feature (where you can create data ingestion pipelines and make use of data flow to perform data transformation) along with spark technology features within itself as there you can create notebooks and write custom codes in pyspark/scala/.Net spark/Spark SQL. You can make use of datawarehouse features using dedicated and serverless sql pools , you can integrate it with PowerBI to create reports.

    To know more about Azure synapse analytics , please visit Azure syanpse overview

    Coming back to your query, please visit the following resources to know what can be achieved using Copy activity in ADF or Synapse pieplines:

    Copy activity in Azure Data Factory and Azure Synapse Analytics
    Copy Data tool in Azure Data Factory and Synapse Analytics
    https://www.youtube.com/watch?v=XOJeyRBXBos&t=509s
    https://www.youtube.com/watch?v=cEfDGwHvzJE&t=258s

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.
  2. Pratik Somaiya 4,201 Reputation points
    2022-04-25T04:41:56.803+00:00

    Hello @michal

    Copy Activity in Azure Data Factory can definitely be used to copy the files

    You can set Triggers in a Data Factory pipeline which would trigger the job automatically and perform the necessary copy of the files

    0 comments