How to automate triggering of copy pipeline (On-Prem to ADLS in cloud ) once file is available on-prem

Question

Hi,

We have use case where we would like to start copy activity (from on-prem to cloud) when file is available on-prem. As we don't know when the file will be available on-prem we don't want to trigger pipeline on certain time window. We want to keep file based trigger where once files arrived at on-prem folder , ADF pipeline should be trigger in automated way and copy data from on-prem to cloud.

Answer 1

Hello @Joshi, Prasad-ZA-XT ,

Thanks for the query and using Microsoft Q&A platform.

Unfortunately, ADF doesn’t support event-based triggering of copy pipeline from on-premises systems.

Note: The Storage Event Trigger currently supports only Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts.

I would suggest you to provide feedback on the same:

https://feedback.azure.com/forums/270578-data-factory

All of the feedback you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure.

Other alternate solutions:

Option1: Azure Automation has the ability to integrate and automate processes across Azure and on-premises environments using a hybrid worker. This hybrid management capability has been extended to now deliver an automatic response to events in your datacenter using watcher tasks.

Watcher tasks deliver the ability to author a watcher runbook that polls a system in your environment and then call an action runbook to process any events found. Typical scenarios that this new functionality is used for are:

• Look for new files that arrive in a folder and take action based on the content of the files.

For more details, refer Track updated files with a watcher task.

Option2: Since the files are on-premise you can use PowerShell to monitor a folder for new files. Then fire an event to upload the file to an Azure blob.

There is a video showing how to do this here: https://www.youtube.com/watch?v=Usih7UywZYA

The changes you need to make are:

• Replace the action with an upload to azure https://argonsys.com/microsoft-cloud/library/how-to-upload-files-to-azure-blob-storage-using-powershell-and-azcopy/
• Run powershell in the context of a user that can upload file.

Hope this helps. Do let us know if you any further queries.

------------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.