Setting dependancy of Azure Data Factory with on prem scheduler

Utpal Brahma 1 Reputation point
2022-04-09T13:10:56.94+00:00

Hi all,

We schedule our jobs in on prem through a mainframe enterprise scheduler called ESP. ESP agents can not be installed in the Cloud (Azure). I want to set up dependency between an on prem job scheduled through ESP and Azure Data Factory pipeline.

I am new to Azure but had worked on AWS before.

What I had done earlier in AWS was the following and I was wondering if we can do the same in Azure, Your comments and guidance is welcome

  1. For AWS - we had installed AWS SDK (boto3) on a Linux server (where the ESP agent was installed) and from a Python Script with Boto3 routines we used to make a call to a Step Function or Glue module after retrieving the credentials (access key and secret key) from enterprise security vault (CyberArk) which is in Synch with AWS credentials. The Python code was called from a Shell Script which in turn was linked to the Scheduler. So upon the completion predecessor ESP job the shell script that initiates the AWS execution will be initiated.

Question : Is this something I can do in Azure ? That way I can set up the dependancy between on prem ESP and Azure Data Factory pipeline. If it can be done , what do I need to do ? Azure SDK installed on an on-prem server where ESP agent is installed ?

  1. The 2nd approach I am thinking is upon the completion of an on prem ESP job - send a trigger file over the an Azure storage location and set up an Azure Function that will detect that a file has landed (the filename will be mapped to the azure data factory pipeline name) and it will initiate the data pipeline execution. Do I need to use Event Hub for that (I could do with Lambda Function in Azure). Can I use Azure Gen2 storage for the destination where the trigger file will be sent ? We use a software called GoAnywhere to transfer files to Azure.

I am just 2 weeks into Azure and my new employer.

Any guidance will be most welcome.

Utpal

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,333 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
564 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,662 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Nandan Hegde 29,896 Reputation points MVP
    2022-04-11T03:40:52.17+00:00

    Hey,
    There are 2 way via which you can achieve it:

    1) ADF Event trigger :
    You can generate an event trigger file in blob which would auto trigger the ADF. Mo need for Azure functions etc
    https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory

    2) You can leverage ADF REST API to directly trigger the pipeline :
    https://medium.com/@rajbca00/execute-azure-adf-pipeline-using-rest-api-76d6cee5c473