Trigger azure Data Factory pipeline from elastic search index

Question

I have a pipeline on azure Data Factory. I need to trigger the pipeline automatically each time new data enters my elastic search index. Please how do I do this.

Answer 1

Hello @Confidence Sunday,

Welcome to the MS Q&A platform.

You can use the custom event trigger to trigger an Azure Data Factory pipeline automatically each time when new data enters your Elasticsearch index. Please follow the below steps.

• Create an event subscription on the Elastic Search index you want to monitor.
• Create a custom event trigger in your Azure Data Factory pipeline.
• Configure the event trigger to use the event subscription you created in step 1. Here, you will need to specify the name of the subscription, event grid topic
• Configure the trigger to filter the specific event you want to monitor. If you want to trigger the pipeline when new data is added to the Elastic Search index by specifying a filter based on the event type and data.
• Publish the pipeline and test the trigger.

I hope this helps. Please let us know if you have any further questions.