Create a custom event trigger to run a pipeline in Azure Data Factory

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Tip

Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!

Event-driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption, and reaction to events. Data integration scenarios often require Azure Data Factory customers to trigger pipelines when certain events occur. Data Factory native integration with Azure Event Grid now covers custom topics. You send events to an event grid topic. Data Factory subscribes to the topic, listens, and then triggers pipelines accordingly.

Note

The integration described in this article depends on Azure Event Grid. Make sure that your subscription is registered with the Event Grid resource provider. For more information, see Resource providers and types. You must be able to do the Microsoft.EventGrid/eventSubscriptions/ action. This action is part of the EventGrid EventSubscription Contributor built-in role.

Important

If you are using this feature in Azure Synapse Analytics, please ensure that your subscription is also registered with Data Factory resource provider, or otherwise you will get an error stating that the creation of an "Event Subscription" failed.

If you combine pipeline parameters and a custom event trigger, you can parse and reference custom data payloads in pipeline runs. Because the data field in a custom event payload is a free-form, JSON key-value structure, you can control event-driven pipeline runs.

Important

If a key referenced in parameterization is missing in the custom event payload, trigger run will fail. You'll get an error that states the expression cannot be evaluated because property keyName doesn't exist. In this case, no pipeline run will be triggered by the event.

Set up a custom topic in Event Grid

To use the custom event trigger in Data Factory, you need to first set up a custom topic in Event Grid.

Go to Azure Event Grid and create the topic yourself. For more information on how to create the custom topic, see Azure Event Grid portal tutorials and CLI tutorials.

Note

The workflow is different from Storage Event Trigger. Here, Data Factory doesn't set up the topic for you.

Data Factory expects events to follow the Event Grid event schema. Make sure event payloads have the following fields:

[
  {
    "topic": string,
    "subject": string,
    "id": string,
    "eventType": string,
    "eventTime": string,
    "data":{
      object-unique-to-each-publisher
    },
    "dataVersion": string,
    "metadataVersion": string
  }
]

Use Data Factory to create a custom event trigger

1. Go to Azure Data Factory and sign in.

2. Switch to the Edit tab. Look for the pencil icon.

3. Select Trigger on the menu and then select New/Edit.

4. On the Add Triggers page, select Choose trigger, and then select +New.

5. Select Custom events for Type.

   

6. Select your custom topic from the Azure subscription dropdown or manually enter the event topic scope.

   Note

   To create or modify a custom event trigger in Data Factory, you need to use an Azure account with appropriate role-based access control (Azure RBAC). No additional permission is required. The Data Factory service principal does not require special permission to your Event Grid. For more information about access control, see the Role-based access control section.

7. The Subject begins with and Subject ends with properties allow you to filter for trigger events. Both properties are optional.

8. Use + New to add Event Types to filter on. The list of custom event triggers uses an OR relationship. When a custom event with an eventType property that matches one on the list, a pipeline run is triggered. The event type is case insensitive. For example, in the following screenshot, the trigger matches all copycompleted or copysucceeded events that have a subject that begins with factories.

   

9. A custom event trigger can parse and send a custom data payload to your pipeline. You create the pipeline parameters, and then fill in the values on the Parameters page. Use the format @triggerBody().event.data._keyName_ to parse the data payload and pass values to the pipeline parameters.

   For a detailed explanation, see the following articles:

   • Reference trigger metadata in pipelines
   • System variables in custom event trigger

   

   

10. After you've entered the parameters, select OK.

Advanced filtering

Custom event trigger supports advanced filtering capabilities, similar to Event Grid Advanced Filtering. These conditional filters allow pipelines to trigger based upon the values of event payload. For instance, you may have a field in the event payload, named Department, and pipeline should only trigger if Department equals to Finance. You may also specify complex logic, such as date field in list [1, 2, 3, 4, 5], month field not in list [11, 12], tag field contains any of ['Fiscal Year 2021', 'FiscalYear2021', 'FY2021'].

As of today custom event trigger supports a subset of advanced filtering operators in Event Grid. Following filter conditions are supported:

• NumberIn
• NumberNotIn
• NumberLessThan
• NumberGreaterThan
• NumberLessThanOrEquals
• NumberGreaterThanOrEquals
• BoolEquals
• StringContains
• StringBeginsWith
• StringEndsWith
• StringIn
• StringNotIn

Select +New to add new filter conditions.

Additionally, custom event triggers obey the same limitations as Event Grid, including:

• 5 advanced filters and 25 filter values across all the filters per custom event trigger
• 512 characters per string value
• 5 values for in and not in operators
• keys cannot have . (dot) character in them, for example, john.doe@contoso.com. Currently, there's no support for escape characters in keys.
• The same key can be used in more than one filter.

Data Factory relies upon the latest GA version of Event Grid API. As new API versions get to GA stage, Data Factory will expand its support for more advanced filtering operators.

JSON schema

The following table provides an overview of the schema elements that are related to custom event triggers:

JSON element	Description	Type	Allowed values	Required
scope	The Azure Resource Manager resource ID of the Event Grid topic.	String	Azure Resource Manager ID	Yes
events	The type of events that cause this trigger to fire.	Array of strings		Yes, at least one value is expected.
subjectBeginsWith	The subject field must begin with the provided pattern for the trigger to fire. For example, factories only fire the trigger for event subjects that start with factories.	String		No
subjectEndsWith	The subject field must end with the provided pattern for the trigger to fire.	String		No
advancedFilters	List of JSON blobs, each specifying a filter condition. Each blob specifies key, operatorType, and values.	List of JSON blob		No

Role-based access control

Azure Data Factory uses Azure role-based access control (RBAC) to prohibit unauthorized access. To function properly, Data Factory requires access to:

• Listen to events.
• Subscribe to updates from events.
• Trigger pipelines linked to custom events.

To successfully create or update a custom event trigger, you need to sign in to Data Factory with an Azure account that has appropriate access. Otherwise, the operation will fail with an Access Denied error.

Data Factory doesn't require special permission to your Event Grid. You also do not need to assign special Azure RBAC role permission to the Data Factory service principal for the operation.

Specifically, you need Microsoft.EventGrid/EventSubscriptions/Write permission on /subscriptions/####/resourceGroups//####/providers/Microsoft.EventGrid/topics/someTopics.

• When authoring in the data factory (in the development environment for instance), the Azure account signed in needs to have the above permission
• When publishing through CI/CD, the account used to publish the ARM template into the testing or production factory needs to have the above permission.

Related content

• Get detailed information about trigger execution.
• Learn how to reference trigger metadata in pipeline runs.