CloudEvent to trigger Data Factory pipeline

Question

CloudEvent to trigger Data Factory pipeline

TOMA 0

Hey,

I have a question if it is possible to start a pipeline in synapse via custom trigger, but with an event that will be in CloudEvent format.

I was able to do it with an event in EventGrid format by following this article
https://medium.com/@ashish.kharbanda/triggering-azure-data-factory-pipeline-when-a-pipeline-execution-in-some-other-data-factory-2e3909e8dc10

However, the question is whether it is possible to do it with CloudEvent schema, when trying to create a trigger I get an error.

InternalServerError executing request

phemanth 15,855 Reputation points Microsoft External Staff Moderator

2024-09-23T17:11:04.7766667+00:00
@TOMA

Thanks for reaching out to Microsoft Q&A

Yes, it is possible to trigger an Azure Data Factory (ADF) or Azure Synapse pipeline using a custom event in the CloudEvent format.

However, there are some specific steps and configurations you need to follow to ensure it works correctly.

Set Up a Custom Topic in Event Grid:

First, you need to create a custom topic in Azure Event Grid. This topic will receive the CloudEvent messages.

Ensure your subscription is registered with the Event Grid resource provider.

Create a Custom Event Trigger in Data Factory:

Go to your Data Factory or Synapse workspace.

Navigate to the “Manage” section and select “Triggers”.

Create a new trigger and select “Custom events”.

Configure the trigger to listen to the custom topic you created in Event Grid.

Ensure CloudEvent Schema Compatibility:

Make sure your CloudEvent messages conform to the expected schema. The payload should include fields like id, source, type, time, and data.

Debugging Errors:

If you encounter an InternalServerError, check the following:

Ensure the Event Grid topic and subscription are correctly configured.

Verify that the CloudEvent message schema matches the expected format.

Check for any network or permission issues that might be preventing the trigger from executing.

For more detailed guidance, you can refer to the official Microsoft documentation on creating custom event triggers 1.

If you continue to face issues, share more details about the error message,
Sina Salam 26,661 Reputation points Volunteer Moderator

2024-09-23T18:02:05.27+00:00

Hello TOMA,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

It is possible to trigger an Azure Data Factory (ADF) or Azure Synapse pipeline using a custom event in the CloudEvent format.

To ensure your event payloads follow the Event Grid event schema, that include all necessary fields and configure the trigger to listen to the custom topic you created in Event Grid use this link: https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger and https://hevodata.com/learn/azure-data-factory-trigger

You can resolve the InternalServerError executing request by ensuring: Your Azure subscription is registered with the Event Grid resource provider with necessary permissions to create event subscriptions Microsoft.EventGrid/eventSubscriptions/ action https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-custom-event-trigger and check properly your event payload for any missing or incorrect fields as it shown in the links above.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.
phemanth 15,855 Reputation points Microsoft External Staff Moderator

2024-09-26T12:47:21.19+00:00

@TOMA We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

2 answers

Your answer

phemanth 15,855 Reputation points Microsoft External Staff Moderator

2024-09-23T17:11:04.7766667+00:00

@TOMA

Thanks for reaching out to Microsoft Q&A

Yes, it is possible to trigger an Azure Data Factory (ADF) or Azure Synapse pipeline using a custom event in the CloudEvent format.

However, there are some specific steps and configurations you need to follow to ensure it works correctly.

Set Up a Custom Topic in Event Grid:

First, you need to create a custom topic in Azure Event Grid. This topic will receive the CloudEvent messages.

Ensure your subscription is registered with the Event Grid resource provider.

Create a Custom Event Trigger in Data Factory:

Go to your Data Factory or Synapse workspace.

Navigate to the “Manage” section and select “Triggers”.

Create a new trigger and select “Custom events”.

Configure the trigger to listen to the custom topic you created in Event Grid.

Ensure CloudEvent Schema Compatibility:

Make sure your CloudEvent messages conform to the expected schema. The payload should include fields like id, source, type, time, and data.

Debugging Errors:

If you encounter an InternalServerError, check the following:

Ensure the Event Grid topic and subscription are correctly configured.

Verify that the CloudEvent message schema matches the expected format.

Check for any network or permission issues that might be preventing the trigger from executing.

For more detailed guidance, you can refer to the official Microsoft documentation on creating custom event triggers 1.

If you continue to face issues, share more details about the error message,
Sina Salam 26,661 Reputation points Volunteer Moderator

2024-09-23T18:02:05.27+00:00

Hello TOMA,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

It is possible to trigger an Azure Data Factory (ADF) or Azure Synapse pipeline using a custom event in the CloudEvent format.

To ensure your event payloads follow the Event Grid event schema, that include all necessary fields and configure the trigger to listen to the custom topic you created in Event Grid use this link: https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger and https://hevodata.com/learn/azure-data-factory-trigger

You can resolve the InternalServerError executing request by ensuring: Your Azure subscription is registered with the Event Grid resource provider with necessary permissions to create event subscriptions Microsoft.EventGrid/eventSubscriptions/ action https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-custom-event-trigger and check properly your event payload for any missing or incorrect fields as it shown in the links above.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.
phemanth 15,855 Reputation points Microsoft External Staff Moderator

2024-09-26T12:47:21.19+00:00

@TOMA We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Thanks for the answers, however, I followed what you described, as I created an event grid custom topic, where I set the input schema to CloudEventSchemaV1_0.

eventgrid1

I checked if the subscription is registered with resource provider.

eventgrid2 eventgrid3 eventgrid4

I created a custom trigger as below, selecting the appropriate event grid, eventgrid6

However, when trying to activate this trigger I get this message in Synapse

User's image

and in Event grid, I came across this error message

eventgrid5

The test message I send looks like the following, I get no error when sending.

{
    "specversion" : "1.0",
    "type" : "com.example.someevent",
    "source" : "/mycontext",
    "id" : "C234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "application/json",
    "data" : {
        "appinfoA" : "abc",
        "appinfoB" : 123,
        "appinfoC" : true
    }
}

Answer 2

@TOMA

please check the below steps

DataContentType: Try changing datacontenttype to datacontenttype with a value of application/cloudevents+json in your CloudEvent schema.
Event Grid Subscription: Double-check that your Event Grid subscription is correctly configured and that the subscription is active.
Network or Permission Issues: Verify that there are no network or permission issues that might be preventing the trigger from executing.

To further troubleshoot the issue, I recommend checking the following:

Check the Event Grid subscription logs to see if there are any additional error messages.
Verify that the Azure Synapse pipeline has the necessary permissions to receive events from the Event Grid topic.
Try sending a test event with a simplified payload to see if the issue persists.

If you're still encountering issues, please provide more details about the error messages,

Share via

CloudEvent to trigger Data Factory pipeline

2 answers

Your answer