Configuring the capture events on Azure Event Hub

Question

Configuring the capture events on Azure Event Hub

Mohamed Masood 1

We have a Azure Event hub to which we publish events using apache kafka protocol. We save the events as avro files by enabling Azure portal's capture events to do that on azure storage gen2. But the problem with this approach is we do not have control over the folder structure that the data is generated on. We want to drive the folder structure based on a particular field available in the event payload to drive our business needs to isolate the data.

Example payload:

{
"id": 1,
"folderName":1,
"body": {}
}

Now we want to drive the folder structure based on the "folder" in the event payload above to isolate data based on our business needs. Something like below:

{folderName}/{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}

1 answer

Your answer

Answer 1

PRADEEPCHEEKATLA 90,641 Moderator

Hello @Mohamed Masood ,

Thanks for the question and using MS Q&A platform.

Unfortunately, it is not possible to create a custom folder structure based on the payload.

Note: Microsoft only supports the below three capture file formats and we need to use any one of the them.

Hope this helps. Do let us know if you any further queries.

------------

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Mohamed Masood 1 Reputation point

2021-06-24T11:35:52.953+00:00

Is there any other utility/way we can consume events and save it as avro files by turning off the capture events?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-06-28T06:37:32.953+00:00

Hello @Mohamed Masood ,

Unfortunately, there is no way to consume events and save it as avro files by turning off the capture events.

Hope this helps.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-06-29T11:32:07.643+00:00

Hello @Mohamed Masood ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-07-01T09:29:00.487+00:00

Hello @Mohamed Masood ,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Mohamed Masood 1 Reputation point

2021-07-01T09:50:30.947+00:00

there is a way to do that perhaps, we can write a event hub consumer to consume the payload, convert it to avro by using json2avro utility and save to the datalake
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2021-07-02T09:59:32.957+00:00

Hello @Mohamed Masood ,

Glad to know that your issue has resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.

Share via

Configuring the capture events on Azure Event Hub

1 answer

Your answer