When capturing data from Event Hubs in Parquet format, is there any way to preserve the event's offset or sequence number?

Question

When capturing an event from Event Hub to Avro format, we capture all it's metadata: offset, sequence number, properties, timestamp
However, when capturing it using stream analitycs to save into the parquet format, it looks like only the partitionID, Processed and Enqued timestamps are persisted.

Is there a way to persist information about the offset or sequence number when capturing to Parquet files?

Answer 1

Hello @Michał Sobieraj ,

As long as you are using an Event Hub as the input, you should be able to get the metadata (offset, sequence number, and others) using this function GetMetadataPropertyValue. It has nothing to do with Avro or Parquet output format. More detail can be found here: GetMetadataPropertyValue (Azure Stream Analytics) - Stream Analytics Query | Microsoft Learn

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators