When capturing data from Event Hubs in Parquet format, is there any way to preserve the event's offset or sequence number?

Michał Sobieraj 1 Reputation point
2022-11-18T14:41:29.187+00:00

When capturing an event from Event Hub to Avro format, we capture all it's metadata: offset, sequence number, properties, timestamp
However, when capturing it using stream analitycs to save into the parquet format, it looks like only the partitionID, Processed and Enqued timestamps are persisted.

Is there a way to persist information about the offset or sequence number when capturing to Parquet files?

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
660 questions
Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
366 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,351 Reputation points
    2022-11-28T10:17:10.5+00:00

    Hello @Michał Sobieraj ,

    As long as you are using an Event Hub as the input, you should be able to get the metadata (offset, sequence number, and others) using this function GetMetadataPropertyValue. It has nothing to do with Avro or Parquet output format. More detail can be found here: GetMetadataPropertyValue (Azure Stream Analytics) - Stream Analytics Query | Microsoft Learn

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators
    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.