Stream Analytics Output to BlobStorage Folder Structure and Time

James "JC" Clark 1 Reputation point
2021-03-31T16:43:10.957+00:00

We have a stream analytics job that is simply processing json files and passing them into cosmos or to another event hub. On the event hub we have capture events to make avro files for later consumption. We are having an issue there and have thought to use stream analytics and blob storage output and simply make the json files ourselves. However we are running into a problem/concern and thought to bring here before a ticket is made.

Path Pattern: {date}/{time}/asa-piiCleaner/{MachineId}
Date Format: YYYY/MM/DD
Time Format: HH/mm
Format: Array
Minimum rows: 1
Maximum time: 1 minute

Given this setup I was expecting to see in blob storage something akin to the following
2021/03/31/16/00/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile
2021/03/31/16/01/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile
2021/03/31/16/02/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile
2021/03/31/16/03/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile

I have yet to run this for a full hour however at this time not seeing what we are expecting instead
2021/03/31/16/00/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile

is being created and that same file is being updated for now going on 10 minutes. Everything else feels like its working as designed except that the "mm" column is not changing

I did this right at the bottom of an hour so was able to see it change from 03/31/15 to 03/31/16. However the folder "00" seems to stay constant

Also anyone know how to just take the stream and make individual files?

Thoughts?

Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
330 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Samara Soucy - MSFT 5,051 Reputation points
    2021-04-01T18:12:35.907+00:00

    Currently only hours are supported in the default time format, producing the result you are seeing. However, setting your path pattern to {date}/{datetime:HH}/{datetime:mm}/asa-piiCleaner/{MachineId} should create the results you are expecting.

  2. JS Azure 76 Reputation points
    2021-04-02T16:43:37.663+00:00

    Hi, can you confirm you have outputs going at the minute level (e.g. if you have a 10 minute tumbling windows, you will have output only every 10 minutes).
    Also what is your out-of-order and late arrival settings in the settings? Out-of-order add some time before the first output, and late arrival events may adjust the events to another folder. You can also see if you have late events in the metrics.
    Thanks,
    JS (Stream Analytics)