Disappearing files from Blob Storage

Question

Hi everyone.

I have recently set up an Azure IoT Hub that is accepting telemetry data from an MQTT source.

I have also set up an Azure Stream Analytics Job which simply takes the input and writes it to a Gen 2 Storage account (Data Lake, Hierarchy enabled).

The script is very simple :

SELECT
*
INTO
[tg-stream-output]
FROM
[tg-hub-stream-input]

The output format is {date}/{time} - so everything is as basic as it can get.

The events arrived without a problem and started writing to storage. The first folder was 04, and a file was created. Then on hour 5, a new folder was created and the stream continued writing. I downloaded the file to have a look at it and everything seemed fine.

Hour 6 came, new folder, new file. Same with hour 7,when the events stopped (which was expected).

I came back after some time to download everthing and folder 4 and 7 have data, but the files in 5 and 6 were missing. I had absolutely seen the files in there, but now they are missing completely.

Any help would be greatly appreciated, as I am stumped as to what could have happened.

Cheers.