Why did scaling not help on delayed Stream Analytics outputs?

Question

Hello :)

We have an Azure Function that sends a lot of data in a single format to an Event Hub, which sends it to a Stream Analytics, which forwards the data to multiple outputs.

This have worked very good for a long time, but a couple of days ago I noticed that one of the outputs were pretty much stopping generating data (not all of them, only one). I took a quick look at the Stream Analytics and saw that Watermark Delay and Backlogged Input Events were peaking up.

Fast forward, I investigated the metrics further and in the graph for Watermark Delay, I found that by splitting on Logical Name, the output function that was supposed to generate the missing data really stood out. I further started logging the Azure Function, and the runtime of it was on average around 45 seconds. For the record, this is much more than expected. But I figured I would do the safe bet and just increase the Streaming Units of the Stream Analytics and Scale Out the App Service Plan that the Azure function was running on, and then start improving on the actual code. But to my despair, scaling out did not really solve my problem. Restarting the Stream Analytics would make everything work for a while, but suddenly the Watermark Delay and Backlogged Input Events would spike again, and the output data will be missing. There are no errors whatsoever, and the function is still running.

My theory right now is that the Stream Analytics keeps pushing the same batch of data over and over to the output, and I am gonna investigate this thought. But can someone read out any obvious issues from what I am experiencing? It is a bit concerning to me that this would happen out of nowhere and also that the scaling does not work as we will at some point get a lot more data than today. I have temporarily solved my problem by disabling some of the not critical code in my Azure Function to make it run faster, and this seems to work.