How to recover Stream Analytics job to normal Running status from Degraded state?

Question

Hello, My stream analytics job had been working fine with IoT Hub stream input and SQL DB as reference input, output is written to storage account. Yesterday there was an unplanned Azure service issue with IoT Hub in my region and it lasted for around 6 hours. Before this being noticed, I was testing my Stream analytics Queries and restarting the job etc.. I observed the Stream Analytics had completely gone to a 'Degraded' Status - its around more than 6 hours now, and I see this is after the same time IoT Hub unavailability issue was reported.
MS mitigated the service issue with IoT Hub in the late evening hours, however I'm unable to restore back the Stream analytics job to normal running state - I've tried many restarts, but no luck. Note: I already tried increasing the Streaming Units, enable diagnostics logs, but Job doesn't seem in execution mode and there are no metrics available on the dashboards - so no idea what's going on.. Any suggestions/help how to handle such situations better?

Answer 1

Degraded job status incident was resolved last friday.. One of the key learnings is around how ASA Jobs use Table storage service for Checkpoint/Offset, Last Run timestamp - As long as we ensure ASA job is able to connect and operate with storage service, Job gets to start normally..

Another key learning is around Time handling - When there are sparse input events (lets say hourly) ASA Job continuous to poll and process events when it arrives.. During any downtimes or health events, the ASA job restarts - It appears to me, the Job replays all the old events, instead of resuming to process only the new events.

Further I want to check is there a better way to handle this in query to process only latest events, whenever Job resumes after some hours (due to platform downtimes/upgrades).

Answer 2

@Vishwanath Hariharan - Thanks for the question and using MS Q&A platform.

When a Stream Analytics job enters a degraded state, it means that there might be intermittent issues with your input and output connections. These errors are called transient errors that might make your job enter a degraded state. Stream Analytics will immediately try to recover from such errors and return to a Running state (within few minutes). These errors could happen due to network issues, availability of other Azure resources, deserialization errors, etc. Your job's performance may be impacted when the job is in a degraded state.

You can look at the diagnostic or activity logs to learn more about the cause of these transient errors. In cases such as deserialization errors, it's recommended to take corrective action to ensure events aren't malformed. If the job keeps reaching the resource utilization limit, try to increase the SU number or parallelize your job. In other cases where you can't take any action, Stream Analytics will try to recover to a Running state. You can use watermark delay metric to understand if these transient errors are impacting your job's performance.

In your case, since you have already tried increasing the Streaming Units and enabling diagnostic logs, I would suggest you to look at the diagnostic or activity logs to identify the root cause and address the issue. You can also configure alerts so that you get notified when the job goes to a Failed state. Additionally, you can try restarting your Stream Analytics job from the last output time.

Here is the recommended actions for the Azure Stream Analytics job state degraded:

For more details, refer to Azure Stream Analytics job states.

If none of these steps work, you may open a support ticket for further assistance.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.