Clarifying the watermark formula for stream analytics when no data is received

Question

The azure documentation for time handling in azure stream analytics states the following.

When there's no incoming event, the watermark is the current estimated arrival time minus the late arrival tolerance window. The estimated arrival time is the time that has elapsed from the last time an input event was seen plus that input event's arrival time.

Isn't is basically the same as 'current time - late arrival tolerance window'?

Answer 1

Hey Dhruv Singla,

Thankyou for reaching microsoft Q&A! At first glance the “estimated arrival time” formula can look more complicated than it really is. Let’s break it down:

1. Documentation says: estimatedArrival = lastEventArrivalTime + (now – lastEventArrivalWallClock) watermark (when no new events) = estimatedArrival – lateArrivalTolerance
2. But notice that lastEventArrivalTime + (now – lastEventArrivalWallClock) is mathematically equal to “now” (assuming system clocks stay in sync).
3. So when there’s truly no incoming event, “estimated arrival time” effectively becomes the current wall-clock time, and the watermark becomes: currentTime – lateArrivalTolerance

In other words, yes—it boils down to the same thing. The doc spells it out in that way so the service can reliably re-compute watermarks (e.g. after a failover or when you have multiple substreams), but in steady-state with no new data it’s just “now minus your late-arrival window.”

Hope that clears it up!

Reference list

• Time handling in Azure Stream Analytics (watermark definition) https://learn.microsoft.com/azure/stream-analytics/stream-analytics-time-handling#how-time-progresses-in-azure-stream-analytics
• Illustrated example of watermarks https://learn.microsoft.com/azure/stream-analytics/stream-analytics-time-handling#illustrated-example-of-watermarks
• Events dropped when arriving outside the allowed time policy https://docs.microsoft.com/azure/stream-analytics/stream-analytics-time-handling#events-dropped-when-arriving-outside-the-allowed-time-policy

Note: This content was drafted with the help of an AI system. Please verify the information before relying on it for decision-making.