Edit

User-defined functions in Azure Stream Analytics

  • Article

The SQL-like query language in Azure Stream Analytics makes it easy to implement real-time analytics logic on streaming data. Stream Analytics provides additional flexibility through custom functions that are invoked in your query. The following code example is a UDF called sampleFunction that accepts one parameter, each input record the job receives, and the result is written to the output as sampleResult.

SELECT 
    UDF.sampleFunction(InputStream) AS sampleResult 
INTO 
    output 
FROM 
    InputStream

Types of functions

Azure Stream Analytics supports the following four function types:

  • JavaScript user-defined functions
  • JavaScript user-defined aggregates
  • C# user-defined functions (using Visual Studio)
  • Azure Machine Learning

You can use these functions for scenarios such as real-time scoring using machine learning models, string manipulations, complex mathematical calculations, encoding and decoding data.

Important

C# user-defined functions for Azure Stream Analytics will be retired on September 30th 2024. After that date, it won't be possible to use the feature.

Limitations

User-defined functions are stateless, and the return value can only be a scalar value. You can't call out to external REST endpoints from these user-defined functions, as it will likely impact performance of your job.

Azure Stream Analytics doesn't keep a record of all functions invocations and returned results. To guarantee repeatability - for example, re-running your job from older timestamp produces the same results again - don't to use functions such as Date.GetData() or Math.random(), as these functions don't return the same result for each invocation.

Resource logs

Any runtime errors are considered fatal and are surfaced through activity and resource logs. It's recommended that your function handles all exceptions and errors and return a valid result to your query. This will prevent your job from going to a Failed state.

Exception handling

Any exception during data processing is considered a catastrophic failure when consuming data in Azure Stream Analytics. User-defined functions have a higher potential to throw exceptions and cause the processing to stop. To avoid this issue, use a try-catch block in JavaScript or C# to catch exceptions during code execution. Exceptions that are caught can be logged and treated without causing a system failure. You're encouraged to always wrap your custom code in a try-catch block to avoid throwing unexpected exceptions to the processing engine.

Next steps