Azure Stream Analytics: ML Service function call in cloud job results in no output events

Question

Hey,
I've got a problem with an Azure Stream Analytics (ASA) job that should call an Azure ML Service function to score the provided input data.
The query was developed und tested in Visual Studio (VS) 2019 with the "Azure Data Lake and Stream Analytics Tools" Extension.
As input the job uses an Azure IoT-Hub and as output the VS local output for testing purposes (and later even with Blobstorage).
Within this environment everything works fine, the call to the ML Service function is successfull and it returns the desired response.
Using the same query, user-defined functions and aggregates like in VS in the cloud job, no output events are generated (with neither Blobstorage nor Power BI as output).
In the ML Webservice it can be seen, that ASA successfully calls the function, but somehow does not return any response data.
Deleting the ML function call from the query results in a successfull run of the job with output events.

For the deployment of the ML Webservice I tried the following (working for VS, no output in cloud):

• ACI (1 CPU, 1 GB RAM)
• AKS dev/test (Standard_B2s VM)
• AKS production (Standard_D3_v2 VM)

The inference script function schema:

• input: array
• output: record

The ASA job subquery with ML function call:

"Sequence" is a subquery that aggregates the data into sequences (arrays) with an user-defined aggragate.

I hope the provided information is sufficient and you can help me.

Answer 1

Hi, can you share a link to your swagger? This is a public link that can be accessed without any keys.

Few questions:

1. In VS 2019, did you test using live data flowing in from your IoT Hub? Or did you use a sample input file/local input?
2. You mention that the input is an array, however the sample input says [[1.]*18]. I am trying to understand what this represents. And what does numpyfySeq UDF do?
3. Can you open the ml UDF on the portal and see if the "function signature" is listed correctly?

Answer 2

Thank you for your reply.

Yes sure, here it is:
swagger.json

To your questions:

1. In VS i was using live data from the IoT-Hub.
2. The numpyfySeq UDF looks like this:
3. List item

It uses the aggregates created in the "Sequence" subquery and puts them all together in a N x 18 size array.
The "Sequence" subquery uses also an UDA ("stackSequence") to aggregate data into arrays of arbitrary length (all occured events in a hopping window with width N seconds and hop size 1 second should be appended to an array).
Part of the "Sequence" subquery:

The used schema [[1.]*18] creates a reduced form (1 x 18 shape) of the desired input array shape and with the optional parameter "enforce_shape=False" in the decorator i could manage to call the function with an abitrary N x 18 shape array.

3. On the portal the function signature is listed correctly:
   

Answer 3

Thanks for the details. Everything seems like it is configured correctly. And from what you say, it sounds like this scoring works well when using VS but not when job is running? One last thing to check is maybe the event ordering settings of your job on the portal. When job is running, you can also see the metric "late events" to see if these inputs are getting filtered out.

In this case it will require additional debugging from our team. Can you please file a support ticket and reference this thread?

Answer 4

Yes, this is correct, in VS the scoring works well and works both with local and blobstorage output and when the job is running, the function is called successfully, but no output is generated.
I checked the metric "late input events" within a period of the last 30 days and the rate is very low (0.0003 %).

I will file a support ticket then.
Thanks for your help so far.