Automated ML endpoint questions (performance, multiple return values, scoring)

Question

Automated ML endpoint questions (performance, multiple return values, scoring)

Adrian Pfisterer 1

Hi....

I used Automated ML to train a model on a set of grouping ids and titles, very simple use case, just two columns...predict the group id from the title. There were about 32000 rows in the input split into a training set of 90% and a validation set of 10%. Best model was 'MaxAbsScaler, LogisticRegression'.

I deployed the endpoint using the 'realtimeendpoint' method. But each request takes 10 seconds to return a response. I took the default ML compute type VM which isn't a wimpy machine. Is it normal to be so slow? Will better hardware get the response time into the sub-one-second time-frame I need it to be? Are there other options to improve performance?
I only ever get back one value in the response. Is it possible to get multiple predicted values?
I don't see a 'confidence' score in the response, or see a way to request one. Is that possible?

Thank you.

romungi-MSFT 49,096 Reputation points Microsoft Employee Moderator

2022-09-08T11:21:19.723+00:00

@Adrian Pfisterer For the first point, the response shouldn't really take 10 seconds for any scenario. This endpoint maybe having some performance issues. I think looking at your endpoint logs might give some insights but if you are seeing this slowness, I would recommend to report this through an Azure support case or through ML portal using the smiley icon on the top right-hand corner.

For the second question, yes, batch scoring endpoints can be used to get multiple predicted values. If you look at configuring the batch endpoints using the SDK or cli, it basically uses mini batches or sets of input data to predict on compute clusters and the output can be stored on the default workspace datastore.

For the final scenario, I think this depends on what is configured to be displayed as the output of your experiment. If you are able to view the experiment in the studio the edit and submit option helps to update your job, I think you could add the required output to be displayed with your endpoint.
Adrian Pfisterer 1 Reputation point

2022-09-08T17:40:11.303+00:00

Thank you. I could use some additional clarification.

For the response time and the suggestion to view the endpoint logs....how would I locate those? I only see an option in the UI to view the deployment logs.

For the 'multiple predicted values' question: my wording was ambiguous. I'm not looking to get one prediction back from multiple inputs in batch mode. I'm looking to get the top 'n' predicted values back for one request. For example, I'd like the top three predicted values back for my input of "lorem ipsum etc'.

For the 'confidence' score question: I don't see where in the UI I can configure additional outputs of a given experiment. I went thru the 'edit and submit' to update the job but don't see anything that looks like you can configure it for additional outputs. Do you have a screenshot?

Thank you.
romungi-MSFT 49,096 Reputation points Microsoft Employee Moderator

2022-09-09T09:07:04.203+00:00

The logs for endpoint needs to be enabled from the diagnostic settings. You will see this screen to enable the logs for an endpoint.

This document details the setup procedure and metrics that are available for monitoring an endpoint.

For prediction, most of the automated ML jobs have wrapper objects that mirror functionality from their open-source origin class. Classification model wrapper objects returned by automated ML implement the predict_proba() function. I am not too sure, if the predictions provide multiple values for a single request. I will have to check internally and confirm if this is possible.

I was referring to similar screen to update the output but I noticed automated ML jobs are not editable from the designer anymore.

Your answer

romungi-MSFT 49,096 Reputation points Microsoft Employee Moderator

2022-09-08T11:21:19.723+00:00

@Adrian Pfisterer For the first point, the response shouldn't really take 10 seconds for any scenario. This endpoint maybe having some performance issues. I think looking at your endpoint logs might give some insights but if you are seeing this slowness, I would recommend to report this through an Azure support case or through ML portal using the smiley icon on the top right-hand corner.

For the second question, yes, batch scoring endpoints can be used to get multiple predicted values. If you look at configuring the batch endpoints using the SDK or cli, it basically uses mini batches or sets of input data to predict on compute clusters and the output can be stored on the default workspace datastore.

For the final scenario, I think this depends on what is configured to be displayed as the output of your experiment. If you are able to view the experiment in the studio the edit and submit option helps to update your job, I think you could add the required output to be displayed with your endpoint.
Adrian Pfisterer 1 Reputation point

2022-09-08T17:40:11.303+00:00

Thank you. I could use some additional clarification.

For the response time and the suggestion to view the endpoint logs....how would I locate those? I only see an option in the UI to view the deployment logs.

For the 'multiple predicted values' question: my wording was ambiguous. I'm not looking to get one prediction back from multiple inputs in batch mode. I'm looking to get the top 'n' predicted values back for one request. For example, I'd like the top three predicted values back for my input of "lorem ipsum etc'.

For the 'confidence' score question: I don't see where in the UI I can configure additional outputs of a given experiment. I went thru the 'edit and submit' to update the job but don't see anything that looks like you can configure it for additional outputs. Do you have a screenshot?

Thank you.
romungi-MSFT 49,096 Reputation points Microsoft Employee Moderator

2022-09-09T09:07:04.203+00:00

The logs for endpoint needs to be enabled from the diagnostic settings. You will see this screen to enable the logs for an endpoint.

This document details the setup procedure and metrics that are available for monitoring an endpoint.

For prediction, most of the automated ML jobs have wrapper objects that mirror functionality from their open-source origin class. Classification model wrapper objects returned by automated ML implement the predict_proba() function. I am not too sure, if the predictions provide multiple values for a single request. I will have to check internally and confirm if this is possible.

I was referring to similar screen to update the output but I noticed automated ML jobs are not editable from the designer anymore.

Share via

Automated ML endpoint questions (performance, multiple return values, scoring)

Your answer