Long Running Experiment

Question

Long Running Experiment

Graham Benson 126

I am running a Machine Learning experiment in Azure Machine Learning Studio (not classic version). The total time of each step in the experiment is 7.5 min, however the overall execution time of the run was 20.5 min. Could someone please explain why there is such a large discrepancy between the total time the experiment took to run compared to the total of each step and what can be done to improve performance.

Thanks for your help.

Rohit Mungi 49,131 Reputation points Microsoft Employee Moderator

2021-11-02T08:29:04.013+00:00

@Graham Benson I believe you are using the studio to create the experiment. The newer version of the studio actually uses Azure VMs as compute based on the configuration selected for the cluster. The initial steps of an experiment usually setup compute and then run the steps or modules as part of your run. The older version of studio(classic) used dedicated compute which was faster but did not have the scaling capacity. In your case I think the experiment setup and compute needs to be reviewed since you are reporting every step took 7.5 mins and the overtime was 20.5 mins which is roughly only 3 steps for an experiment but I think your experiment might have more modules than 3.

Does this happen for every run of your experiment? I think it might be easier to report your case to check from backend from the ML portal ml.azure.com from the top right hand corner by choosing the smiley icon.
Graham Benson 126 Reputation points

2021-11-02T20:33:37.457+00:00

Thanks for your response romungi.

To confirm, yes I am using the Machine Learning Designer and not the classic version. In regards to the time taken to run the experiment, there are 8 steps which take a total time of > 7min. yet the overall run time of the experiment is ~ 20min. It varies across different runs. e.g the latest run of the experiment took 8.3 min yet the total time taken for each individual step was 4.3 min.

To give an example, there is an 'Apply SQL Transformation' module which is a simple select statement, "select * from t1 order by Interval asc limit 10;". On one particular run this took 45 sec. to complete on a table with < 2000 rows. On another occasion it took 0 sec to complete. The compute target I am using is Standard_DS3_v2 (4 cores, 14 GB RAM, 28 GB disk).

Is the performance a function of the VM specs or is there some other underlying issue?

I tried to log a request via the portal but do not have permissions to raise a support request.

Thanks.