How do i know the reason about inconsistent execution times for the same code running in Azure ML Studio ?
I was trying to optimize my code using multiprocessing in python. I was successful as well in doing so, and the code which initially took 6 hours to run, was now brought down to 1hr 15mins. However that happened only on one particular day, and after that since the last two days it is again taking 5 hours to run in Azure ML studio. I am new to this platform and not able to find the reason for this inconsistency in time taken by the code to run. I have checked on the following points -
- No notifications or alerts from Azure regarding performance issue.
- Enough resources allocated to run my job.
- No network issue.
- No other workload is running concurrently.
I am attaching a part of my code here .. import time import multiprocessing as mp new_df_to_pred = pd.DataFrame() def process_data(i): The function running from 0 to 40000 if __name__ == "__main__": start_time = time.time() # Create a pool of workers num_workers = 4 pool = mp.Pool(num_workers) # Submit jobs to the pool results = pool.map(process_data, range(0, 1000, 1)) # Wait for all jobs to finish pool.close() pool.join() # Get the results from the jobs new_df_to_pred = pd.concat(results, ignore_index=True) end_time = time.time() execution_time = end_time - start_time
Can someone help me on this ?
@Aakash Verma Did you get a chance to check the logs of the job that ran faster and all other jobs that were taking longer? The job tab of the studio should provide the job id and the related logs.
What kind of compute are you using to run these jobs?
@romungi-MSFT i tried but for all the notebook runs i have made, no jobs are visible and hence no logs as well. I am simply running a python code(which is running from range of 1-40000, on multiprocessors) on Azure Notebook.
- can you tell me why the logs are not visible for me? whereas its visible for a colleague of mine, although he has also not created any pipeline or job.
- Once i see the log, may be will be able to see something
@Aakash Verma I think you are just using the notebook to run the above in jupyter notebook cells. Is that correct? In that case there will be no job created since you are not submitting any experiments to the Azure ML workspace.
I thought you are using the above snippet as part of an experiment, your colleague might be submitting experiments in his workspace which is creating jobs.
Coming back to the original issue, the notebook runs on a compute instance and uses a kernel that is selected. Are all the runs using the same settings?
@romungi-MSFT yes all the runs are using the same settings, running for the same user as well, and the code is exactly the same also. The only thing that differs is the time of run.
Hi team and @romungi-MSFT can you please provide some clarification on the issue ? its been a long wait.
Sign in to comment