How do i know the reason about inconsistent execution times for the same code running in Azure ML Studio ?

Aakash Verma 0 Reputation points
I was trying to optimize my code using multiprocessing in python. I was successful as well in doing so, and the code which initially took 6 hours to run, was now brought down to 1hr 15mins. However that happened only on one particular day, and after that since the last two days it is again taking 5 hours to run in Azure ML studio. I am new to this platform and not able to find the reason for this inconsistency in time taken by the code to run. I have checked on the following points  - 
  1. No notifications or alerts from Azure regarding performance issue.
  2. Enough resources allocated to run my job.
  3. No network issue.
  4. No other workload is running concurrently.
I am attaching a part of my code here ..

import time
import multiprocessing as mp
new_df_to_pred = pd.DataFrame()

def process_data(i):
    The function running from 0 to 40000

if __name__ == "__main__":
    start_time = time.time()

    # Create a pool of workers
    num_workers = 4
    pool = mp.Pool(num_workers)

    # Submit jobs to the pool
    results =, range(0, 1000, 1))

    # Wait for all jobs to finish

    # Get the results from the jobs
    new_df_to_pred = pd.concat(results, ignore_index=True)

    end_time = time.time()

    execution_time = end_time - start_time

Can someone help me on this ?

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
1,893 questions
{count} votes