Schedule machine learning pipeline jobs

APPLIES TO: Azure CLI ml extension v2 (current) Python SDK azure-ai-ml v2 (current)

In this article, you learn how to schedule machine learning pipelines to run on Azure. You can schedule routine tasks like retraining models or regularly updating batch predictions based on elapsed time.

This article shows you how to create, retrieve, update, and deactivate schedules by using the Azure Machine Learning CLI, Azure Machine Learning SDK v2 for Python, or Azure Machine Learning studio UI.

Tip

To schedule jobs by using an external orchestrator, like Azure Data Factory or Microsoft Fabric, consider deploying your pipeline jobs under a batch endpoint. For more information, see Deploy existing pipeline jobs to batch endpoints and Run Azure Machine Learning models from Fabric by using batch endpoints (preview).

Prerequisites

• An Azure subscription. If you don't have one, create a free account before you begin.
• An Azure Machine Learning workspace. To create a workspace, see Create workspace resources.
• An understanding of Azure Machine Learning pipelines. For information, see What are machine learning pipelines.

Azure CLI

• The Azure CLI and ml extension installed by following the instructions in Install, set up, and use the CLI (v2).
• Knowledge of how to create Azure Machine Learning YAML pipelines. For information, see Create and run machine learning pipelines using components with the Azure Machine Learning CLI.

Python SDK

• The Azure Machine Learning SDK v2 for Python installed.
• Knowledge of how to create Python pipelines for Azure Machine Learning. For information, see Create and run machine learning pipelines using components with the Azure Machine Learning SDK.

Studio UI

• Knowledge of how to create pipelines in Azure Machine Learning studio. For information, see Create and run machine learning pipelines using components with the Azure Machine Learning studio.

Limitations

• Azure Machine Learning v2 schedules don't support event-based triggers.
• CLI and SDK v2 schedules support specifying complex recurrence patterns that contain multiple trigger timestamps. The studio UI displays the complex patterns but doesn't support editing them.
• The studio UI supports only v2 schedules, and can't list or access v1 schedules that are based on published pipelines or pipeline endpoints. You can create a schedule for an unpublished pipeline.
• If recurrence is set as the 31st or 30th day of every month, the schedule doesn't trigger jobs in months that have fewer days.
• DAYS and MONTHS values aren't supported in cron schedule expressions. Values passed for these parameters are ignored and treated as *.

Create a schedule

When you have a pipeline job with satisfying performance and outputs, you can set up a schedule to automatically trigger the job on a regular basis. To do so, you must create a schedule that associates the job with a trigger. The trigger can be either a recurrence pattern or a cron expression that specifies the interval and frequency to run the job.

In both cases, you need to define a pipeline job first, either inline or by specifying an existing pipeline job. You can define pipelines in YAML and run them from the CLI, author pipelines inline in Python, or compose pipelines in Azure Machine Learning studio. You can create pipeline jobs locally or from existing jobs in the workspace.

You can create v2 schedules for v2 or v1 pipeline jobs by using the studio UI, SDK v2, or CLI v2. You don't have to publish existing pipelines first to set up schedules for pipeline jobs.

Azure CLI

The code examples in this article are from Working with Schedule in Azure Machine Learning CLI 2.0.

Python SDK

The code examples in this article are excerpts from the Working with Schedule Azure Machine Learning notebook. Run the notebook to create and manage the schedules as described.

Studio UI

To create a schedule for an Azure Machine Learning pipeline job in the studio UI, open the pipeline job detail page.

Define a time-based schedule with a recurrence pattern

Azure CLI

The following YAML code defines a recurring schedule for a pipeline job. The required type parameter specifies that the trigger type is recurrence.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_recurrence_job_schedule
display_name: Simple recurrence job schedule
description: a simple hourly recurrence job schedule

trigger:
  type: recurrence
  frequency: day #can be minute, hour, day, week, month
  interval: 1 #every day
  schedule:
    hours: [4,5,10,11,12]
    minutes: [0,30]
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

create_job: ./simple-pipeline-job.yml
# create_job: azureml:simple-pipeline-job

You must or can provide the following schedule parameters:

Python SDK

The following code uses RecurrenceTrigger to provide a better coding experience.

schedule_name = "simple_sdk_create_schedule_recurrence"

schedule_start_time = datetime.utcnow()
recurrence_trigger = RecurrenceTrigger(
    frequency="day",
    interval=1,
    schedule=RecurrencePattern(hours=10, minutes=[0, 1]),
    start_time=schedule_start_time,
    time_zone=TimeZone.UTC,
)

job_schedule = JobSchedule(
    name=schedule_name, trigger=recurrence_trigger, create_job=pipeline_job
)

You must or can provide the following schedule parameters:

Studio UI

To open the schedule creation wizard, select Schedule > Create new schedule at the top of the page.

Azure CLI / Python SDK

Parameters

• frequency (required) is the time unit on which basis the schedule fires. Can be minutes, hours, days, weeks, or months.
• interval (required) is the number of time units between schedule recurrences.
• schedule (optional) defines the recurrence pattern, which can contain hours, minutes, and weekdays. If omitted, jobs trigger according to the logic of start_time, frequency, and interval.
  • When frequency is day, the pattern can specify hours and minutes.
  • When frequency is week or month, the pattern can specify hours, minutes, and weekdays.
    • hours is an integer or list from 0 to 23.
    • minutes is an integer or list from 0 to 59.
    • weekdays is a string or list from monday to sunday.
• start_time (optional) is the start date and time with timezone. If omitted, the default is equal to schedule creation time. If the start time is in the past, the first job runs at the next calculated run time.
• end_time (optional) is the end date and time with timezone. If omitted, the schedule remains active until manually disabled.
• time_zone (optional) specifies the time zone of the recurrence schedule. If omitted, the default is Coordinated Universal Time (UTC). For more information about timezone values, see the appendix for timezone values.

Studio UI

To define a recurrence-based schedule, on the Basic settings screen, define the following properties. Only the Name property requires you to enter a value. If you don't specify values for the other properties, the default schedule is a Recurrence pattern that starts at schedule creation and runs every Monday through Friday at 4:00 PM UTC.

• Name: Unique identifier of the schedule within the workspace.
• Description: Schedule description.
• Trigger: Recurrence pattern of the schedule, including the following properties:
  • Time zone: Time zone to use for the trigger time, UTC by default.
  • Select Recurrence and specify a recurring pattern of minutes, hours, days, weeks, or months.
  • Start: Date the schedule becomes active, by default the date created.
  • End: Date when the schedule becomes inactive. By default the value is none, and the schedule remains active until manually disabled.
  • Tags: Tags on the schedule.

Azure CLI

After you create the schedule YAML, use the following command to create the schedule via CLI:

# This action creates related resources for a schedule. It takes dozens of seconds to complete.
az ml schedule create --file simple-pipeline-job.yml --no-wait

Python SDK

The following Python code creates the schedule you defined:

job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Studio UI

After you configure the basic settings, select Review + Create, review the settings, and then select Review + Create again to create the schedule.

Define a time-based schedule with a cron expression

A cron expression can specify a flexible and customized recurrence pattern for a schedule. A standard crontab expression is composed of the space-delimited fields MINUTES HOURS DAYS MONTHS DAYS-OF-WEEK. A wildcard * means all values for a field.

In an Azure Machine Language schedule cron expression:

• MINUTES is an integer or list from 0 to 59.
• HOURS is an integer or list from 0 to 23.
• DAYS values aren't supported, and are always treated as *. The * value in DAYS means all days in a month, which varies with month and year.
• MONTHS values aren't supported, and are always treated as *.
• DAYS-OF-WEEK is an integer or list from 0 to 6, where 0 = Sunday. Names of days are also accepted.

For example, the expression 15 16 * * 1 means 4:15 PM UTC every Monday. For more information about crontab expressions, see the Crontab Expression wiki on GitHub.

Azure CLI

The following YAML code defines a recurring schedule for a pipeline job. The required type parameter specifies that the trigger type is cron.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_cron_job_schedule
display_name: Simple cron job schedule
description: a simple hourly cron job schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

# create_job: azureml:simple-pipeline-job
create_job: ./simple-pipeline-job.yml

You must or can provide the following schedule parameters:

Python SDK

The following code uses CronTrigger to provide a better coding experience.

schedule_name = "simple_sdk_create_schedule_cron"

schedule_start_time = datetime.utcnow()
cron_trigger = CronTrigger(
    expression="15 10 * * *",
    start_time=schedule_start_time,  # start time
    time_zone="Eastern Standard Time",  # time zone of expression
)

job_schedule = JobSchedule(
    name=schedule_name, trigger=cron_trigger, create_job=pipeline_job
)

You must or can provide the following schedule parameters:

Studio UI

To define a cron-based schedule, select Cron expression instead of Recurrence on the Basic settings screen for the schedule.

Azure CLI / Python SDK

Parameters

• expression (required) is a standard crontab expression that expresses a recurring schedule.
• start_time (optional) is the schedule start date and time with timezone. For example, start_time: "2022-05-10T10:15:00-04:00" means the schedule starts from 10:15:00 AM on May 10, 2022 in UTC-4 timezone. If omitted, the default is equal to schedule creation time. If the start time is in the past, the first job runs at the next calculated run time.
• end_time (optional) is the end date and time with timezone. If omitted, the schedule remains active until manually disabled.
• time_zone (optional) specifies the time zone of the recurrence schedule. If omitted, the default is UTC.

Studio UI

Only the Name property requires you to enter a value. If you don't specify a cron expression, the default cron expression creates a schedule that runs daily at 4:00 PM UTC.

• Name: Unique identifier of the schedule within the workspace.
• Description: Description of the schedule.
• Trigger: Recurrence pattern of the schedule, including the following properties:
  • Time zone: Time zone to use for the trigger time, Coordinated Universal Time (UTC) by default.
  • Select Cron expression and provide a standard crontab expression that expresses a recurring schedule.
  • Start: Date the schedule becomes active, by default the date created.
  • End: Date the schedule becomes inactive. By default the value is none, and a schedule remains active until you manually disable it.
  • Tags: Tags on the schedule.

Azure CLI

After you create the schedule YAML, use the following command to create the schedule via CLI:

# This action creates related resources for a schedule. It takes dozens of seconds to complete.
az ml schedule create --file simple-pipeline-job.yml --no-wait

Python SDK

The following Python code creates the schedule you defined:

job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Studio UI

After you configure the basic settings, select Review + Create, review the settings, and then select Review + Create again to create the schedule.

Change job settings when you define schedules

Sometimes you might want the jobs triggered by schedules to have different configurations from the test jobs. When you define a schedule by using an existing job, you can change the job settings. This approach lets you define multiple schedules that use the same job with different inputs.

Azure CLI

When you define a schedule, you can change the settings, inputs, or outputs to use when running the pipeline job. You can also change the experiment_name of the triggered job.

The following schedule definition changes the settings of an existing job.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: cron_with_settings_job_schedule
display_name: Simple cron job schedule
description: a simple hourly cron job schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

create_job: 
  type: pipeline
  job: ./simple-pipeline-job.yml
  # job: azureml:simple-pipeline-job
  # runtime settings
  settings:
    #default_compute: azureml:cpu-cluster
    continue_on_step_failure: true
  inputs:
    hello_string_top_level_input: ${{name}} 
  tags: 
    schedule: cron_with_settings_schedule

Python SDK

When you define a schedule, you can change the settings, inputs, or outputs to use when running the pipeline job. You can also change the experiment_name of the triggered job.

The following schedule definition changes the settings of a pipeline per job.

# set run time settings
pipeline_job = pipeline_with_components_from_yaml(
    training_input=Input(type="uri_folder", path=parent_dir + "/data/"),
    test_input=Input(type="uri_folder", path=parent_dir + "/data/"),
    training_max_epochs=20,
    training_learning_rate=1.8,
    learning_rate_schedule="time-based",
)

# set pipeline level compute
pipeline_job.settings.default_compute = "cpu-cluster"

Studio UI

In the studio UI, you can use Advanced settings in the schedule creation wizard to modify inputs, outputs, and runtime settings for a pipeline job. You can't change the experiment_name in the studio UI.

1. In Job inputs & outputs, you can modify inputs and outputs for future jobs triggered by the schedule. You can use macro expressions for the inputs and outputs paths.

   

2. In Job runtime settings, you can modify compute and other runtime settings for jobs triggered by the schedule.

   

3. Select Review + Create to review the schedule settings you configured, and then select Review + Create again to create the schedule.

   

Use supported expressions in schedules

When you define a schedule, you can use the following macro expressions to define dynamic parameter values that resolve to actual values during job runtime.

Expression	Description	Supported properties
${{name}}	Name of the job	outputs path of the pipeline job
${{creation_context.trigger_time}}	Trigger time of the job	String type inputs of the pipeline job

Manage schedule

You can list, view details, update, disable, enable, and delete schedules in a workspace.

List schedules

Azure CLI

az ml schedule list

Python SDK

schedules = ml_client.schedules.list()
[s.name for s in schedules]

Studio UI

Under Jobs in the studio, select the All schedules tab to see a list of all the v2 job schedules created by SDK, CLI, or studio UI in this workspace.

View schedule details

Azure CLI

az ml schedule show -n simple_cron_job_schedule

Python SDK

created_schedule = ml_client.schedules.get(name=schedule_name)
[created_schedule.name]

Studio UI

Select a schedule name to show the schedule detail page, which contains the following tabs:

• Overview provides basic information about the schedule.

  

• Job definition defines the job triggered by the schedule.

  

• Jobs history provides a list of all jobs triggered by the schedule.

  

Update a schedule

Azure CLI

az ml schedule update -n simple_cron_job_schedule  --set description="new description" --no-wait

Note

To update more than just tags and description, consider using az ml schedule create --file update_schedule.yml.

Python SDK

# Update trigger expression
job_schedule.trigger.expression = "10 10 * * 1"
job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Studio UI

In the schedule detail page, you can select Update settings to update the schedule settings, including job input/output and runtime settings.

Update a new version pipeline to an existing schedule

Once you set up a schedule to do regular retraining or batch inference on production, you might continue fine tuning or optimizing the model. When you have a new version pipeline job with optimized performance, you can update the new version pipeline to run on an existing schedule.

1. In the new version pipeline job detail page, select Schedule > Update to existing schedule.

   

2. Select an existing schedule to update its job definition.

   

   Important

   Make sure you select the correct schedule you want to update.

3. Optionally, select Next to modify the job inputs/outputs and runtime settings for the future jobs triggered by the schedule.

4. Select Review + Update to review the schedule settings, and then select Review + Update again to finish the update.

After the update completes, you can view the new job definition in the schedule detail page. The schedule now triggers the new job.

Disable a schedule

Azure CLI

az ml schedule disable -n simple_cron_job_schedule --no-wait

Python SDK

The following code returns False:

job_schedule = ml_client.schedules.begin_disable(name=schedule_name).result()
job_schedule.is_enabled

Studio UI

You can disable schedules from the All schedules tab or disable the current schedule from the schedule detail page.

Enable a schedule

Azure CLI

az ml schedule enable -n simple_cron_job_schedule --no-wait

Python SDK

The following code returns True:

job_schedule = ml_client.schedules.begin_enable(name=schedule_name).result()
job_schedule.is_enabled

Studio UI

You can enable schedules from the All schedules tab or enable the current schedule from the schedule detail page.

Delete a schedule

Important

You must first disable a schedule to delete it. Deletion is permanent and unrecoverable.

Azure CLI

az ml schedule delete -n simple_cron_job_schedule

Python SDK

# Only disabled schedules can be deleted
ml_client.schedules.begin_disable(name=schedule_name).result()
ml_client.schedules.begin_delete(name=schedule_name).result()

Studio UI

You can delete schedules from the All schedules tab or delete the current schedule from the schedule detail page.

Query triggered jobs from a schedule

Jobs triggered by a specific schedule all have the display name <schedule_name>-YYYYMMDDThhmmssZ. For example, if a schedule named named-schedule runs every 12 hours starting at 6 AM on January 1, 2021, the display names of the jobs created are as follows:

• named-schedule-20210101T060000Z
• named-schedule-20210101T180000Z
• named-schedule-20210102T060000Z
• named-schedule-20210102T180000Z, and so on

You can also apply Azure CLI JMESPath query to query the jobs triggered by a schedule name.

# query triggered jobs from schedule, please replace the simple_cron_job_schedule to your schedule name
az ml job list --query "[?contains(display_name,'simple_cron_schedule')]"

Tip

The Jobs history tab on the schedule detail page in the studio provides a simple way to find all jobs triggered by a schedule.

---

Role-based access control (RBAC) support

Because schedules are used for production, it's important to reduce the possibility and impact of misoperation. Workspace admins can restrict access to schedule creation and management in a workspace.

Admins can configure the following action rules related to schedules in the Azure portal. For more information, see Manage access to Azure Machine Learning workspaces.

Action	Description	Rule
Read	Get and list schedules	Microsoft.MachineLearningServices/workspaces/schedules/read
Write	Create, update, disable, and enable schedules	Microsoft.MachineLearningServices/workspaces/schedules/write
Delete	Delete schedules	Microsoft.MachineLearningServices/workspaces/schedules/delete

Cost considerations

Schedules are billed based on the number of schedules. Each schedule creates a logic app that Azure Machine Learning hosts on behalf of (HOBO) the user.

The logic app charges back to the user's Azure subscription. HOBO resource costs are billed using the same meter emitted by the original resource provider. Charges appear under the host resource, which is the Azure Machine Learning workspace.

Related content

• CLI (v2) job schedule YAML schema
• CLI (v2) core YAML syntax
• What are machine learning pipelines?
• What is an Azure Machine Learning component?