Schedule machine learning pipeline jobs

APPLIES TO: Azure CLI ml extension v2 (current) Python SDK azure-ai-ml v2 (current)

In this article, you learn how to schedule machine learning pipelines to run on Azure. You can schedule routine tasks like retraining models or regularly updating batch predictions based on elapsed time.

This article shows you how to create, list, view, update, disable, enable, and delete schedules by using the Azure Machine Learning CLI, Azure Machine Learning SDK v2 for Python, or Azure Machine Learning studio UI.

Tip

To schedule jobs by using an external orchestrator, like Azure Data Factory or Microsoft Fabric, consider deploying your pipeline jobs under a batch endpoint. For more information, see Deploy existing pipeline jobs to batch endpoints and Run Azure Machine Learning models from Fabric by using batch endpoints (preview).

Prerequisites

• An Azure subscription. If you don't have one, create a free account before you begin.
• An Azure Machine Learning workspace. To create a workspace, see Create workspace resources.
• An understanding of Azure Machine Learning pipelines. For information, see What are machine learning pipelines.

Azure CLI

• The Azure CLI and ml extension installed by following the instructions in Install, set up, and use the CLI (v2).
• Knowledge of how to create Azure Machine Learning YAML pipelines. For information, see Create and run machine learning pipelines using components with the Azure Machine Learning CLI.

Python SDK

• The Azure Machine Learning SDK v2 for Python installed.
• Knowledge of how to create Python pipelines for Azure Machine Learning. For information, see Create and run machine learning pipelines using components with the Azure Machine Learning SDK.

Studio UI

• Knowledge of how to create pipelines in Azure Machine Learning studio. For more information, see Create and run machine learning pipelines using components with the Azure Machine Learning studio.

Limitations

• Azure Machine Learning v2 schedules don't support event-based triggers.
• CLI and SDK v2 schedules support specifying complex recurrence patterns that contain multiple trigger timestamps. The studio UI displays the complex patterns but doesn't support editing them.
• The studio UI supports only v2 schedules, and can't list or access v1 schedules that are based on published pipelines or pipeline endpoints. You can create a schedule for an unpublished pipeline.
• If you set recurrence as the 31st or 30th day of every month, the schedule doesn't trigger jobs in months that have fewer days.
• DAYS and MONTHS values aren't supported in cron schedule expressions. Values passed for these parameters are ignored and treated as *.
• Even after assigning a managed identity to a schedule, the author must retain their job run permissions for the schedule to function.

Create a schedule

When you have a pipeline job with satisfying performance and outputs, you can set up a schedule to automatically trigger the job regularly. To do so, you must create a schedule that associates the job with a trigger. The trigger can be either a recurrence pattern or a cron expression that specifies the interval and frequency to run the job.

In both cases, you need to define a pipeline job first, either inline or by specifying an existing pipeline job. You can define pipelines in YAML and run them from the CLI, author pipelines inline in Python, or compose pipelines in Azure Machine Learning studio. You can create pipeline jobs locally or from existing jobs in the workspace.

You can create v2 schedules for v2 or v1 pipeline jobs by using the studio UI, SDK v2, or CLI v2. You don't have to publish existing pipelines first to set up schedules for pipeline jobs.

Azure CLI

The code examples in this article are from Working with Schedule in Azure Machine Learning CLI 2.0.

Python SDK

The code examples in this article are excerpts from the Working with Schedule Azure Machine Learning notebook. Run the notebook to create and manage the schedules as described.

Studio UI

To create a schedule for an Azure Machine Learning pipeline job in the studio UI, open the pipeline job detail page.

Define a time-based schedule with a recurrence pattern

Azure CLI

The following YAML code defines a recurring schedule for a pipeline job. The required type parameter specifies that the trigger type is recurrence.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_recurrence_job_schedule
display_name: Simple recurrence job schedule
description: a simple hourly recurrence job schedule

trigger:
  type: recurrence
  frequency: day #can be minute, hour, day, week, month
  interval: 1 #every day
  schedule:
    hours: [4,5,10,11,12]
    minutes: [0,30]
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

create_job: ./simple-pipeline-job.yml
# create_job: azureml:simple-pipeline-job

Provide the following schedule parameters. Required parameters are marked; the rest are optional.

Python SDK

The following code uses RecurrenceTrigger to provide a better coding experience.

schedule_name = "simple_sdk_create_schedule_recurrence"

schedule_start_time = datetime.utcnow()
recurrence_trigger = RecurrenceTrigger(
    frequency="day",
    interval=1,
    schedule=RecurrencePattern(hours=10, minutes=[0, 1]),
    start_time=schedule_start_time,
    time_zone=TimeZone.UTC,
)

job_schedule = JobSchedule(
    name=schedule_name, trigger=recurrence_trigger, create_job=pipeline_job
)

Provide the following schedule parameters. Required parameters are marked; the rest are optional.

Studio UI

To open the schedule creation wizard, select Schedule > Create new schedule at the top of the page.

Azure CLI / Python SDK

Parameters

• frequency (required) is the time unit on which basis the schedule fires. Can be minute, hour, day, week, or month.
• interval (required) is the number of time units between schedule recurrences.
• schedule (optional) defines the recurrence pattern, which can contain hours, minutes, and weekdays. If omitted, jobs trigger according to the logic of start_time, frequency, and interval.
  • When frequency is day, the pattern can specify hours and minutes.
  • When frequency is week or month, the pattern can specify hours, minutes, and weekdays.
    • hours is an integer or list from 0 to 23.
    • minutes is an integer or list from 0 to 59.
    • weekdays is a string or list from monday to sunday.
• start_time (optional) is the start date and time with timezone. If omitted, the default is equal to schedule creation time. If the start time is in the past, the first job runs at the next calculated run time.
• end_time (optional) is the end date and time with timezone. If you omit this parameter, the schedule remains active until you manually disable it.
• time_zone (optional) specifies the time zone of the recurrence schedule. If omitted, the default is Coordinated Universal Time (UTC). For more information about timezone values, see the appendix for timezone values.

Studio UI

To define a recurrence-based schedule, on the Basic settings screen, set the following properties. Only the Name property requires a value. If you don't specify values for the other properties, the default schedule is a Recurrence pattern that starts at schedule creation and runs every Monday through Friday at 4:00 PM UTC.

• Name: Unique identifier of the schedule within the workspace.
• Description: Schedule description.
• Trigger: Recurrence pattern of the schedule, including the following properties:
  • Time zone: Time zone to use for the trigger time, UTC by default.
  • Select Recurrence and specify a recurring pattern of minutes, hours, days, weeks, or months.
  • Start: Date the schedule becomes active, by default the date created.
  • End: Date when the schedule becomes inactive. By default the value is none, and the schedule remains active until manually disabled.
  • Tags: Tags on the schedule.

Azure CLI

After you create the schedule YAML, use the following command to create the schedule via CLI:

# This action creates related resources for a schedule and can take up to a minute to complete.
az ml schedule create --file recurrence-job-schedule.yml --no-wait

To confirm the schedule was created and is active, list your schedules as shown in List schedules or open it in View schedule details. A new schedule shows is_enabled: true.

Python SDK

The following Python code creates the schedule you defined:

job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Studio UI

After you configure the basic settings, select Review + Create, review the settings, and then select Review + Create again to create the schedule.

Define a time-based schedule with a cron expression

A cron expression specifies a flexible and customized recurrence pattern for a schedule. A standard crontab expression is composed of the space-delimited fields MINUTES HOURS DAYS MONTHS DAYS-OF-WEEK. A wildcard * means all values for a field.

In an Azure Machine Learning schedule cron expression:

• MINUTES is an integer or list from 0 to 59.
• HOURS is an integer or list from 0 to 23.
• DAYS values aren't supported, and are always treated as *. The * value in DAYS means all days in a month, which varies with month and year.
• MONTHS values aren't supported, and are always treated as *.
• DAYS-OF-WEEK is an integer or list from 0 to 6, where 0 = Sunday. Names of days are also accepted.

For example, the expression 15 16 * * 1 means 4:15 PM UTC every Monday. For more information about crontab expressions, see the Crontab Expression wiki on GitHub.

Azure CLI

The following YAML code defines a recurring schedule for a pipeline job. The required type parameter specifies that the trigger type is cron.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_cron_job_schedule
display_name: Simple cron job schedule
description: a simple hourly cron job schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

# create_job: azureml:simple-pipeline-job
create_job: ./simple-pipeline-job.yml

Provide the following schedule parameters. Required parameters are marked; the rest are optional.

Python SDK

The following code uses CronTrigger to provide a better coding experience.

schedule_name = "simple_sdk_create_schedule_cron"

schedule_start_time = datetime.utcnow()
cron_trigger = CronTrigger(
    expression="15 10 * * *",
    start_time=schedule_start_time,  # start time
    time_zone="Eastern Standard Time",  # time zone of expression
)

job_schedule = JobSchedule(
    name=schedule_name, trigger=cron_trigger, create_job=pipeline_job
)

Provide the following schedule parameters. Required parameters are marked; the rest are optional.

Studio UI

To define a cron-based schedule, select Cron expression instead of Recurrence on the Basic settings screen for the schedule.

Azure CLI / Python SDK

Parameters

• expression (required) is a standard crontab expression that defines a recurring schedule.
• start_time (optional) is the schedule start date and time with timezone. For example, start_time: "2022-05-10T10:15:00-04:00" means the schedule starts at 10:15:00 AM on May 10, 2022, in the UTC-4 timezone. If you omit this parameter, the default is the schedule creation time. If the start time is in the past, the first job runs at the next calculated run time.
• end_time (optional) is the end date and time with timezone. If you omit this parameter, the schedule remains active until you manually disable it.
• time_zone (optional) specifies the time zone of the recurrence schedule. If you omit this parameter, the default is UTC.

Studio UI

Only the Name property requires a value. If you don't specify a cron expression, the default cron expression creates a schedule that runs daily at 4:00 PM UTC.

• Name: Unique identifier of the schedule within the workspace.
• Description: Description of the schedule.
• Trigger: Recurrence pattern of the schedule, including the following properties:
  • Time zone: Time zone to use for the trigger time, Coordinated Universal Time (UTC) by default.
  • Select Cron expression and provide a standard crontab expression that defines a recurring schedule.
  • Start: Date the schedule becomes active, by default the date created.
  • End: Date the schedule becomes inactive. By default the value is none, and a schedule remains active until you manually disable it.
  • Tags: Tags on the schedule.

Azure CLI

After you create the schedule YAML, use the following command to create the schedule via CLI:

# This action creates related resources for a schedule and can take up to a minute to complete.
az ml schedule create --file cron-job-schedule.yml --no-wait

To confirm the schedule was created and is active, list your schedules as shown in List schedules or open it in View schedule details. A new schedule shows is_enabled: true.

Python SDK

The following Python code creates the schedule you defined:

job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Studio UI

After you configure the basic settings, select Review + Create, review the settings, and then select Review + Create again to create the schedule.

Change job settings when you define schedules

Sometimes you want the jobs that schedules trigger to have different configurations from the test jobs. When you define a schedule by using an existing job, you can change the job settings. This approach lets you define multiple schedules that use the same job with different inputs.

Azure CLI

When you define a schedule, you can change the settings, inputs, or outputs to use when running the pipeline job. You can also change the experiment_name of the triggered job.

The following schedule definition changes the settings of an existing job.

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: cron_with_settings_job_schedule
display_name: Simple cron job schedule
description: a simple hourly cron job schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

create_job: 
  type: pipeline
  job: ./simple-pipeline-job.yml
  # job: azureml:simple-pipeline-job
  # runtime settings
  settings:
    #default_compute: azureml:cpu-cluster
    continue_on_step_failure: true
  inputs:
    hello_string_top_level_input: ${{name}} 
  tags: 
    schedule: cron_with_settings_schedule

Python SDK

When you define a schedule, you can change the settings, inputs, or outputs to use when running the pipeline job. You can also change the experiment_name of the triggered job.

The following schedule definition changes the settings of a pipeline per job.

# set run time settings
pipeline_job = pipeline_with_components_from_yaml(
    training_input=Input(type="uri_folder", path=parent_dir + "/data/"),
    test_input=Input(type="uri_folder", path=parent_dir + "/data/"),
    training_max_epochs=20,
    training_learning_rate=1.8,
    learning_rate_schedule="time-based",
)

# set pipeline level compute
pipeline_job.settings.default_compute = "cpu-cluster"

Studio UI

In the studio UI, use Advanced settings in the schedule creation wizard to modify inputs, outputs, and runtime settings for a pipeline job. You can't change the experiment_name in the studio UI.

1. In Job inputs & outputs, modify inputs and outputs for future jobs triggered by the schedule. Use macro expressions for the inputs and outputs paths.

   

2. In Job runtime settings, modify compute and other runtime settings for jobs triggered by the schedule.

   

3. Select Review + Create to review the schedule settings you configured, and then select Review + Create again to create the schedule.

   

Use supported expressions in schedules

When you define a schedule, use the following macro expressions to define dynamic parameter values that resolve to actual values during job runtime.

Expression	Description	Supported properties
${{name}}	Name of the job	outputs path of the pipeline job
${{creation_context.trigger_time}}	Trigger time of the job	String type inputs of the pipeline job

Manage schedule

You can list, view details, update, disable, enable, and delete schedules in a workspace.

List schedules

Azure CLI

az ml schedule list

Python SDK

schedules = ml_client.schedules.list()
[s.name for s in schedules]

Studio UI

Under Jobs in the studio, select the All schedules tab to see a list of all the v2 job schedules created by SDK, CLI, or studio UI in this workspace.

View schedule details

Azure CLI

az ml schedule show -n simple_cron_job_schedule

Python SDK

created_schedule = ml_client.schedules.get(name=schedule_name)
[created_schedule.name]

Studio UI

Select a schedule name to show the schedule detail page, which contains the following tabs:

• Overview provides basic information about the schedule.

  

• Job definition defines the job triggered by the schedule.

  

• Jobs history provides a list of all jobs triggered by the schedule.

  

Update a schedule

Azure CLI

az ml schedule update -n simple_cron_job_schedule  --set description="new description" --no-wait

Note

To update more properties than tags and description, consider using az ml schedule create --file update_schedule.yml.

Python SDK

# Update trigger expression
job_schedule.trigger.expression = "10 10 * * 1"
job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Studio UI

In the schedule detail page, select Update settings to update the schedule settings, including job input/output and runtime settings.

Update a new version pipeline to an existing schedule

After you set up a schedule to regularly retrain or batch infer on production, you might continue fine-tuning or optimizing the model. When you have a new version pipeline job with optimized performance, you can update the new version pipeline to run on an existing schedule.

1. In the new version pipeline job detail page, select Schedule > Update to existing schedule.

   

2. Select an existing schedule to update its job definition.

   

   Important

   Ensure you select the correct schedule to update.

3. Optionally, select Next to modify the job inputs/outputs and runtime settings for the future jobs triggered by the schedule.

4. Select Review + Update to review the schedule settings, and then select Review + Update again to finish the update.

After the update finishes, you can view the new job definition in the schedule detail page. The schedule now triggers the new job.

Disable a schedule

Azure CLI

az ml schedule disable -n simple_cron_job_schedule --no-wait

Python SDK

The following code returns False:

job_schedule = ml_client.schedules.begin_disable(name=schedule_name).result()
job_schedule.is_enabled

Studio UI

You can disable schedules from the All schedules tab or disable the current schedule from the schedule detail page.

Enable a schedule

Azure CLI

az ml schedule enable -n simple_cron_job_schedule --no-wait

Python SDK

The following code returns True:

job_schedule = ml_client.schedules.begin_enable(name=schedule_name).result()
job_schedule.is_enabled

Studio UI

You can enable schedules from the All schedules tab or enable the current schedule from the schedule detail page.

Delete a schedule

Important

You must first disable a schedule to delete it. Deletion is permanent and unrecoverable.

Azure CLI

az ml schedule delete -n simple_cron_job_schedule

Python SDK

# Only disabled schedules can be deleted
ml_client.schedules.begin_disable(name=schedule_name).result()
ml_client.schedules.begin_delete(name=schedule_name).result()

Studio UI

You can delete schedules from the All schedules tab or delete the current schedule from the schedule detail page.

Query triggered jobs from a schedule

Jobs triggered by a specific schedule all use the display name <schedule_name>-YYYYMMDDThhmmssZ. For example, if a schedule named named-schedule runs every 12 hours starting at 6 AM on January 1, 2021, the display names of the jobs created are as follows:

• named-schedule-20210101T060000Z
• named-schedule-20210101T180000Z
• named-schedule-20210102T060000Z
• named-schedule-20210102T180000Z

The display names continue at each scheduled interval.

You can also apply Azure CLI JMESPath query to query the jobs triggered by a schedule name.

# query triggered jobs from schedule, please replace the simple_cron_job_schedule to your schedule name
az ml job list --query "[?contains(display_name,'simple_cron_schedule')]"

Tip

The Jobs history tab on the schedule detail page in the studio provides an easy way to find all jobs triggered by a schedule.

---

Role-based access control (RBAC) support

Because schedules are used for production, it's important to reduce the possibility and impact of misoperation. Workspace admins can restrict access to schedule creation and management in a workspace.

Admins can configure the following action rules related to schedules in the Azure portal. For more information, see Manage access to Azure Machine Learning workspaces.

Action	Description	Rule
Read	Get and list schedules	Microsoft.MachineLearningServices/workspaces/schedules/read
Write	Create, update, disable, and enable schedules	Microsoft.MachineLearningServices/workspaces/schedules/write
Delete	Delete schedules	Microsoft.MachineLearningServices/workspaces/schedules/delete

Cost considerations

Azure bills you for schedules based on the number of schedules. Each schedule creates a logic app that Azure Machine Learning hosts on behalf of (HOBO) you. Therefore, you can't see the logic app as a resource under your subscription in Azure portal.

The logic app charges back to your Azure subscription. HOBO resource costs use the same meter emitted by the original resource provider. Charges appear under the host resource, which is the Azure Machine Learning workspace.

Troubleshoot schedules

Use the following table to diagnose common issues with scheduled pipeline jobs.

Symptom	Cause	Resolution
The schedule doesn't trigger jobs.	The schedule is disabled, or its start_time is in the future.	Confirm the schedule is enabled, and check start_time. Use View schedule details to inspect the trigger.
Triggered jobs fail after they previously succeeded.	The schedule author lost the permissions needed to submit the job, even when a managed identity is assigned.	Restore the author's job-submission permissions. See Role-based access control (RBAC) support.
No job runs on the 30th or 31st of some months.	Those days don't exist in every month.	Use a cron expression, or choose a day that exists in every month.
DAYS or MONTHS values in a cron expression are ignored.	These fields aren't supported and are always treated as *.	Express the pattern by using the supported fields, or use a recurrence trigger.

Related content

• CLI (v2) job schedule YAML schema
• CLI (v2) core YAML syntax
• What are machine learning pipelines?
• What is an Azure Machine Learning component?
• Deploy existing pipeline jobs to batch endpoints