Schedule data import jobs (preview)

APPLIES TO: Azure CLI ml extension v2 (current) Python SDK azure-ai-ml v2 (current)

In this article, you'll learn how to programmatically schedule data imports, using the schedule UI to do it. You can create a schedule based on elapsed time. Time-based schedules can handle routine tasks - for example, regular data imports to keep them up-to-date. After learning how to create schedules, you'll learn how to retrieve, update and deactivate them via CLI, SDK, and studio UI resources.

Prerequisites

• You need an Azure subscription to use Azure Machine Learning. If you don't have an Azure subscription, create a free account before you begin. Try the free or paid version of Azure Machine Learning today.

Azure CLI

• Install the Azure CLI and the ml extension. Follow the installation steps in Install, set up, and use the CLI (v2).

• Create an Azure Machine Learning workspace if you don't have one. For workspace creation, see Install, set up, and use the CLI (v2).

Python SDK

• Create an Azure Machine Learning workspace if you don't have one.
• The Azure Machine Learning SDK v2 for Python.

Studio

• An Azure Machine Learning workspace. See Create workspace resources.
• Understanding of Azure Machine Learning pipelines. See what are machine learning pipelines, and how to create pipeline job in CLI v2 or SDK v2.

Schedule data import

To import data on a recurring basis, you must create a schedule. A Schedule associates a data import action with a trigger. The trigger can either be cron, which uses a cron expression to describe the delay between runs, or a recurrence, which specifies the frequency to trigger a job. In each case, you must first build an import data definition. An existing data import, or a data import that is defined inline, works for this. For more information, visit Create a data import in CLI, SDK and UI.

Create a schedule

Create a time-based schedule with recurrence pattern

Azure CLI

APPLIES TO: Azure CLI ml extension v2 (current)

YAML: Schedule for data import with recurrence pattern

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_recurrence_import_schedule
display_name: Simple recurrence import schedule
description: a simple hourly recurrence import schedule

trigger:
  type: recurrence
  frequency: day #can be minute, hour, day, week, month
  interval: 1 #every day
  schedule:
    hours: [4,5,10,11,12]
    minutes: [0,30]
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

import_data: ./my-snowflake-import-data.yaml

YAML: Schedule for data import definition inline with recurrence pattern on managed datastore

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: inline_recurrence_import_schedule
display_name: Inline recurrence import schedule
description: an inline hourly recurrence import schedule

trigger:
  type: recurrence
  frequency: day #can be minute, hour, day, week, month
  interval: 1 #every day
  schedule:
    hours: [4,5,10,11,12]
    minutes: [0,30]
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

import_data:
  type: mltable
  name: my_snowflake_ds
  path: azureml://datastores/workspacemanagedstore
  source:
    type: database
    query: select * from TPCH_SF1.REGION
    connection: azureml:my_snowflake_connection

A trigger contains these properties:

• (Required) type specifies the schedule type, either recurrence or cron. The following section has more information.

Next, run this command in the CLI:

> az ml schedule create -f <file-name>.yml

Python SDK

APPLIES TO: Python SDK azure-ai-ml v2 (current)

from azure.ai.ml.data_transfer import Database
from azure.ai.ml.constants import TimeZone
from azure.ai.ml.entities import (
    ImportDataSchedule,
    RecurrenceTrigger,
    RecurrencePattern,
)
from datetime import datetime

source = Database(connection="azureml:my_sf_connection", query="select * from my_table")

path = "azureml://datastores/workspaceblobstore/paths/snowflake/schedule/${{name}}"


my_data = DataImport(
    type="mltable", source=source, path=path, name="my_schedule_sfds_test"
)

schedule_name = "my_simple_sdk_create_schedule_recurrence"

schedule_start_time = datetime.utcnow()

recurrence_trigger = RecurrenceTrigger(
    frequency="day",
    interval=1,
    schedule=RecurrencePattern(hours=1, minutes=[0, 1]),
    start_time=schedule_start_time,
    time_zone=TimeZone.UTC,
)

import_schedule = ImportDataSchedule(
    name=schedule_name, trigger=recurrence_trigger, import_data=my_data
)

ml_client.schedules.begin_create_or_update(import_schedule).result()

RecurrenceTrigger contains following properties:

• (Required) For a better coding experience, use RecurrenceTrigger for the recurrence schedule.

Studio

When your data import has satisfactory performance and outputs, you can set up a schedule to automatically trigger that import.

1. Navigate to Azure Machine Learning studio

2. Under Assets in the left navigation, select Data. At the Data import tab, select the imported data asset to which you want to attach a schedule. The Import jobs history page should appear, as shown in this screenshot:

   

3. At the Import jobs history page, select the latest Import job name hyperlink URL, to open the pipelines job details page as shown in this screenshot:

   

4. At the pipeline job details page of any data import, select Schedule -> Create new schedule to open the schedule creation wizard, as shown in this screenshot:

   

5. The Basic settings of the schedule creation wizard have the properties shown in this screenshot:

   

• Name: the unique identifier of the schedule within the workspace.

• Description: the schedule description.

• Trigger: the recurrence pattern of the schedule, which includes these properties:

  • Time zone: the trigger time calculation is based on this time zone; (UTC) Coordinated Universal Time by default.
  • Recurrence or Cron expression: select recurrence to specify the recurring pattern. Under Recurrence, you can specify the recurrence frequency - by minutes, hours, days, weeks, or months.
  • Start: the schedule first becomes active on this date. By default, the creation date of this schedule.
  • End: the schedule will become inactive after this date. By default, it's NONE, which means that the schedule remains active until you manually disable it.
  • Tags: the selected schedule tags.

  After you configure the basic settings, you can select Review + Create, and the schedule will automatically submit the data import based on the recurrence pattern you specified. You can also select Next, and navigate through the wizard to select or update the data import parameters.

Note

These properties apply to CLI and SDK:

• (Required) frequency specifies the unit of time that describes how often the schedule fires. Can have values

  • minute
  • hour
  • day
  • week
  • month

• (Required) interval specifies how often the schedule fires based on the frequency, which is the number of time units to wait until the schedule fires again.

• (Optional) schedule defines the recurrence pattern, containing hours, minutes, and weekdays.

  • When frequency equals day, a pattern can specify hours and minutes.
  • When frequency equals week and month, a pattern can specify hours, minutes and weekdays.
  • hours should be an integer or a list, ranging between 0 and 23.
  • minutes should be an integer or a list, ranging between 0 and 59.
  • weekdays a string or list ranging from monday to sunday.
  • If schedule is omitted, the job(s) triggers fire according to the logic of start_time, frequency and interval.

• (Optional) start_time describes the start date and time, with a timezone. If start_time is omitted, start_time equals the job creation time. For a start time in the past, the first job runs at the next calculated run time.

• (Optional) end_time describes the end date and time with a timezone. If end_time is omitted, the schedule continues to trigger jobs until the schedule is manually disabled.

• (Optional) time_zone specifies the time zone of the recurrence. If omitted, the default timezone is UTC. For more information about timezone values, visit appendix for timezone values.

Create a time-based schedule with cron expression

Azure CLI

YAML: Schedule for a data import with cron expression

APPLIES TO: Azure CLI ml extension v2 (current)

YAML: Schedule for data import with cron expression (preview)

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: simple_cron_import_schedule
display_name: Simple cron import schedule
description: a simple hourly cron import schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

import_data: ./my-snowflake-import-data.yaml

YAML: Schedule for data import definition inline with cron expression (preview)

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: inline_cron_import_schedule
display_name: Inline cron import schedule
description: an inline hourly cron import schedule

trigger:
  type: cron
  expression: "0 * * * *"
  start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time
  time_zone: "Pacific Standard Time" # optional - default will be UTC

import_data:
  type: mltable
  name: my_snowflake_ds
  path: azureml://datastores/workspaceblobstore/paths/snowflake/${{name}}
  source:
    type: database
    query: select * from TPCH_SF1.REGION
    connection: azureml:my_snowflake_connection

The trigger section defines the schedule details and contains these properties:

• (Required) type specifies the cron schedule type.

> az ml schedule create -f <file-name>.yml

The list continues here:

Python SDK

APPLIES TO: Python SDK azure-ai-ml v2 (current)

from azure.ai.ml.data_transfer import Database
from azure.ai.ml.constants import TimeZone
from azure.ai.ml.entities import CronTrigger, ImportDataSchedule

source = Database(connection="azureml:my_sf_connection", query="select * from my_table")

path = "azureml://datastores/workspaceblobstore/paths/snowflake/schedule/${{name}}"


my_data = DataImport(
    type="mltable", source=source, path=path, name="my_schedule_sfds_test"
)

schedule_name = "my_simple_sdk_create_schedule_cron"

cron_trigger = CronTrigger(
    expression="15 10 * * 1",
    start_time=datetime.utcnow(),
    end_time="2023-12-03T18:40:00",
)
import_schedule = ImportDataSchedule(
    name=schedule_name, trigger=cron_trigger, import_data=my_data
)
ml_client.schedules.begin_create_or_update(import_schedule).result()

The CronTrigger section defines the schedule details and contains these properties:

• (Required) For a better coding experience, use CronTrigger for the recurrence schedule.

The list continues here:

Studio

When your data import has satisfactory performance and outputs, you can set up a schedule to automatically trigger that import.

1. Navigate to Azure Machine Learning studio

2. Under Assets in the left navigation, select Data. On the Data import tab, select the imported data asset to which you want to attach a schedule. The Import jobs history page should appear, as shown in this screenshot:

   

3. At the Import jobs history page, select the latest Import job name hyperlink URL, to open the pipelines job details page as shown in this screenshot:

   

4. At the pipeline job details page of any data import, select Schedule -> Create new schedule to open the schedule creation wizard, as shown in this screenshot:

   

5. The Basic settings of the schedule creation wizard have the properties shown in this screenshot:

   

• Name: the unique identifier of the schedule within the workspace.

• Description: the schedule description.

• Trigger: the recurrence pattern of the schedule, which includes these properties:

  • Time zone: the trigger time calculation is based on this time zone; (UTC) Coordinated Universal Time by default.
  • Recurrence or Cron expression: select recurrence to specify the recurring pattern. With Cron expression, you can specify a more flexible and customized recurrence pattern.
  • Start: the schedule first becomes active on this date. By default, the creation date of this schedule.
  • End: the schedule will become inactive after this date. By default, it's NONE, which means that the schedule remains active until you manually disable it.
  • Tags: the selected schedule tags.

  After you configure the basic settings, you can select Review + Create, and the schedule will automatically submit the data import based on the recurrence pattern you specified. You can also select Next, and navigate through the wizard to select or update the data import parameters.

• (Required) expression uses a standard crontab expression to express a recurring schedule. A single expression is composed of five space-delimited fields:

  MINUTES HOURS DAYS MONTHS DAYS-OF-WEEK

  • A single wildcard (*), which covers all values for the field. A *, in days, means all days of a month (which varies with month and year).

  • The expression: "15 16 * * 1" in the sample above means the 16:15PM on every Monday.

  • This table lists the valid values for each field:

    Field	Range	Comment
    MINUTES	0-59	-
    HOURS	0-23	-
    DAYS	-	Not supported. The value is ignored and treated as *.
    MONTHS	-	Not supported. The value is ignored and treated as *.
    DAYS-OF-WEEK	0-6	Zero (0) means Sunday. Names of days also accepted.

  • For more information about crontab expressions, visit the Crontab Expression wiki resource on GitHub.

  Important

  DAYS and MONTH are not supported. If you pass one of these values, it will be ignored and treated as *.

• (Optional) start_time specifies the start date and time with the timezone of the schedule. For example, start_time: "2022-05-10T10:15:00-04:00" means the schedule starts from 10:15:00AM on 2022-05-10 in the UTC-4 timezone. If start_time is omitted, the start_time equals the schedule creation time. For a start time in the past, the first job runs at the next calculated run time.

• (Optional) end_time describes the end date, and time with a timezone. If end_time is omitted, the schedule continues to trigger jobs until the schedule is manually disabled.

• (Optional) time_zonespecifies the time zone of the expression. If time_zone is omitted, the timezone is UTC by default. For more information about timezone values, visit appendix for timezone values.

Limitations:

• Currently, Azure Machine Learning v2 scheduling doesn't support event-based triggers.
• Use the Azure Machine Learning SDK/CLI v2 to specify a complex recurrence pattern that contains multiple trigger timestamps. The UI only displays the complex pattern and doesn't support editing.
• If you set the recurrence as the 31st day of every month, the schedule won't trigger jobs in months with less than 31 days.

List schedules in a workspace

Azure CLI

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule list

Python SDK

APPLIES TO: Python SDK azure-ai-ml v2 (current)

schedules = ml_client.schedules.list()
[s.name for s in schedules]

Studio

In the studio portal, under the Jobs extension, select the All schedules tab. That tab shows all your job schedules created by the SDK/cli/UI, in a single list. In the schedule list, you have an overview of all schedules in this workspace, as shown in this screenshot:

Check schedule detail

Azure CLI

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule show -n simple_cron_data_import_schedule

Python SDK

APPLIES TO: Python SDK azure-ai-ml v2 (current)

created_schedule = ml_client.schedules.get(name=schedule_name)
[created_schedule.name]

Studio

You can select a schedule name to show the schedule details page. The schedule details page contains the following tabs, as shown in this screenshot:

• Overview: basic information for the specified schedule.

  

• Job definition: defines the job that the specified schedule triggers, as shown in this screenshot:

  

Update a schedule

Azure CLI

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule update -n simple_cron_data_import_schedule  --set description="new description" --no-wait

Note

To update more than just tags/description, we recommend use of az ml schedule create --file update_schedule.yml

Python SDK

APPLIES TO: Python SDK azure-ai-ml v2 (current)

job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Studio

Update a data import definition to existing schedule

To change the import frequency, or to create a new association for the data import job, you can update the import definition of an existing schedule.

Note

To update an existing schedule, the association of the schedule with the old import definition will be removed. A schedule can have only one import job definition. However, multiple schedules can call one data import definition.

1. Navigate to Azure Machine Learning studio

2. Under Assets in the left navigation, select Data. On the Data import tab, select the imported data asset to which you want to attach a schedule. The Import jobs history page should appear, as shown in this screenshot:

   

3. At the Import jobs history page, select the latest Import job name link, to open the pipelines job details page as shown in this screenshot:

   

4. At the pipeline job details page of any data import, select Schedule -> Updated to existing schedule, to open the Select schedule wizard, as shown in this screenshot:

   

5. Select an existing schedule from the list, as shown in this screenshot:

   

   Important

   Make sure you select the correct schedule to update. Once you finish the update, the schedule will trigger different data imports.

6. You can also modify the source, query and change the destination path, for future data imports that the schedule triggers.

7. Select Review + Update to finish the update process. The completed update will send a notification.

8. You can view the new data import definition in the schedule details page when the update is completed.

Update in schedule detail page

In the schedule details page, you can select Update settings to update both the basic settings and advanced settings, including the job input/output and runtime settings of the schedule, as shown in this screenshot:

Disable a schedule

Azure CLI

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule disable -n simple_cron_data_import_schedule --no-wait

Python SDK

job_schedule = ml_client.schedules.begin_disable(name=schedule_name).result()
job_schedule.is_enabled

Studio

You can disable the current schedule at the schedule details page. You can also disable schedules at the All schedules tab.

Enable a schedule

Azure CLI

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule enable -n simple_cron_data_import_schedule --no-wait

Python SDK

APPLIES TO: Python SDK azure-ai-ml v2 (current)

# Update trigger expression
job_schedule.trigger.expression = "10 10 * * 1"
job_schedule = ml_client.schedules.begin_create_or_update(
    schedule=job_schedule
).result()
print(job_schedule)

Studio

At the schedule details page, you can enable the current schedule. You can also enable schedules at the All schedules tab.

Delete a schedule

Important

A schedule must be disabled before deletion. Deletion is a permanent, unrecoverable action. After a schedule is deleted, you can never access or recover it.

Azure CLI

APPLIES TO: Azure CLI ml extension v2 (current)

az ml schedule delete -n simple_cron_data_import_schedule

Python SDK

APPLIES TO: Python SDK azure-ai-ml v2 (current)

# Only disabled schedules can be deleted
ml_client.schedules.begin_disable(name=schedule_name).result()
ml_client.schedules.begin_delete(name=schedule_name).result()

Studio

You can delete a schedule from the schedule details page or the all schedules tab.

RBAC (Role-based-access-control) support

Schedules are generally used for production. To prevent problems, workspace admins may want to restrict schedule creation and management permissions within a workspace.

There are currently three action rules related to schedules, and you can configure them in the Azure portal. For more information, visit how to manage access to an Azure Machine Learning workspace..

Action	Description	Rule
Read	Get and list schedules in Machine Learning workspace	Microsoft.MachineLearningServices/workspaces/schedules/read
Write	Create, update, disable and enable schedules in Machine Learning workspace	Microsoft.MachineLearningServices/workspaces/schedules/write
Delete	Delete a schedule in Machine Learning workspace	Microsoft.MachineLearningServices/workspaces/schedules/delete

Next steps

• Learn more about the CLI (v2) data import schedule YAML schema.
• Learn how to manage imported data assets.