How to define and start a migration job

When you migrate a share to Azure, you'll need to describe the source share, the Azure target, and any migration settings you want to apply. These attributes are defined in a job definition within your storage mover resource. This article describes how to create and run such a job definition.

Prerequisites

Before you begin following the examples in this article, it's important that you have an understanding of the Azure Storage Mover resource hierarchy. Review the Understanding the Storage Mover resource hierarchy article, to understand the necessity of the job definition prerequisites.

There are three prerequisites to the definition the migration of your source shares:

  1. You need to have deployed a storage mover resource. Follow the steps in the Create a storage mover resource article to deploy a storage mover resource to the desired region within your Azure subscription.
  2. You need to deploy and register an Azure Storage Mover agent virtual machine (VM). Follow the steps in the Azure Storage Mover agent VM deployment and agent registration articles to deploy at least one agent.
  3. Finally, to define a migration, you'll need to create a job definition. Job definitions are organized in a migration project. You'll need at least one migration project in your storage mover resource. If you haven't already, follow the deployment steps in the manage projects article to create a migration project.

Create and start a job definition

A job definition is created within a project resource. Creating a job definition requires you to select or configure a project, a source and target storage endpoint, and a job name. If you've followed the examples contained in previous articles, you may have an existing project within a previously deployed storage mover resource. Follow the steps below to add a job definition to a project.

Storage endpoints are separate resources in your storage mover. Endpoints must be created before they can be referenced by a job definition.

Refer to the resource naming convention for help with choosing supported resource names.

  1. Navigate to the Project explorer page within the Azure portal to view a list of available projects. If no projects exist, or you need to create a new project, you can follow the steps included in the Manage Azure Storage Mover projects article.

    Screen capture of the Project Explorer's Overview tab within the Azure portal.

    From within the project explorer pane or the results list, select the name of an available project. The project's properties and job summary data are displayed in the details pane. Any existing job definitions defined for the project will also be displayed. The status of any deployed jobs will also be shown.

    In the actions menu within the project's details pane, select Create job definition to open the Create a migration job window. If no job definitions exist within the project, you can also select Create a job definition near the bottom of the pane, as shown in the example below.

    Screen capture of the Project Explorer's Overview tab within the Azure portal highlighting the use of filters.

  2. In the Basics tab of the Create a migration job window, enter a value in the required Name field. You may also add an optional description value of less than 1024 characters. Finally, in the Migration agent section, select the agent to perform the data migration and then select Next to open the Source tab. You should choose an agent located as near your data source as possible. The selected agent should also have resources appropriate to the size and complexity of the job. You can assign a different agent to your job at a later time if desired.

    Screen capture of the migration job's Basics tab, showing the location of the data fields.

  3. In the Source tab, select an option within the Source endpoint field.

    If you want to use a source endpoint you've previously defined, choose the Select an existing endpoint option. Next, select the Select an existing endpoint as a source link to open the source endpoint pane. This pane displays a detailed list of your previously defined endpoints. Select the appropriate endpoint and select Select to return to the Source tab and populate the Existing source endpoint field.

    Screen capture of the Source tab illustrating the location of the Existing Source Endpoint field.

    To define a new source endpoint from which to migrate, select the Create a new endpoint option. Next, provide values for the required Host name or IP, Share name, and Protocol version fields. You may also add an optional description value of less than 1024 characters.

    Screen capture of the Source tab illustrating the location of the New Source Endpoint fields.

    By default, migration jobs will start from the root of your share. However, if your use case involves copying data from a specific path within your source share, you can provide the path in the Sub-path field. Supplying this value will start the data migration from the location you've specified. If the sub path you've specified isn't found, no data will be copied.

    Prior to creating an endpoint and a job resource, it's important to verify that the path you've provided is correct and that the data is accessible. You're unable to modify endpoints or job resources after they're created. If the specified path is wrong, you'll need to delete the resources and re-create them.

    Values for host, share name, and subpath are concatenated to form the full migration source path. The path is displayed in the Full path field within the Verify full path section. Copy the path provided and verify that you're able to access it before committing your changes.

    After you've confirmed that the share is accessible, select Next to save your source endpoint settings and begin defining your target.

  4. In the Target tab, select an option for the Target endpoint field.

    As with the source endpoint, choose the Select an existing endpoint reference option if you want to use a previously defined endpoint. Next, select the Select an existing endpoint as a target link to open the target endpoint pane. A detailed list of your previously defined endpoints is displayed. First, select the desired endpoint, then Select to populate the Existing source endpoint field and return to the Source tab.

    Screen capture of the Target tab illustrating the location of the Existing Target Endpoint field.

    Similarly, to define a new target endpoint, choose the Create a new endpoint option. Next, select values from the drop-down lists for the required Subscription, Storage account, and Container fields. You may also add an optional description value of less than 1024 characters.

    Screen capture of the Target tab illustrating the location of the New Target Endpoint fields.

    A target subpath value can be used to specify a location within the target container where your migrated data will be copied. The subpath value is relative to the container's root. Omitting the subpath value will result in the data being copied to the root, while providing a unique value will generate a new subfolder.

    After ensuring the accuracy of your settings, select Next to continue.

  5. Within the Settings tab, take note of the settings associated with the Copy mode and Migration outcomes. The service's copy mode will affect the behavior of the migration engine when files or folders change between copy iterations.

    The current release of Azure Storage Mover only supports merge mode.

    • Files will be kept in the target, even if they don’t exist in the source.
    • Files with matching names and paths will be updated to match the source.
    • Folder renames between copies may lead to duplicate content in the target.

    Migration outcomes are based upon the specific storage types of the source and target endpoints. For example, because blob storage only supports "virtual" folders, source files in folders will have their paths prepended to their names and placed in a flat list within a blob container. Empty folders will be represented as an empty blob in the target. Source folder metadata will be persisted in the custom metadata field of a blob, as they are with files.

    After viewing the effects of the copy mode and migration outcomes, select Next to review the values from the previous tabs.

  6. Review the settings for job name and description, and source and target storage endpoint settings. Use the Previous and Next options to navigate through the tabs and correct any mistakes, if needed. Finally, select Create to provision the job definition.

    Screen capture of the Review tab illustrating the location of the fields and settings.

Next steps

Now that you've created a job definition with source and target endpoints, learn how to estimate the time required to perform your migration job. Learn about Azure Storage Mover performance targets by visiting the article suggested below.