Create and manage container copy jobs in Azure Cosmos DB (Preview)

Copy jobs help create copies of containers in Azure Cosmos DB accounts.

This article describes how to create, monitor, and manage copy jobs using Azure CLI commands.

Prerequisites

• You can use the portal Cloud Shell to run container copy commands. Alternately, you can run the commands locally; make sure you have Azure CLI downloaded and installed on your machine.
• Currently, container copy is only supported in these regions. Make sure your account's write region belongs to this list.
• Install the Azure Cosmos DB preview extension which contains the container copy commands.

  az extension add --name cosmosdb-preview
  

Online container copy

Set shell variables

First, set all of the variables that each individual script uses.

$sourceSubId = "<source-subscription-id>" 
$destinationSubId = "<destination-subscription-id>" 
$sourceAccountRG = "<source-resource-group-name>"
$destinationAccountRG = "<destination-resource-group-name>"
$sourceAccount = "<cosmos-source-account-name>"
$destinationAccount = "<cosmos-destination-account-name>"
$jobName = ""
$sourceDatabase = ""
$sourceContainer = ""
$destinationDatabase = ""
$destinationContainer = ""

Assign read permission

Note

This step is not required if you're copying data within same Azure Cosmos DB account.

While copying data from one account's container to another account's container, it is required to give read access of source container to destination account's identity to perform the copy operation. Follow the steps below to assign requisite read permission to destination account.

Using System managed identity

1. Set destination subscription context

   az account set --subscription $destinationSubId
   

2. Add system identity on destination account

   $identityOutput = az cosmosdb identity assign -n $destinationAccount -g $destinationAccountRG
   $principalId = ($identityOutput | ConvertFrom-Json).principalId
   

3. Set default identity on destination account

   az cosmosdb update -n $destinationAccount -g $destinationAccountRG --default-identity="SystemAssignedIdentity"
   

4. Set source subscription context

   az account set --subscription $sourceSubId
   

5. Add role assignment on source account

   # Read-only access role
   $roleDefinitionId = "00000000-0000-0000-0000-000000000001" 
   az cosmosdb sql role assignment create --account-name $sourceAccount --resource-group $sourceAccountRG --role-definition-id $roleDefinitionId --scope "/" --principal-id $principalId
   

6. Reset destination subscription context

   az account set --subscription $destinationSubId
   

Using User-assigned managed identity

1. Assign User-assigned managed identity variable

   $userAssignedManagedIdentityResourceId = "<CompleteResourceIdOfUserAssignedManagedIdentity>"
   

2. Set destination subscription context

   az account set --subscription $destinationSubId
   

3. Add user assigned managed identity on destination account

   $identityOutput = az cosmosdb identity assign -n $destinationAccount -g $destinationAccountRG --identities $userAssignedManagedIdentityResourceId
   $principalId = ($identityOutput | ConvertFrom-Json).userAssignedIdentities.$userAssignedManagedIdentityResourceId.principalId
   

4. Set default identity on destination account

   az cosmosdb update -n $destinationAccount -g $destinationAccountRG --default-identity=UserAssignedIdentity=$userAssignedManagedIdentityResourceId
   

5. Set source subscription context

   az account set --subscription $sourceSubId
   

6. Add role assignment on source account

   $roleDefinitionId = "00000000-0000-0000-0000-000000000001"  # Read-only access role
   az cosmosdb sql role assignment create --account-name $sourceAccount --resource-group $sourceAccountRG --role-definition-id $roleDefinitionId --scope "/" --principal-id $principalId
   

7. Reset destination subscription context

   az account set --subscription $destinationSubId
   

Create copy job

az cosmosdb copy create `
    --resource-group $destinationAccountRG `
    --job-name $jobName `
    --dest-account $destinationAccount `
    --src-account $sourceAccount `
    --dest-nosql database=$destinationDatabase container=$destinationContainer `
    --src-nosql database=$sourceDatabase container=$sourceContainer
    --mode Online

Monitor progress

Monitor the progress using below command.

az cosmosdb copy show `
    --resource-group $destinationAccountRG `
    --account-name $destinationAccount `
    --job-name $jobName

1. Total count – It represents the total number of changes (total document + any new changes) in the source container at any given time.
2. Processed count – It represents the total number of events coming from source container’s change feed that have been processed by the copy job.

Complete copy job

1. When the processed count becomes greater than or equal to the total count, turn off any updates on the source container and wait for 5-10 minutes to flush any remaining changes.
2. Run the completion API to finish the copy job and free compute resources, this will also write remaining changes (if any) to the destination container.

    az cosmosdb copy complete `
        --resource-group $destinationAccountRG `
        --account-name $destinationAccount `
        --job-name $jobName

1. Update the client applications to start using the new (destination) container if needed.

Offline container copy

Set shell variables

First, set all of the variables that each individual script uses.

$sourceSubId = "<source-subscription-id>" 
$destinationSubId = "<destination-subscription-id>" 
$sourceAccountRG = "<source-resource-group-name>"
$destinationAccountRG = "<destination-resource-group-name>"
$sourceAccount = "<cosmos-source-account-name>"
$destinationAccount = "<cosmos-destination-account-name>"
$jobName = ""
$sourceDatabase = ""
$sourceContainer = ""
$destinationDatabase = ""
$destinationContainer = ""

Assign read permission

Note

This step is not required if you're copying data within same Azure Cosmos DB account.

While copying data from one account's container to another account's container, it is required to give read access of source container to destination account's identity to perform the copy operation. Follow the steps below to assign requisite read permission to destination account.

Using System managed identity

1. Set destination subscription context

   az account set --subscription $destinationSubId
   

2. Add system identity on destination account

   $identityOutput = az cosmosdb identity assign -n $destinationAccount -g $destinationAccountRG
   $principalId = ($identityOutput | ConvertFrom-Json).principalId
   

3. Set default identity on destination account

   az cosmosdb update -n $destinationAccount -g $destinationAccountRG --default-identity="SystemAssignedIdentity"
   

4. Set source subscription context

   az account set --subscription $sourceSubId
   

5. Add role assignment on source account

   # Read-only access role
   $roleDefinitionId = "00000000-0000-0000-0000-000000000001" 
   az cosmosdb sql role assignment create --account-name $sourceAccount --resource-group $sourceAccountRG --role-definition-id $roleDefinitionId --scope "/" --principal-id $principalId
   

6. Reset destination subscription context

   az account set --subscription $destinationSubId
   

Using User-assigned managed identity

1. Assign User-assigned managed identity variable

   $userAssignedManagedIdentityResourceId = "<CompleteResourceIdOfUserAssignedManagedIdentity>"
   

2. Set destination subscription context

   az account set --subscription $destinationSubId
   

3. Add user assigned managed identity on destination account

   $identityOutput = az cosmosdb identity assign -n $destinationAccount -g $destinationAccountRG --identities $userAssignedManagedIdentityResourceId
   $principalId = ($identityOutput | ConvertFrom-Json).userAssignedIdentities.$userAssignedManagedIdentityResourceId.principalId
   

4. Set default identity on destination account

   az cosmosdb update -n $destinationAccount -g $destinationAccountRG --default-identity=UserAssignedIdentity=$userAssignedManagedIdentityResourceId
   

5. Set source subscription context

   az account set --subscription $sourceSubId
   

6. Add role assignment on source account

   $roleDefinitionId = "00000000-0000-0000-0000-000000000001"  # Read-only access role
   az cosmosdb sql role assignment create --account-name $sourceAccount --resource-group $sourceAccountRG --role-definition-id $roleDefinitionId --scope "/" --principal-id $principalId
   

7. Reset destination subscription context

   az account set --subscription $destinationSubId
   

Create copy job

az cosmosdb copy create `
    --resource-group $destinationAccountRG `
    --job-name $jobName `
    --dest-account $destinationAccount `
    --src-account $sourceAccount `
    --dest-nosql database=$destinationDatabase container=$destinationContainer `
    --src-nosql database=$sourceDatabase container=$sourceContainer

Note

--job-name should be unique for each job within an account.

Set shell variables

First, set all of the variables that each individual script uses.

$destinationRG = "<destination-resource-group-name>"
$sourceAccount = "<cosmos-source-account-name>"
$destinationAccount = "<cosmos-destination-account-name>"
$jobName = ""
$sourceDatabase = ""
$sourceCollection = ""
$destinationDatabase = ""
$destinationCollection = ""

Create copy job

Create a job to copy a collection within an Azure Cosmos DB API for MongoDB account:

az cosmosdb copy create `
    --resource-group $destinationRG `
    --job-name $jobName `
    --dest-account $destinationAccount `
    --src-account $sourceAccount `
    --dest-mongo database=$destinationDatabase collection=$destinationCollection `
    --src-mongo database=$sourceDatabase collection=$sourceCollection 

Note

--job-name should be unique for each job within an account.

Set shell variables

First, set all of the variables that each individual script uses.

$destinationRG = "<destination-resource-group-name>"
$sourceAccount = "<cosmos-source-account-name>"
$destinationAccount = "<cosmos-destination-account-name>"
$jobName = ""
$sourceKeySpace = ""
$sourceTable = ""
$destinationKeySpace = ""
$destinationTable = ""

Create copy job

Create job to copy a table within an Azure Cosmos DB for Apache Cassandra account:

az cosmosdb copy create `
    --resource-group $destinationRG `
    --job-name $jobName `
    --dest-account $destinationAccount `
    --src-account $sourceAccount `
    --dest-cassandra keyspace=$destinationKeySpace table=$destinationTable `
    --src-cassandra keyspace=$sourceKeySpace table=$sourceTable 

Note

--job-name should be unique for each job within an account.

Managing copy jobs

Monitor the progress of a copy job

View the progress and status of a copy job:

az cosmosdb copy show `
    --resource-group $destinationAccountRG `
    --account-name $destinationAccount `
    --job-name $jobName

List all the copy jobs created in an account

To list all the copy jobs created in an account:

az cosmosdb copy list `
    --resource-group $destinationAccountRG `
    --account-name $destinationAccount

Pause a copy job

In order to pause an ongoing copy job, you can use the command:

az cosmosdb copy pause `
    --resource-group $destinationAccountRG `
    --account-name $destinationAccount `
    --job-name $jobName

Resume a copy job

In order to resume an ongoing copy job, you can use the command:

az cosmosdb copy resume `
    --resource-group $destinationAccountRG `
    --account-name $destinationAccount `
    --job-name $jobName

Cancel a copy job

In order to cancel an ongoing copy job, you can use the command:

az cosmosdb copy cancel `
    --resource-group $destinationAccountRG `
    --account-name $destinationAccount `
    --job-name $jobName

Get support for copy issues

For issues related to a copy job, raise a New Support Request from the Azure portal. Set the Problem Type as 'Data Migration' and Problem subtype as 'Container copy'.

Next steps

• For more information about container copy jobs, see Copy jobs.