Migrate data to Azure Cosmos DB using the desktop data migration tool

APPLIES TO: NoSQL MongoDB Table

The Azure Cosmos DB desktop data migration tool is an open-source command-line application to import or export data from Azure Cosmos DB. The tool can migrate data to and from many sources and sinks including, but not limited to:

• Azure Cosmos DB for NoSQL
• Azure Cosmos DB for MongoDB
• Azure Cosmos DB for Table
• Azure Table storage
• JSON
• MongoDB
• SQL Server

Important

For this guide, you will perform a data migration from JSON to Azure Cosmos DB for NoSQL.

Prerequisites

• An existing Azure Cosmos DB for NoSQL account.
  • If you have an Azure subscription, create a new account.
  • If you don't have an Azure subscription, create a free account before you begin.
  • Alternatively, you can try Azure Cosmos DB free before you commit.
• Latest version of Azure CLI.
• .NET 6.0 or later.

Install the desktop data migration tool

First, install the latest version of the desktop data migration tool from the GitHub repository.

Note

The desktop data migration tool requires .NET 6.0 or later on your local machine.

1. In your browser, navigate to the Releases section of the repository: azurecosmosdb/data-migration-desktop-tool/releases.

2. Download the latest compressed folder for your platform. There are compressed folders for the win-x64, mac-x64, and linux-x64 platforms.

3. Extract the files to an install location on your local machine.

4. (Optional) Add the desktop data migration tool to the PATH environment variable of your local machine.

Prepare your migration target

Next, create a target database and container on the Azure Cosmos DB for NoSQL account.

Azure CLI

1. Open a new terminal. If you haven't already, sign in to the Azure CLI.

2. Create new shell variables for the Azure Cosmos DB account's name and resource group.

   # Variable for Azure Cosmos DB account name
   accountName="<name-of-existing-account>"
   
   # Variable for resource group name
   resourceGroupName="<name-of-existing-resource-group>"
   

3. Create a new database using az cosmosdb sql database create. Name the new database cosmicworks and configure the database with 400 RU/s of shared throughput.

   az cosmosdb sql database create \
       --resource-group $resourceGroupName \
       --account-name $accountName \
       --name cosmicworks \
       --throughput 400
   

4. Use az cosmosdb sql container create to create a new container named products within the cosmicworks database. Set the partition key path of the new container to /category.

   az cosmosdb sql container create \
       --resource-group $resourceGroupName \
       --account-name $accountName \
       --database-name cosmicworks \
       --name products \
       --partition-key-path "/category"
   

5. Find the primary connection string from the list of keys for the account with az cosmosdb keys list.

   az cosmosdb keys list \
       --resource-group $resourceGroupName \
       --name $accountName \
       --type connection-strings
   

6. Record the primary connection string value. You use this credential later when migrating data with the tool.

Azure PowerShell

1. Open a new terminal. If you haven't already, sign in to the Azure CLI.

2. Create new variables for the Azure Cosmos DB account's name and resource group.

   # Variable for Azure Cosmos DB account name
   $ACCOUNT_NAME = "<name-of-existing-account>"
   
   # Variable for resource group name
   $RESOURCE_GROUP_NAME = "<name-of-existing-resource-group>"
   

3. Create a new database using New-AzCosmosDBSqlDatabase. Name the new database cosmicworks and configure the database with 400 RU/s of shared throughput.

   $parameters = @{
       ResourceGroupName = $RESOURCE_GROUP_NAME
       AccountName = $ACCOUNT_NAME
       Name = "cosmicworks"
       Throughput = 400
   }
   New-AzCosmosDBSqlDatabase @parameters
   

4. Use New-AzCosmosDBSqlContainer to create a new container named products within the cosmicworks database. Set the partition key path of the new container to /category.

   $parameters = @{
       ResourceGroupName = $RESOURCE_GROUP_NAME
       AccountName = $ACCOUNT_NAME
       DatabaseName = "cosmicworks"
       Name = "products"
       PartitionKeyPath = "/category"
       PartitionKeyKind = "Hash"
   }
   New-AzCosmosDBSqlContainer @parameters
   

5. Find the primary connection string from the list of keys for the account with Get-AzCosmosDBAccountKey.

   $parameters = @{
       ResourceGroupName = $RESOURCE_GROUP_NAME
       Name = $ACCOUNT_NAME
       Type = "ConnectionStrings"
   }
   Get-AzCosmosDBAccountKey @parameters
   

6. Record the primary connection string value. You use this credential later when migrating data with the tool.

Perform a migration operation

Now, migrate data from a JSON array to the newly created Azure Cosmos DB for NoSQL container.

1. Navigate to an empty directory on your local machine. Within that directory, create a new file named migrationsettings.json.

2. Within the JSON file, create a new empty JSON object:

   {}
   

3. Create a new property named Source with the value json. Create another new property named SourceSettings with an empty object as the value.

   {
     "Source": "json",
     "SourceSettings": {}
   }
   

4. Within the SourceSettings object, create a new property named FilePath with the value set to this URI: https://raw.githubusercontent.com/azure-samples/cosmos-db-migration-sample-data/main/nosql-data.json.

   {
     "Source": "json",
     "SourceSettings": {
       "FilePath": "https://raw.githubusercontent.com/azure-samples/cosmos-db-migration-sample-data/main/nosql-data.json"
     }
   }
   

5. Create another new property named Sink with the value cosmos-nosql. Also, create a property named SinkSettings with an empty object.

   {
     "Source": "json",
     "SourceSettings": {
       "FilePath": "https://raw.githubusercontent.com/azure-samples/cosmos-db-migration-sample-data/main/nosql-data.json"
     },
     "Sink": "cosmos-nosql",
     "SinkSettings": {
     }
   }
   

6. Within SinkSettings, create a property named ConnectionString with the primary connection string you recorded earlier in this guide as its value.

   {
     "Source": "json",
     "SourceSettings": {
       "FilePath": "https://raw.githubusercontent.com/azure-samples/cosmos-db-migration-sample-data/main/nosql-data.json"
     },
     "Sink": "cosmos-nosql",
     "SinkSettings": {
       "ConnectionString": "<connection-string-for-existing-account>"
     }
   }
   

7. Add Database, Container, and PartitionKeyPath properties with cosmicworks, products, and /category as their values respectively.

   {
     "Source": "json",
     "SourceSettings": {
       "FilePath": "https://raw.githubusercontent.com/azure-samples/cosmos-db-migration-sample-data/main/nosql-data.json"
     },
     "Sink": "cosmos-nosql",
     "SinkSettings": {
       "ConnectionString": "<connection-string-for-existing-account>",
       "Database": "cosmicworks",
       "Container": "products",
       "PartitionKeyPath": "/category"
     }
   }
   

8. Save the migrationsettings.json file.

9. Open a new terminal and navigate to the directory containing your migrationsettings.json file.

10. Run the desktop data migration tool using the dmt command.

    dmt
    

    Note

    If you did not add the installation path to your PATH environment variable, you may need to specify the full path to the dmt executable.

11. The tool now outputs the sources and sinks used by the migration.

    Using JSON Source
    Using Cosmos-nosql Sink