Get started: Free trial & setup

  • Article
  • 11 minutes to read

If you’re new to Azure Databricks, you’ve found the place to start. Learn how to set up a free trial and cloud account. For information about more online training resources, see Get free Databricks training.

Create an Azure Databricks workspace

Databricks recommends you deploy your first Azure Databricks workspace using the Azure portal. You can also deploy Azure Databricks with one of the following options:

Note

When you create your Azure Databricks workspace, you can select the Trial (Premium - 14-Days Free DBUs) pricing tier to give the workspace access to free Premium Azure Databricks DBUs for 14 days.

Before you begin

Use the portal to create an Azure Databricks workspace

  1. In the Azure portal, select Create a resource > Analytics > Azure Databricks.

  2. Under Azure Databricks Service, provide the values to create a Databricks workspace.

    Property Description
    Workspace name Provide a name for your Databricks workspace
    Subscription From the drop-down, select your Azure subscription.
    Resource group Specify whether you want to create a new resource group or use an existing one. A resource group is a container that holds related resources for an Azure solution. For more information, see Azure Resource Group overview.
    Location Select West US 2. For other available regions, see Azure services available by region.
    Pricing Tier Choose between Standard, Premium, or Trial. For more information on these tiers, see Databricks pricing page.

  3. Select Review + Create, and then Create. The workspace creation takes a few minutes. During workspace creation, you can view the deployment status in Notifications. Once this process is finished, your user account is automatically added as an admin user in the workspace.

Note

When a workspace deployment fails, the workspace is still created in a failed state. Delete the failed workspace and create a new workspace that resolves the deployment errors. When you delete the failed workspace, the managed resource group and any successfully deployed resources are also deleted.

Deploy Azure Databricks with the Azure CLI

Step 1: Sign in

Sign in using the az login command if you’re using a local install of the CLI.

az login

Follow the steps displayed in your terminal to complete the authentication process.

Step 2: Install the Azure CLI extension

When working with extension references for the Azure CLI, you must first install the extension. Azure CLI extensions give you access to experimental and pre-release commands that have not yet shipped as part of the core CLI. To learn more about extensions including updating and uninstalling, see Use extensions with Azure CLI.

Install the extension for databricks by running the following command:

az extension add --name databricks

Step 3: Create a resource group

Azure Databricks, like all Azure resources, must be deployed into a resource group. Resource groups allow you to organize and manage related Azure resources.

For this quickstart, create a resource group named _ databricks-quickstart _ in the westus2 location with the following az group create command:

az group create --name databricks-quickstart --location westus2

Step 4: Create an Azure Databricks workspace

Use the az databricks workspace create create an Azure Databricks workspace.

az databricks workspace create
    --resource-group databricks-quickstart \
    --name mydatabricksws  \
    --location westus  \
    --sku standard

Deploy Azure Databricks with PowerShell

Note

If you choose to use PowerShell locally, this article requires that you install the Az PowerShell module and connect to your Azure account using the Connect-AzAccount cmdlet. For more information about installing the Az PowerShell module, see Install Azure PowerShell.

Important

While the Az.Databricks PowerShell module is in preview, you must install it separately from the Az PowerShell module using the following command: Install-Module -Name Az.Databricks -AllowPrerelease. Once the Az.Databricks PowerShell module is generally available, it becomes part of future Az PowerShell module releases and available natively from within Azure Cloud Shell.

Note

If you want to create an Azure Databricks workspace in the Azure Commercial Cloud that holds US Government compliance certifications like FedRAMP High, please reach out to your Microsoft or Databricks representative to gain access to this experience.

If this is your first time using Azure Databricks, you must register the Microsoft.Databricks resource provider.

  Register-AzResourceProvider -ProviderNamespace Microsoft.Databricks

Use Azure Cloud Shell

Azure hosts Azure Cloud Shell, an interactive shell environment that you can use through your browser. You can use either Bash or PowerShell with Cloud Shell to work with Azure services. You can use the Cloud Shell preinstalled commands to run the code in this article without having to install anything on your local environment.

To start Azure Cloud Shell:

  • Select Try It in the upper-right corner of a code block. Selecting Try It doesn’t automatically copy the code to Cloud Shell.
  • Go to https://shell.azure.com, or select the Launch Cloud Shell button to open Cloud Shell in your browser.
  • Select the Cloud Shell button on the menu bar at the upper right in the Azure portal.

To run the code in this article in Azure Cloud Shell:

  • Start Cloud Shell.
  • Select the Copy button on a code block to copy the code.
  • Paste the code into the Cloud Shell session by selecting Ctrl+Shift+V on Windows and Linux or by selecting Cmd+Shift+V on macOS.
  • Select Enter to run the code.

If you have multiple Azure subscriptions, choose the appropriate subscription in which the resources should be billed. Select a specific subscription ID using the Set-AzContext cmdlet.

Set-AzContext -SubscriptionId 00000000-0000-0000-0000-000000000000

Create a resource group

Create an Azure resource group using the New-AzResourceGroup cmdlet. A resource group is a logical container in which Azure resources are deployed and managed as a group.

The following example creates a resource group named myresourcegroup in the West US 2 region.

New-AzResourceGroup -Name myresourcegroup -Location westus2

Create an Azure Databricks workspace

In this section, you create an Azure Databricks workspace using PowerShell.

New-AzDatabricksWorkspace -Name mydatabricksws -ResourceGroupName myresourcegroup -Location westus2 -ManagedResourceGroupName databricks-group -Sku standard

Provide the following values:

Property Description
Name Provide a name for your Databricks workspace
ResourceGroupName Specify an existing resource group name
Location Select West US 2. For other available regions, see Azure services available by region
ManagedResourceGroupName Specify whether you want to create a new managed resource group or use an existing one.
Sku Choose between Standard, Premium, or Trial. For more information on these tiers, see Databricks pricing

The workspace creation takes a few minutes. Once this process is finished, your user account is automatically added as an admin user in the workspace.

Note

When a workspace deployment fails, the workspace is still created in a failed state. Delete the failed workspace and create a new workspace that resolves the deployment errors. When you delete the failed workspace, the managed resource group and any successfully deployed resources are also deleted.

Determine the provisioning state of a Databricks workspace

To determine if a Databricks workspace was provisioned successfully, you can use the Get-AzDatabricksWorkspace cmdlet.

Get-AzDatabricksWorkspace -Name mydatabricksws -ResourceGroupName myresourcegroup |
  Select-Object -Property Name, SkuName, Location, ProvisioningState

Name            SkuName   Location  ProvisioningState
----            -------   --------  -----------------
mydatabricksws  standard  westus2   Succeeded

Deploy Azure Databricks with an ARM template

An ARM template is a JavaScript Object Notation (JSON) file that defines the infrastructure and configuration for your project. The template uses declarative syntax, which lets you state what you intend to deploy without having to write the sequence of programming commands to create it.

If your environment meets the prerequisites and you’re familiar with using ARM templates, select the Deploy to Azure button. The template will open in the Azure portal.

Review the template

The template used in this quickstart is from Azure Quickstart Templates.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.5.6.12127",
      "templateHash": "14509124136721506545"
    }
  },
  "parameters": {
    "disablePublicIp": {
      "type": "bool",
      "defaultValue": false,
      "metadata": {
        "description": "Specifies whether to deploy Azure Databricks workspace with Secure Cluster Connectivity (No Public IP) enabled or not"
      }
    },
    "workspaceName": {
      "type": "string",
      "metadata": {
        "description": "The name of the Azure Databricks workspace to create."
      }
    },
    "pricingTier": {
      "type": "string",
      "defaultValue": "premium",
      "allowedValues": [
        "standard",
        "premium"
      ],
      "metadata": {
        "description": "The pricing tier of workspace."
      }
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]",
      "metadata": {
        "description": "Location for all resources."
      }
    }
  },
  "variables": {
    "managedResourceGroupName": "[format('databricks-rg-{0}-{1}', parameters('workspaceName'), uniqueString(parameters('workspaceName'), resourceGroup().id))]"
  },
  "resources": [
    {
      "type": "Microsoft.Databricks/workspaces",
      "apiVersion": "2018-04-01",
      "name": "[parameters('workspaceName')]",
      "location": "[parameters('location')]",
      "sku": {
        "name": "[parameters('pricingTier')]"
      },
      "properties": {
        "managedResourceGroupId": "[subscriptionResourceId('Microsoft.Resources/resourceGroups', variables('managedResourceGroupName'))]",
        "parameters": {
          "enableNoPublicIp": {
            "value": "[parameters('disablePublicIp')]"
          }
        }
      }
    }
  ],
  "outputs": {
    "workspace": {
      "type": "object",
      "value": "[reference(resourceId('Microsoft.Databricks/workspaces', parameters('workspaceName')))]"
    }
  }
}

The Azure resource defined in the template is Microsoft.Databricks/workspaces: create an Azure Databricks workspace.

Deploy the template

In this section, you create an Azure Databricks workspace using an ARM template.

  • Use the provided link to sign in to Azure and open a template.

  • Provide the following required values to create your Azure Databricks workspace:

    Property Description
    Subscription From the drop-down, select your Azure subscription.
    Resource group Specify whether you want to create a new resource group or use an existing one. A resource group is a container that holds related resources for an Azure solution. For more information, see Azure Resource Group overview.
    Location Select East US 2. For other available regions, see Azure services available by region.
    Workspace name Provide a name for your Databricks workspace
    Pricing Tier Choose between Standard or Premium. For more information on these tiers, see Databricks pricing page.

  • Select Review + Create, then Create.

  • The workspace creation takes a few minutes. When a workspace deployment fails, the workspace is still created in a failed state. Delete the failed workspace and create a new workspace that resolves the deployment errors. When you delete the failed workspace, the managed resource group and any successfully deployed resources are also deleted.

Review deployed resources

You can either use the Azure portal to check the Azure Databricks workspace or use the following Azure CLI or Azure PowerShell script to list the resource.

Azure CLI

echo "Enter your Azure Databricks workspace name:" &&
read databricksWorkspaceName &&
echo "Enter the resource group where the Azure Databricks workspace exists:" &&
read resourcegroupName &&
az databricks workspace show -g $resourcegroupName -n $databricksWorkspaceName

Azure PowerShell

$resourceGroupName = Read-Host -Prompt "Enter the resource group name where your Azure Databricks workspace exists"
(Get-AzResource -ResourceType "Microsoft.Databricks/workspaces" -ResourceGroupName $resourceGroupName).Name
 Write-Host "Press [ENTER] to continue..."

Deploy Azure Databricks with Bicep

Bicep is a domain-specific language (DSL) that uses declarative syntax to deploy Azure resources. It provides concise syntax, reliable type safety, and support for code reuse. Bicep offers the best authoring experience for your infrastructure-as-code solutions in Azure.

Review the Bicep file

The Bicep file used in this quickstart is from Azure Quickstart Templates.

@description('Specifies whether to deploy Azure Databricks workspace with Secure Cluster Connectivity (No Public IP) enabled or not')
param disablePublicIp bool = false

@description('The name of the Azure Databricks workspace to create.')
param workspaceName string

@description('The pricing tier of workspace.')
@allowed([
  'standard'
  'premium'
])
param pricingTier string = 'premium'

@description('Location for all resources.')
param location string = resourceGroup().location

var managedResourceGroupName = 'databricks-rg-${workspaceName}-${uniqueString(workspaceName, resourceGroup().id)}'

resource ws 'Microsoft.Databricks/workspaces@2018-04-01' = {
  name: workspaceName
  location: location
  sku: {
    name: pricingTier
  }
  properties: {
    managedResourceGroupId: managedResourceGroup.id
    parameters: {
      enableNoPublicIp: {
        value: disablePublicIp
      }
    }
  }
}

resource managedResourceGroup 'Microsoft.Resources/resourceGroups@2021-04-01' existing = {
  scope: subscription()
  name: managedResourceGroupName
}

output workspace object = ws.properties

The Azure resource defined in the Bicep file is Microsoft.Databricks/workspaces: create an Azure Databricks workspace.

Deploy the Bicep file

  • Save the Bicep file as main.bicep to your local computer.
  • Deploy the Bicep file using either Azure CLI or Azure PowerShell.

CLI

az group create --name exampleRG --location eastus
az deployment group create --resource-group exampleRG --template-file main.bicep --parameters workspaceName=<workspace-name>

PowerShell

New-AzResourceGroup -Name exampleRG -Location eastus
New-AzResourceGroupDeployment -ResourceGroupName exampleRG -TemplateFile ./main.bicep -workspaceName "<workspace-name>"

Note

Replace <workspace-name> with the name of the Azure Databricks workspace you want to create.

When the deployment finishes, you should see a message indicating the deployment succeeded.

Review deployed resources

Use the Azure portal, Azure CLI, or Azure PowerShell to list the deployed resources in the resource group.

CLI

az resource list --resource-group exampleRG

PowerShell

Get-AzResource -ResourceGroupName exampleRG

Configure workspace and data permissions

Azure Databricks restricts certain activities to workspace administrators to ensure that customers are able to securely configure access to data and code. Some Azure Databricks features must be enabled by a workspace administrator before users can use them. For common tasks, see Get started with Azure Databricks administration.

If you’re in a Unity Catalog-enabled workspace, you can leverage a number of additional features for security and governance. See Tutorial: Unity Catalog metastore admin tasks for Databricks SQL.

Administrators need to configure Databricks SQL before most common tasks can be completed. See Admin onboarding for Databricks SQL and Set up a user to query a table.

Note

When you create a Azure Databricks workspace, you are granted administrator privileges.