Create and configure Enterprise Security Package clusters in Azure HDInsight

Enterprise Security Package (ESP) for Azure HDInsight gives you access to Active Directory-based authentication, multiuser support, and role-based access control for your Apache Hadoop clusters in Azure. HDInsight ESP clusters enable organizations that adhere to strict corporate security policies to process sensitive data securely.

This guide shows how to create an ESP-enabled Azure HDInsight cluster. It also shows how to create a Windows IaaS VM on which Active Directory and Domain Name System (DNS) are enabled. Use this guide to configure the necessary resources to allow on-premises users to sign in to an ESP-enabled HDInsight cluster.

The server you create will act as a replacement for your actual on-premises environment. You'll use it for the setup and configuration steps. Later you'll repeat the steps in your own environment.

This guide will also help you create a hybrid identity environment by using password hash sync with Azure Active Directory (Azure AD). The guide complements Use ESP in HDInsight.

Before you use this process in your own environment:

  • Set up Active Directory and DNS.
  • Enable Azure AD.
  • Sync on-premises user accounts to Azure AD.

Azure AD architecture diagram

Create an on-premises environment

In this section, you'll use an Azure Quickstart deployment template to create new VMs, configure DNS, and add a new Active Directory forest.

  1. Go to the Quickstart deployment template to Create an Azure VM with a new Active Directory forest.

  2. Select Deploy to Azure.

  3. Sign in to your Azure subscription.

  4. On the Create an Azure VM with a new AD Forest page, provide the following information:

    Property Value
    Subscription Select the subscription where you want to deploy the resources.
    Resource group Select Create new, and enter the name OnPremADVRG
    Location Select a location.
    Admin Username HDIFabrikamAdmin
    Admin Password Enter a password.
    Domain Name
    Dns Prefix hdifabrikam

    Leave the remaining default values.

    Template for Create an Azure VM with a new Azure AD Forest

  5. Review the Terms and Conditions, and then select I agree to the terms and conditions stated above.

  6. Select Purchase, and monitor the deployment and wait for it to complete. The deployment takes about 30 minutes to complete.

Configure users and groups for cluster access

In this section, you'll create the users that will have access to the HDInsight cluster by the end of this guide.

  1. Connect to the domain controller by using Remote Desktop.

    1. From the Azure portal, navigate to Resource groups > OnPremADVRG > adVM > Connect.
    2. From the IP address drop-down list, select the public IP address.
    3. Select Download RDP File, and then open the file.
    4. Use HDIFabrikam\HDIFabrikamAdmin as the user name.
    5. Enter the password that you chose for the admin account.
    6. Select OK.
  2. From the domain controller Server Manager dashboard, navigate to Tools > Active Directory Users and Computers.

    On the Server Manager dashboard, open Active Directory Management

  3. Create two new users: HDIAdmin and HDIUser. These two users will sign in to HDInsight clusters.

    1. From the Active Directory Users and Computers page, right-click, and then navigate to New > User.

      Create a new Active Directory user

    2. On the New Object - User page, enter HDIUser for First name and User logon name. The other fields will autopopulate. Then select Next.

      Create the first admin user object

    3. In the pop-up window that appears, enter a password for the new account. Select Password never expires, and then OK at the pop-up message.

    4. Select Next, and then Finish to create the new account.

    5. Repeat the above steps to create the user HDIAdmin.

      Create a second admin user object

  4. Create a global security group.

    1. From Active Directory Users and Computers, right-click, and then navigate to New > Group.

    2. Enter HDIUserGroup in the Group name text box.

    3. Select OK.

    Create a new Active Directory group

    Create a new object

  5. Add members to HDIUserGroup.

    1. Right-click HDIUser and select Add to a group....

    2. In the Enter the object names to select text box, enter HDIUserGroup. Then select OK, and OK again at the pop-up.

    3. Repeat the previous steps for the HDIAdmin account.

      Add the member HDIUser to the group HDIUserGroup

You've now created your Active Directory environment. You've added two users and a user group that can access the HDInsight cluster.

The users will be synchronized with Azure AD.

Create an Azure AD directory

  1. Sign in to the Azure portal.

  2. Select Create a resource and type directory. Select Azure Active Directory > Create.

  3. Under Organization name, enter HDIFabrikam.

  4. Under Initial domain name, enter HDIFabrikamoutlook.

  5. Select Create.

    Create an Azure AD directory

Create a custom domain

  1. From your new Azure Active Directory, under Manage, select Custom domain names.

  2. Select + Add custom domain.

  3. Under Custom domain name, enter, and then select Add domain.

  4. Then complete Add your DNS information to the domain registrar.

    Create a custom domain

Create a group

  1. From your new Azure Active Directory, under Manage, select Groups.
  2. Select + New group.
  3. In the group name text box, enter AAD DC Administrators.
  4. Select Create.

Configure your Azure AD tenant

Now you'll configure your Azure AD tenant so that you can synchronize users and groups from the on-premises Active Directory instance to the cloud.

Create an Active Directory tenant administrator.

  1. Sign in to the Azure portal and select your Azure AD tenant, HDIFabrikam.

  2. Navigate to Manage > Users > New user.

  3. Enter the following details for the new user:


    Property Description
    User name Enter fabrikamazureadmin in the text box. From the domain name drop-down list, select
    Name Enter fabrikamazureadmin.


    1. Select Let me create the password.
    2. Enter a secure password of your choice.

    Groups and roles

    1. Select 0 groups selected.

    2. Select AAD DC Administrators, and then Select.

      The Azure AD Groups dialog box

    3. Select User.

    4. Select Global administrator, and then Select.

      The Azure AD role dialog box

  4. Select Create.

  5. Then have the new user sign in to the Azure portal where it will be prompted to change the password. You'll need to do this before configuring Microsoft Azure Active Directory Connect.

Sync on-premises users to Azure AD

Configure Microsoft Azure Active Directory Connect

  1. From the domain controller, download Microsoft Azure Active Directory Connect.

  2. Open the executable file that you downloaded, and agree to the license terms. Select Continue.

  3. Select Use express settings.

  4. On the Connect to Azure AD page, enter the username and password of the global administrator for Azure AD. Use the username that you created when you configured your Active Directory tenant. Then select Next.

    Connect to Azure A D

  5. On the Connect to Active Directory Domain Services page, enter the username and password for an enterprise admin account. Use the username HDIFabrikam\HDIFabrikamAdmin and its password that you created earlier. Then select Next.

    Connect to A D D S page.

  6. On the Azure AD sign-in configuration page, select Next.

    Azure AD sign-in configuration page

  7. On the Ready to configure page, select Install.

    Ready to configure page

  8. On the Configuration complete page, select Exit. Configuration complete page

  9. After the sync completes, confirm that the users you created on the IaaS directory are synced to Azure AD.

    1. Sign in to the Azure portal.
    2. Select Azure Active Directory > HDIFabrikam > Users.

Create a user-assigned managed identity

Create a user-assigned managed identity that you can use to configure Azure AD Domain Services (Azure AD DS). For more information, see Create, list, delete, or assign a role to a user-assigned managed identity by using the Azure portal.

  1. Sign in to the Azure portal.
  2. Select Create a resource and type managed identity. Select User Assigned Managed Identity > Create.
  3. For the Resource Name, enter HDIFabrikamManagedIdentity.
  4. Select your subscription.
  5. Under Resource group, select Create new and enter HDIFabrikam-CentralUS.
  6. Under Location, select Central US.
  7. Select Create.

Create a new user-assigned managed identity

Enable Azure AD DS

Follow these steps to enable Azure AD DS. For more information, see Enable Azure AD DS by using the Azure portal.

  1. Create a virtual network to host Azure AD DS. Run the following PowerShell code.

    # Sign in to your Azure subscription
    $sub = Get-AzSubscription -ErrorAction SilentlyContinue
    # If you have multiple subscriptions, set the one to use
    # Select-AzSubscription -SubscriptionId "<SUBSCRIPTIONID>"
    $virtualNetwork = New-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-CentralUS' -Location 'Central US' -Name 'HDIFabrikam-AADDSVNET' -AddressPrefix
    $subnetConfig = Add-AzVirtualNetworkSubnetConfig -Name 'AADDS-subnet' -AddressPrefix -VirtualNetwork $virtualNetwork
    $virtualNetwork | Set-AzVirtualNetwork
  2. Sign in to the Azure portal.

  3. Select Create resource, enter Domain services, and select Azure AD Domain Services > Create.

  4. On the Basics page:

    1. Under Directory name, select the Azure AD directory you created: HDIFabrikam.

    2. For DNS domain name, enter

    3. Select your subscription.

    4. Specify the resource group HDIFabrikam-CentralUS. For Location, select Central US.

      Azure AD DS basic details

  5. On the Network page, select the network (HDIFabrikam-VNET) and the subnet (AADDS-subnet) that you created by using the PowerShell script. Or choose Create new to create a virtual network now.

    Create virtual network step

  6. On the Administrator group page, you should see a notification that a group named AAD DC Administrators has already been created to administer this group. You can modify the membership of this group if you want to, but in this case you don't need to change it. Select OK.

    View the Azure AD administrator group

  7. On the Synchronization page, enable complete synchronization by selecting All > OK.

    Enable Azure AD DS synchronization

  8. On the Summary page, verify the details for Azure AD DS and select OK.

    Enable Azure AD Domain Services

After you enable Azure AD DS, a local DNS server runs on the Azure AD VMs.

Configure your Azure AD DS virtual network

Use the following steps to configure your Azure AD DS virtual network (HDIFabrikam-AADDSVNET) to use your custom DNS servers.

  1. Locate the IP addresses of your custom DNS servers.

    1. Select the Azure AD DS resource.
    2. Under Manage, select Properties.
    3. Find the IP addresses under IP address on virtual network.

    Locate custom DNS IP addresses for Azure AD DS

  2. Configure HDIFabrikam-AADDSVNET to use custom IP addresses and

    1. Under Settings, select DNS Servers.
    2. Select Custom.
    3. In the text box, enter the first IP address (
    4. Select Save.
    5. Repeat the steps to add the other IP address (

In our scenario, we configured Azure AD DS to use IP addresses and, setting the same IP address on the Azure AD DS virtual network:

The custom DNS servers page

Securing LDAP traffic

Lightweight Directory Access Protocol (LDAP) is used to read from and write to Azure Active Directory. You can make LDAP traffic confidential and secure by using Secure Sockets Layer (SSL) or Transport Layer Security (TLS) technology. You can enable LDAP over SSL (LDAPS) by installing a properly formatted certificate.

For more information about secure LDAP, see Configure LDAPS for an Azure AD DS managed domain.

In this section, you create a self-signed certificate, download the certificate, and configure LDAPS for the HDIFabrikam Azure AD DS managed domain.

The following script creates a certificate for HDIFabrikam. The certificate is saved in the LocalMachine path.

$lifetime = Get-Date
New-SelfSignedCertificate -Subject `
-NotAfter $lifetime.AddDays(365) -KeyUsage DigitalSignature, KeyEncipherment `
-Type SSLServerAuthentication -DnsName *,


Any utility or application that creates a valid Public Key Cryptography Standards (PKCS) #10 request can be used to form the TLS/SSL certificate request.

Verify that the certificate is installed in the computer's Personal store:

  1. Start Microsoft Management Console (MMC).

  2. Add the Certificates snap-in that manages certificates on the local computer.

  3. Expand Certificates (Local Computer) > Personal > Certificates. A new certificate should exist in the Personal store. This certificate is issued to the fully qualified host name.

    Verify local certificate creation

  4. In pane on the right, right-click the certificate that you created. Point to All Tasks, and then select Export.

  5. On the Export Private Key page, select Yes, export the private key. The computer where the key will be imported needs the private key to read the encrypted messages.

    The Export Private Key page of the Certificate Export Wizard

  6. On the Export File Format page, leave the default settings, and then select Next.

  7. On the Password page, type a password for the private key. For Encryption, select TripleDES-SHA1. Then select Next.

  8. On the File to Export page, type the path and the name for the exported certificate file, and then select Next. The file name has to have a .pfx extension. This file is configured in the Azure portal to establish a secure connection.

  9. Enable LDAPS for an Azure AD DS managed domain.

    1. From the Azure portal, select the domain
    2. Under Manage, select Secure LDAP.
    3. On the Secure LDAP page, under Secure LDAP, select Enable.
    4. Browse for the .pfx certificate file that you exported on your computer.
    5. Enter the certificate password.

    Enable secure LDAP

  10. Now that you've enabled LDAPS, make sure it's reachable by enabling port 636.

    1. In the HDIFabrikam-CentralUS resource group, select the network security group

    2. Under Settings, select Inbound security rules > Add.

    3. On the Add inbound security rule page, enter the following properties, and select Add:

      Property Value
      Source Any
      Source port ranges *
      Destination Any
      Destination port range 636
      Protocol Any
      Action Allow
      Priority <Desired number>
      Name Port_LDAP_636

      The Add inbound security rule dialog box

HDIFabrikamManagedIdentity is the user-assigned managed identity. The HDInsight Domain Services Contributor role is enabled for the managed identity that will allow this identity to read, create, modify, and delete domain services operations.

Create a user-assigned managed identity

Create an ESP-enabled HDInsight cluster

This step requires the following prerequisites:

  1. Create a new resource group HDIFabrikam-WestUS in the location West US.

  2. Create a virtual network that will host the ESP-enabled HDInsight cluster.

    $virtualNetwork = New-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-WestUS' -Location 'West US' -Name 'HDIFabrikam-HDIVNet' -AddressPrefix
    $subnetConfig = Add-AzVirtualNetworkSubnetConfig -Name 'SparkSubnet' -AddressPrefix -VirtualNetwork $virtualNetwork
    $virtualNetwork | Set-AzVirtualNetwork
  3. Create a peer relationship between the virtual network that hosts Azure AD DS (HDIFabrikam-AADDSVNET) and the virtual network that will host the ESP-enabled HDInsight cluster (HDIFabrikam-HDIVNet). Use the following PowerShell code to peer the two virtual networks.

    Add-AzVirtualNetworkPeering -Name 'HDIVNet-AADDSVNet' -RemoteVirtualNetworkId (Get-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-CentralUS').Id -VirtualNetwork (Get-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-WestUS')
    Add-AzVirtualNetworkPeering -Name 'AADDSVNet-HDIVNet' -RemoteVirtualNetworkId (Get-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-WestUS').Id -VirtualNetwork (Get-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-CentralUS')
  4. Create a new Azure Data Lake Storage Gen2 account called Hdigen2store. Configure the account with the user-managed identity HDIFabrikamManagedIdentity. For more information, see Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters.

  5. Set up custom DNS on the HDIFabrikam-AADDSVNET virtual network.

    1. Go to the Azure portal > Resource groups > OnPremADVRG > HDIFabrikam-AADDSVNET > DNS servers.

    2. Select Custom and enter and

    3. Select Save.

      Save custom DNS settings for a virtual network

  6. Create a new ESP-enabled HDInsight Spark cluster.

    1. Select Custom (size, settings, apps).

    2. Enter details for Basics (section 1). Ensure that the Cluster type is Spark 2.3 (HDI 3.6). Ensure that the Resource group is HDIFabrikam-CentralUS.

    3. For Security + networking (section 2), fill in the following details:

      • Under Enterprise Security Package, select Enabled.

      • Select Cluster admin user and select the HDIAdmin account that you created as the on-premises admin user. Click Select.

      • Select Cluster access group > HDIUserGroup. Any user that you add to this group in the future will be able to access HDInsight clusters.

        Select the cluster access group HDIUserGroup

    4. Complete the other steps of the cluster configuration and verify the details on the Cluster summary. Select Create.

  7. Sign in to the Ambari UI for the newly created cluster at Use your admin username and its password.

    The Apache Ambari UI sign-in window

  8. From the cluster dashboard, select Roles.

  9. On the Roles page, under Assign roles to these, next to the Cluster Administrator role, enter the group hdiusergroup.

    Assign the cluster admin role to hdiusergroup

  10. Open your Secure Shell (SSH) client and sign in to the cluster. Use the hdiuser that you created in the on-premises Active Directory instance.

    Sign in to the cluster by using the SSH client

If you can sign in with this account, you've configured your ESP cluster correctly to sync with your on-premises Active Directory instance.

Next steps

Read An introduction to Apache Hadoop security with ESP.