Create an Azure Red Hat OpenShift cluster with managed identities (preview)

This article shows you how to deploy an Azure Red Hat OpenShift cluster utilizing managed identities.

Important

Currently, this Azure Red Hat OpenShift feature is being offered in preview only. Preview features are available on a self-service, opt-in basis. Previews are provided as is and as available, and they're excluded from the service-level agreements and limited warranty. Azure Red Hat OpenShift previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use.

Azure Red Hat OpenShift is a managed OpenShift service that supports managed identities and workload identities. Managed identities and workload identities help minimize risk when securing workloads and applications by providing short-lived tokens rather than long-lived credentials such as a service principal with client secret credentials.

For more information, see:

• Understanding managed identities in Azure Red Hat OpenShift
• What are workload identities?
• What are managed identities for Azure resources?

Prerequisites

Ensure you're using Azure CLI version 2.67.0 or higher. Use az--version to find the version of Azure CLI you installed. If you need to install or upgrade, see Install Azure CLI.

Azure Red Hat OpenShift requires a minimum of 44 cores to create an OpenShift cluster. The default Azure resource quota for a new Azure subscription doesn't meet this requirement. To request an increase in your resource limit, see Increase VM-family vCPU quotas.

The 44 cores are used as follows:

• Bootstrap machine: 8 cores
• Control plane (master machines): 24 cores
• Compute (worker machines): 12 cores

When the installation is complete, the bootstrap machine is removed and your cluster uses a total of 36 cores. For more information, see Installing on Azure.

For example, to check the current subscription quota of the smallest supported virtual machine family SKU "Standard DSv5":

LOCATION=eastus
az vm list-usage -l $LOCATION \
--query "[?contains(name.value, 'standardDSv5Family')]" -o table

Download Azure Red Hat OpenShift extension wheel file (Preview only)

To run the commands in this article, first download the Azure Red Hat OpenShift extension wheel file from https://aka.ms/az-aroext-latest. To install the extension, run the following command:

az extension add -s <path to downloaded whl file>

Verify your permissions

In this article, you create a resource group which contains the virtual network and managed identities for the cluster. To create a resource group, you need Contributor and User Access Administrator permissions or Owner permissions on the resource group or subscription containing it.

You also need sufficient Microsoft Entra permissions (either a member user of the tenant, or a guest assigned with the role Application administrator) to create a set of managed identities and assign roles for the cluster to use. For more information, see Member and guests and Assign administrator and nonadministrator roles to users with Microsoft Entra ID.

Register the resource providers

Some Azure resource providers, including the Azure Red Hat OpenShift resource provider, require registration in order to function. Registering a resource provider creates a service principal inside of your subscription that authorizes the resource provider to perform certain actions, like resource creation. For more information around resource provider registration, see Register resource provider.

1. If you have multiple Azure subscriptions, specify the relevant subscription ID:

   az account set --subscription <SUBSCRIPTION ID>
   

2. Register the Microsoft.RedHatOpenShift resource provider:

   az provider register -n Microsoft.RedHatOpenShift --wait
   

3. Register the Microsoft.Compute resource provider:

   az provider register -n Microsoft.Compute --wait
   

4. Register the Microsoft.Storage resource provider:

   az provider register -n Microsoft.Storage --wait
   

5. Register the Microsoft.Authorization resource provider:

   az provider register -n Microsoft.Authorization --wait
   

Get a Red Hat pull secret (optional)

Note

Azure Red Hat OpenShift pull secret doesn't change the cost of the Red Hat OpenShift license for it.

A Red Hat pull secret enables your cluster to access Red Hat container registries, along with other content such as operators from OperatorHub. This step is optional but recommended. If you decide to add the pull secret later, follow this guidance. The field cloud.openshift.com is removed from your secret even if your pull-secret contains that field. This field enables an extra monitoring feature, which sends data to RedHat and is thus disabled by default. To enable this feature, see Enabling remote health reporting.

1. Navigate to your Red Hat OpenShift cluster manager portal and sign-in.

   You need to sign in to your Red Hat account or create a new Red Hat account with your business email and accept the terms and conditions.

2. Select Download pull secret, then download a pull secret to be used with your Azure Red Hat OpenShift cluster.

   Keep the saved pull-secret.txt file somewhere safe. The file is used in each cluster creation if you need to create a cluster that includes samples or operators for Red Hat or certified partners.

   When running the az aro create command, you can reference your pull secret using the --pull-secret @pull-secret.txt parameter. Execute az aro create from the directory where you stored your pull-secret.txt file. Otherwise, replace @pull-secret.txt with @/path/to/my/pull-secret.txt.

   If you're copying your pull secret or referencing it in other scripts, your pull secret should be formatted as a valid JSON string.

Prepare a custom domain for your cluster (optional)

When running the az aro create command, you can specify a custom domain for your cluster by using the --domain foo.example.com parameter.

Note

Adding a domain name is optional when creating a cluster through Azure CLI. A domain name (or a prefix used as part of the autogenerated DNS name for OpenShift console and API servers) is needed when adding a cluster through the portal. For more information, see Quickstart: Deploy an Azure Red Hat OpenShift cluster using the Azure portal.

If you provide a custom domain for your cluster, note the following points:

• After creating your cluster, you must create two DNS A records in your DNS server for the --domain specified:
  • api - pointing to the API server IP address
  • *.apps - pointing to the ingress IP address
  • Retrieve these values by executing the following command after cluster creation: az aro show -n -g --query '{api:apiserverProfile.ip, ingress:ingressProfiles[0].ip}'.
• The OpenShift console is available at a URL such as https://console-openshift-console.apps.example.com, instead of the built-in domain https://console-openshift-console.apps.<random>.<location>.aroapp.io.
• By default, OpenShift uses self-signed certificates for all of the routes created on custom domains *.apps.example.com. If you choose to use custom DNS after connecting to the cluster, you need to follow the OpenShift documentation to configure a custom CA for your ingress controller and a custom CA for your API server.

Installation

You can use Azure CLI or Bicep to deploy an Azure Red Hat OpenShift cluster utilizing managed identities.

Install using Azure CLI

This section describes how to use Azure CLI to create an Azure Red Hat OpenShift cluster utilizing managed identities.

Create a virtual network containing two empty subnets

Create a virtual network containing two empty subnets. If you have an existing virtual network that meets your needs, skip this step.

For information about networking and requirements, see Networking for Azure Red Hat OpenShift.

1. Set the following variables in the shell environment in which you execute the az commands.

   LOCATION=eastus                 # the location of your cluster
   RESOURCEGROUP=aro-rg            # the name of the resource group where you want to create your cluster
   CLUSTER=cluster                 # the name of your cluster
   

2. Create a resource group.

   An Azure resource group is a logical group in which Azure resources are deployed and managed. When you create a resource group, you're asked to specify a location. This location is where resource group metadata is stored and where your resources run in Azure if you don't specify another region during resource creation. Create a resource group using the az group create command.

   Note

   Azure Red Hat OpenShift isn't available in all regions where an Azure resource group can be created. See Available regions for information on where Azure Red Hat OpenShift is supported.

   az group create \
     --location $LOCATION \
     --name $RESOURCEGROUP
   

3. Create a virtual network, master, and worker subnets in the same resource group previously created.

   Azure Red Hat OpenShift clusters require a virtual network with two empty subnets for the master and worker nodes. You can either create a new virtual network or use an existing virtual network.

   az network vnet create \
      --resource-group $RESOURCEGROUP \
      --name aro-vnet \
      --address-prefixes 10.0.0.0/22
   

   az network vnet subnet create \
      --resource-group $RESOURCEGROUP \
      --vnet-name aro-vnet \
      --name master \
      --address-prefixes 10.0.0.0/23
   

   az network vnet subnet create \
      --resource-group $RESOURCEGROUP \
      --vnet-name aro-vnet \
      --name worker \
      --address-prefixes 10.0.2.0/23
   

Create the required user assigned managed identities

1. Create the following required identities. Azure Red Hat OpenShift requires nine managed identities, each needs to have an assigned, built-in role:

   • Seven managed identities related to core OpenShift operators.
   • One managed identity for the Azure Red Hat OpenShift service operator.
   • One other identity for the cluster to enable the use of these identities.

   The managed identity components are:

   • OpenShift Image Registry Operator (image-registry)
   • OpenShift Network Operator (cloud-network-config)
   • OpenShift Disk Storage Operator (disk-csi-driver)
   • OpenShift File Storage Operator (file-csi-driver)
   • OpenShift Cluster Ingress Operator (ingress)
   • OpenShift Cloud Controller Manager (cloud-controller-manager)
   • OpenShift Machine API Operator (machine-api)
   • Azure Red Hat OpenShift Service Operator (aro-operator)

   There are eight different managed identities and corresponding built-in roles that represent the permissions needed for each component of Azure Red Hat OpenShift to perform its duties. In addition, the platform requires one other identity, the cluster identity, to perform federated credential creation for the previously listed managed identity components (aro-cluster).

   For more information about Red Hat OpenShift cluster operators, see Cluster Operators reference.

   For more information about managed identities in Azure Red Hat OpenShift, see Understanding managed identities in Azure Red Hat OpenShift.

   Create the required identities:

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name aro-cluster
   

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name cloud-controller-manager
   

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name ingress
   

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name machine-api
   

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name disk-csi-driver
   

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name cloud-network-config
   

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name image-registry
   

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name file-csi-driver
   

   az identity create \
   --resource-group $RESOURCEGROUP \
   --name aro-operator
   

2. Create the required role assignments for each operator identity, cluster identity, and the first party service principal.

   Note

   This article assumes that only master and worker subnets are present. If you configured more cluster subnets at install time, you need to grant role assignment scope to those subnets, for operators that require it.

   The following role assignments for master and worker subnets assume there's no network security group (NSG), route table, or network address translation (NAT) gateway attached. If you bring any of those network resources to the install, you need to create more role assignments that grant operator identities permissions for those extra network resources. For each operator that requires a role assignment for the following subnets or the virtual network, it also requires a role assignment for the extra network resource.

   SUBSCRIPTION_ID=$(az account show --query 'id' -o tsv)
   
   # assign cluster identity permissions over identities previously created
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-cluster --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/ef318e2a-8334-4a05-9e4a-295a196c6a6e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aro-operator"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-cluster --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/ef318e2a-8334-4a05-9e4a-295a196c6a6e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-cluster --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/ef318e2a-8334-4a05-9e4a-295a196c6a6e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-cluster --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/ef318e2a-8334-4a05-9e4a-295a196c6a6e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/machine-api"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-cluster --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/ef318e2a-8334-4a05-9e4a-295a196c6a6e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-cluster --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/ef318e2a-8334-4a05-9e4a-295a196c6a6e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-cluster --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/ef318e2a-8334-4a05-9e4a-295a196c6a6e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-cluster --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/ef318e2a-8334-4a05-9e4a-295a196c6a6e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver"
   
   # assign vnet-level permissions for operators that require it, and subnets-level permission for operators that require it
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name cloud-controller-manager --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/a1f96423-95ce-4224-ab27-4e3dc72facd4" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/master"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name cloud-controller-manager --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/a1f96423-95ce-4224-ab27-4e3dc72facd4" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/worker"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name ingress --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/0336e1d3-7a87-462b-b6db-342b63f7802c" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/master"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name ingress --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/0336e1d3-7a87-462b-b6db-342b63f7802c" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/worker"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name machine-api --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/0358943c-7e01-48ba-8889-02cc51d78637" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/master"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name machine-api --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/0358943c-7e01-48ba-8889-02cc51d78637" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/worker"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name cloud-network-config --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/be7a6435-15ae-4171-8f30-4a343eff9e8f" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name file-csi-driver --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/0d7aedc0-15fd-4a67-a412-efad370c947e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/master"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name file-csi-driver --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/0d7aedc0-15fd-4a67-a412-efad370c947e" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/worker"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-operator --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/4436bae4-7702-4c84-919b-c4069ff25ee2" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/master"
   
   az role assignment create \
       --assignee-object-id "$(az identity show --resource-group $RESOURCEGROUP --name aro-operator --query principalId -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/4436bae4-7702-4c84-919b-c4069ff25ee2" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet/subnets/worker"
   
   az role assignment create \
       --assignee-object-id "$(az ad sp list --display-name "Azure Red Hat OpenShift RP" --query '[0].id' -o tsv)" \
       --assignee-principal-type ServicePrincipal \
       --role "/subscriptions/$SUBSCRIPTION_ID/providers/Microsoft.Authorization/roleDefinitions/4d97b98b-1d4f-4787-a291-c67834d212e7" \
       --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aro-vnet"
   

Create the cluster

To create a cluster, run the following command shown under the options. If you choose to use either of the following options, modify the command accordingly:

• Option 1: You can pass your Red Hat pull secret, which enables your cluster to access Red Hat container registries along with other content. Add the --pull-secret @pull-secret.txt argument to your command.
• Option 2: You can use a custom domain. Add the --domain foo.example.com argument to your command, replacing foo.example.com with your own custom domain.

Create the cluster with the required environment variables. For each --assign-platform-workload-identity flag, the first argument represents the key, which tells the Azure Red Hat OpenShift resource provider which OpenShift operator to use for a given identity. The second argument represents the reference to the identity itself.

az aro create \
    --resource-group $RESOURCEGROUP \
    --name $CLUSTER \
    --vnet aro-vnet \
    --master-subnet master \
    --worker-subnet worker \
    --version <VERSION> \
    --enable-managed-identity \
    --assign-cluster-identity aro-cluster \
    --assign-platform-workload-identity file-csi-driver file-csi-driver \
    --assign-platform-workload-identity cloud-controller-manager cloud-controller-manager \
    --assign-platform-workload-identity ingress ingress \
    --assign-platform-workload-identity image-registry image-registry \
    --assign-platform-workload-identity machine-api machine-api \
    --assign-platform-workload-identity cloud-network-config cloud-network-config \
    --assign-platform-workload-identity aro-operator aro-operator \
    --assign-platform-workload-identity disk-csi-driver disk-csi-driver

As an option, if identity resources exist in a different region or resource group, you can pass full resource IDs to create. See the following example:

az aro create \
    --resource-group $RESOURCEGROUP \
    --name $CLUSTER \
    --vnet aro-vnet \
    --master-subnet master \
    --worker-subnet worker \
    --version <VERSION> \
    --enable-managed-identity \
    --assign-cluster-identity /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aro-cluster \
    --assign-platform-workload-identity file-csi-driver /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver \
    --assign-platform-workload-identity cloud-controller-manager /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager \
    --assign-platform-workload-identity ingress /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress \
    --assign-platform-workload-identity image-registry /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry \
    --assign-platform-workload-identity machine-api /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/machine-api \
    --assign-platform-workload-identity cloud-network-config /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config \
    --assign-platform-workload-identity aro-operator /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aro-operator \
    --assign-platform-workload-identity disk-csi-driver /subscriptions/$SUBSCRIPTION_ID/resourcegroups/$RESOURCEGROUP/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver

Select a different Azure Red Hat OpenShift version

You can choose to use a specific version of Azure Red Hat OpenShift when creating your cluster. First, use the CLI to query for available Azure Red Hat OpenShift versions:

az aro get-versions --location <REGION>

Once the version is chosen, specify it using the --version parameter in the az aro create command.

Install using Bicep

This section describes how to use Bicep to create an Azure Red Hat OpenShift cluster utilizing managed identities.

1. Save the Example Bicep template (located after step 5) to a file.

2. Set the following variables in the shell environment that you plan to execute the az commands.

   LOCATION=eastus                     # the location of your cluster
   RESOURCEGROUP=aro-rg                # the name of the resource group where you want to create your cluster
   CLUSTER=cluster                     # the name of your cluster
   VERSION=4.15.35                     # the version of the cluster
   PULL_SECRET=$(cat pull-secret.txt)  # the Red Hat pull secret JSON, provided as file or string
   

3. Create a resource group to hold the cluster resource and the cluster virtual network and identities.

   az group create --name $RESOURCEGROUP --location $LOCATION
   

4. For the resource provider, check the first-party service principal object ID for your subscription.

   ARO_RP_SP_OBJECT_ID=$(az ad sp list --display-name "Azure Red Hat OpenShift RP" --query '[0].id' -o tsv)
   

5. Apply the Bicep template:

   az deployment group create \
       --name aroDeployment \
       --resource-group $RESOURCEGROUP \
       --template-file azuredeploy.bicep \
       --parameters location=$LOCATION \
       --parameters version=$VERSION \
       --parameters clusterName=$CLUSTER \
       --parameters rpObjectId=$ARO_RP_SP_OBJECT_ID
       (--parameters domain=$DOMAIN) \ #optional
       (--parameters pullSecret=$PULL_SECRET) # optional
   

Example Bicep Template

Note

This article assumes that only master and worker subnets are present. If you configured more cluster subnets at install time, you need to grant role assignment scope to those subnets, for operators that require it.

The following role assignments for master and worker subnets assume there's no network security group (NSG), route table, or network address translation (NAT) gateway attached. If you bring any of those network resources to the install, you need to create more role assignments that grant operator identities permissions for those extra network resources. For each operator that requires a role assignment for the following subnets or the virtual network, it also requires a role assignment for the extra network resource.

@description('Location')
param location string = resourceGroup().location

@description('Domain Prefix')
param domain string

@description('Version of the OpenShift cluster')
param version string

@description('Pull secret from cloud.redhat.com. The json should be input as a string')
@secure()
param pullSecret string = ''

@description('Name of ARO vNet')
param clusterVnetName string = 'aro-vnet'

@description('ARO vNet Address Space')
param clusterVnetCidr string = '10.100.0.0/15'

@description('Worker node subnet address space')
param workerSubnetCidr string = '10.100.70.0/23'

@description('Master node subnet address space')
param masterSubnetCidr string = '10.100.76.0/24'

@description('Master Node VM Type')
param masterVmSize string = 'Standard_D8s_v3'

@description('Worker Node VM Type')
param workerVmSize string = 'Standard_D4s_v3'

@description('Worker Node Disk Size in GB')
@minValue(128)
param workerVmDiskSize int = 128

@description('Cidr for Pods')
param podCidr string = '10.128.0.0/14'

@metadata({
 description: 'Cidr of service'
})
param serviceCidr string = '172.30.0.0/16'

@description('Unique name for the cluster')
param clusterName string

@description('Api Server Visibility')
@allowed([
 'Private'
 'Public'
])
param apiServerVisibility string = 'Public'

@description('Ingress Visibility')
@allowed([
 'Private'
 'Public'
])
param ingressVisibility string = 'Public'

@description('The ObjectID of the Resource Provider Service Principal')
param rpObjectId string

@description('Specify if FIPS validated crypto modules are used')
@allowed([
 'Enabled'
 'Disabled'
])
param fips string = 'Disabled'

@description('Specify if master VMs are encrypted at host')
@allowed([
 'Enabled'
 'Disabled'
])
param masterEncryptionAtHost string = 'Disabled'

@description('Specify if worker VMs are encrypted at host')
@allowed([
 'Enabled'
 'Disabled'
])
param workerEncryptionAtHost string = 'Disabled'

var resourceGroupId = '/subscriptions/${subscription().subscriptionId}/resourceGroups/aro-${domain}-${location}'
var masterSubnetId=resourceId('Microsoft.Network/virtualNetworks/subnets', clusterVnetName, 'master')
var workerSubnetId=resourceId('Microsoft.Network/virtualNetworks/subnets', clusterVnetName, 'worker')

resource vnet 'Microsoft.Network/virtualNetworks@2023-06-01' = {
 name: clusterVnetName
 location: location
 properties: {
   addressSpace: { addressPrefixes: [ clusterVnetCidr ] }
   subnets: [
     {
       name: 'master'
       properties: {
         addressPrefixes: [ masterSubnetCidr ]
         serviceEndpoints: [ { service: 'Microsoft.ContainerRegistry' } ]
       }
     }
     {
       name: 'worker'
       properties: {
         addressPrefixes: [ workerSubnetCidr ]
         serviceEndpoints: [ { service: 'Microsoft.ContainerRegistry' } ]
       }
     }
   ]
 }
}

resource workerSubnet 'Microsoft.Network/virtualNetworks/subnets@2020-08-01' existing = {
 parent: vnet
 name: 'worker'
}

resource masterSubnet 'Microsoft.Network/virtualNetworks/subnets@2020-08-01' existing = {
 parent: vnet
 name: 'master'
}

// create required identities

resource cloudControllerManager 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'cloud-controller-manager'
   location: location
}

resource ingress 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'ingress'
   location: location
}

resource machineApi 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'machine-api'
   location: location
}

resource diskCsiDriver 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'disk-csi-driver'
   location: location
}

resource cloudNetworkConfig 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'cloud-network-config'
   location: location
}

resource imageRegistry 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'image-registry'
   location: location
}

resource fileCsiDriver 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'file-csi-driver'
   location: location
}

resource aroOperator 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'aro-operator'
   location: location
}

resource clusterMsi 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
   name: 'cluster'
   location: location
}

// create required role assignments on vnet / subnets

resource cloudControllerManagerMasterSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(masterSubnet.id, 'cloud-controller-manager')
   scope: masterSubnet
   properties: {
       principalId: cloudControllerManager.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'a1f96423-95ce-4224-ab27-4e3dc72facd4')
       principalType: 'ServicePrincipal'
   }
}

resource cloudControllerManagerWorkerSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
 name: guid(workerSubnet.id, 'cloud-controller-manager')
 scope: workerSubnet
 properties: {
     principalId: cloudControllerManager.properties.principalId
     roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'a1f96423-95ce-4224-ab27-4e3dc72facd4')
     principalType: 'ServicePrincipal'
 }
}

resource ingressMasterSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(masterSubnet.id, 'ingress')
   scope: masterSubnet
   properties: {
       principalId: ingress.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '0336e1d3-7a87-462b-b6db-342b63f7802c')
       principalType: 'ServicePrincipal'
   }
}

resource ingressWorkerSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
 name: guid(workerSubnet.id, 'ingress')
 scope: workerSubnet
 properties: {
     principalId: ingress.properties.principalId
     roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '0336e1d3-7a87-462b-b6db-342b63f7802c')
     principalType: 'ServicePrincipal'
 }
}

resource machineApiMasterSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(masterSubnet.id, 'machine-api')
   scope: masterSubnet
   properties: {
       principalId: machineApi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '0358943c-7e01-48ba-8889-02cc51d78637')
       principalType: 'ServicePrincipal'
   }
}

resource machineApiWorkerSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(workerSubnet.id, 'machine-api')
   scope: workerSubnet
   properties: {
       principalId: machineApi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '0358943c-7e01-48ba-8889-02cc51d78637')
       principalType: 'ServicePrincipal'
   }
}

resource cloudNetworkConfigVnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(vnet.id, 'cloud-network-config')
   scope: vnet
   properties: {
       principalId: cloudNetworkConfig.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'be7a6435-15ae-4171-8f30-4a343eff9e8f')
       principalType: 'ServicePrincipal'
   }
}

resource fileCsiDriverMasterSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(masterSubnet.id, 'file-csi-driver')
   scope: masterSubnet
   properties: {
       principalId: fileCsiDriver.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '0d7aedc0-15fd-4a67-a412-efad370c947e')
       principalType: 'ServicePrincipal'
   }
}

resource fileCsiDriverWorkerSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
 name: guid(workerSubnet.id, 'file-csi-driver')
 scope: workerSubnet
 properties: {
     principalId: fileCsiDriver.properties.principalId
     roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '0d7aedc0-15fd-4a67-a412-efad370c947e')
     principalType: 'ServicePrincipal'
 }
}

resource aroOperatorMasterSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(masterSubnet.id, 'aro-operator')
   scope: masterSubnet
   properties: {
       principalId: aroOperator.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '4436bae4-7702-4c84-919b-c4069ff25ee2')
       principalType: 'ServicePrincipal'
   }
}

resource aroOperatorWorkerSubnetRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
 name: guid(workerSubnet.id, 'aro-operator')
 scope: workerSubnet
 properties: {
     principalId: aroOperator.properties.principalId
     roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '4436bae4-7702-4c84-919b-c4069ff25ee2')
     principalType: 'ServicePrincipal'
 }
}

// create required role assignments on cluster MSI

resource clusterMsiRoleAssignmentCloudControllerManager 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(cloudControllerManager.id, 'cluster')
   scope: cloudControllerManager
   properties: {
       principalId: clusterMsi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ef318e2a-8334-4a05-9e4a-295a196c6a6e')
       principalType: 'ServicePrincipal'
   }
}

resource clusterMsiRoleAssignmentIngress 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(ingress.id, 'cluster')
   scope: ingress
   properties: {
       principalId: clusterMsi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ef318e2a-8334-4a05-9e4a-295a196c6a6e')
       principalType: 'ServicePrincipal'
   }
}

resource clusterMsiRoleAssignmentMachineApi 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(machineApi.id, 'cluster')
   scope: machineApi
   properties: {
       principalId: clusterMsi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ef318e2a-8334-4a05-9e4a-295a196c6a6e')
       principalType: 'ServicePrincipal'
   }
}

resource clusterMsiRoleAssignmentDiskCsiDriver 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(diskCsiDriver.id, 'cluster')
   scope: diskCsiDriver
   properties: {
       principalId: clusterMsi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ef318e2a-8334-4a05-9e4a-295a196c6a6e')
       principalType: 'ServicePrincipal'
   }
}

resource clusterMsiRoleAssignmentCloudNetworkConfig 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(cloudNetworkConfig.id, 'cluster')
   scope: cloudNetworkConfig
   properties: {
       principalId: clusterMsi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ef318e2a-8334-4a05-9e4a-295a196c6a6e')
       principalType: 'ServicePrincipal'
   }
}

resource clusterMsiRoleAssignmentCloudImageRegistry 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(imageRegistry.id, 'cluster')
   scope: imageRegistry
   properties: {
       principalId: clusterMsi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ef318e2a-8334-4a05-9e4a-295a196c6a6e')
       principalType: 'ServicePrincipal'
   }
}

resource clusterMsiRoleAssignmentCloudFileCsiDriver 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(fileCsiDriver.id, 'cluster')
   scope: fileCsiDriver
   properties: {
       principalId: clusterMsi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ef318e2a-8334-4a05-9e4a-295a196c6a6e')
       principalType: 'ServicePrincipal'
   }
}

resource clusterMsiRoleAssignmentCloudAroOperator 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(aroOperator.id, 'cluster')
   scope: aroOperator
   properties: {
       principalId: clusterMsi.properties.principalId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ef318e2a-8334-4a05-9e4a-295a196c6a6e')
       principalType: 'ServicePrincipal'
   }
}

// create first party role assignment over the vnet

resource fpspRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
   name: guid(vnet.id, rpObjectId)
   scope: vnet
   properties: {
       principalId: rpObjectId
       roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '4d97b98b-1d4f-4787-a291-c67834d212e7')
       principalType: 'ServicePrincipal'
   }
}

// create cluster

resource cluster 'Microsoft.RedHatOpenShift/openShiftClusters@2024-08-12-preview' = {
   name: clusterName
   location: location
   properties: {
       clusterProfile: {
           domain: domain
           #disable-next-line use-resource-id-functions
           resourceGroupId: resourceGroupId
           version: version
           fipsValidatedModules: fips
           pullSecret: pullSecret
       }
       networkProfile: {podCidr: podCidr, serviceCidr: serviceCidr}
       masterProfile: {
           vmSize: masterVmSize
           subnetId: masterSubnetId
           encryptionAtHost: masterEncryptionAtHost
       }
       workerProfiles: [{
           name: 'worker'
           count: 3
           diskSizeGB: workerVmDiskSize
           vmSize: workerVmSize
           subnetId: workerSubnetId
           encryptionAtHost: workerEncryptionAtHost
       }]
       apiserverProfile: {visibility: apiServerVisibility}
       ingressProfiles: [{name: 'default', visibility: ingressVisibility}]
       platformWorkloadIdentityProfile: {
           platformWorkloadIdentities: {
               'cloud-controller-manager': {resourceId: cloudControllerManager.id}
               ingress: {resourceId: ingress.id}
               'machine-api': {resourceId: machineApi.id}
               'disk-csi-driver': {resourceId: diskCsiDriver.id}
               'cloud-network-config': {resourceId: cloudNetworkConfig.id}
               'image-registry': {resourceId: imageRegistry.id}
               'file-csi-driver': {resourceId: fileCsiDriver.id}
               'aro-operator': {resourceId: aroOperator.id}
           }
       }
   }
   identity: {
       type: 'UserAssigned'
       userAssignedIdentities: {
           '${clusterMsi.id}': {}
       }
   }
}

Clean up

To delete a managed identity cluster, run the following command:

az aro delete -n $CLUSTER -g $RESOURCEGROUP

Note this delete command doesn't clean up the cluster-assigned managed identities that were created as part of the installation. You need to manually delete the identities and role assignments.