Deploy and manage Azure Arc-enabled Kubernetes cluster extensions

Article
03/14/2023
7 minutes to read

You can create extension instances in an Arc-enabled Kubernetes cluster, setting required and optional parameters including options related to updates and configurations. You can also view, list, update, and delete extension instances.

Before you begin, read the conceptual overview of Arc-enabled Kubernetes cluster extensions and review the list of currently available extensions.

Prerequisites

The latest version of Azure CLI.
The latest versions of the connectedk8s and k8s-extension Azure CLI extensions. Install these extensions by running the following commands:
```
az extension add --name connectedk8s
az extension add --name k8s-extension
```
If the connectedk8s and k8s-extension extensions are already installed, make sure they're updated to the latest version using the following commands:
```
az extension update --name connectedk8s
az extension update --name k8s-extension
```
An existing Azure Arc-enabled Kubernetes connected cluster, with at least one node of operating system and architecture type linux/amd64.
- If you haven't connected a cluster yet, use our quickstart.
- Upgrade your agents to the latest version.

Note

Installing Azure Arc extensions on AKS hybrid clusters provisioned from Azure is currently in preview, with support for the Azure Arc-enabled Open Service Mesh, Azure Key Vault Secrets Provider, Flux (GitOps) and Microsoft Defender for Cloud extensions.

Create extension instance

To create a new extension instance, use k8s-extension create, passing in values for the required parameters.

This example creates an Azure Monitor Container Insights extension instance on an Azure Arc-enabled Kubernetes cluster:

az k8s-extension create --name azuremonitor-containers  --extension-type Microsoft.AzureMonitor.Containers --scope cluster --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

{
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": null,
  "configurationSettings": {
    "logAnalyticsWorkspaceResourceID": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/defaultresourcegroup-eus/providers/microsoft.operationalinsights/workspaces/defaultworkspace-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-eus"
  },
  "creationTime": "2021-04-02T12:13:06.7534628+00:00",
  "errorInfo": {
    "code": null,
    "message": null
  },
  "extensionType": "microsoft.azuremonitor.containers",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/demo/providers/Microsoft.Kubernetes/connectedClusters/demo/providers/Microsoft.KubernetesConfiguration/extensions/azuremonitor-containers",
  "identity": null,
  "installState": "Pending",
  "lastModifiedTime": "2021-04-02T12:13:06.753463+00:00",
  "lastStatusTime": null,
  "name": "azuremonitor-containers",
  "releaseTrain": "Stable",
  "resourceGroup": "demo",
  "scope": {
    "cluster": {
      "releaseNamespace": "azuremonitor-containers"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": null,
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "2.8.2"
}

Note

The service is unable to retain sensitive information for more than 48 hours. If Azure Arc-enabled Kubernetes agents don't have network connectivity for more than 48 hours and can't determine whether to create an extension on the cluster, the extension transitions to Failed state. Once that happens, you'll need to run k8s-extension create again to create a fresh extension Azure resource.

Azure Monitor Container Insights is a singleton extension (only one required per cluster). You'll need to clean up any previous Helm chart installations of Azure Monitor Container Insights (without extensions) before installing the same via extensions. Follow the instructions for deleting the Helm chart before running az k8s-extension create.

Required parameters

The following parameters are required when using az k8s-extension create to create an extension instance.

Parameter name	Description
`--name`	Name of the extension instance
`--extension-type`	The type of extension you want to install on the cluster. For example: Microsoft.AzureMonitor.Containers, microsoft.azuredefender.kubernetes
`--scope`	Scope of installation for the extension: `cluster` or `namespace`
`--cluster-name`	Name of the Azure Arc-enabled Kubernetes resource on which the extension instance has to be created
`--resource-group`	The resource group containing the Azure Arc-enabled Kubernetes resource
`--cluster-type`	The cluster type on which the extension instance has to be created. For most scenarios, use `connectedClusters`, which corresponds to Azure Arc-enabled Kubernetes clusters.

Note

When working with [AKS hybrid clusters provisioned from Azure](#aks-hybrid-clusters-provisioned-from-azure-preview, you must set --cluster-type to use provisionedClusters and also add --cluster-resource-provider microsoft.hybridcontainerservice to the command. Installing Azure Arc extensions on AKS hybrid clusters provisioned from Azure is currently in preview.

Optional parameters

Use one or more of these optional parameters as needed for your scenarios, along with the required parameters.

Parameter name	Description
`--auto-upgrade-minor-version`	Boolean property that determines whether the extension minor version is automatically upgraded. The default setting is `true`. If this parameter is set to `true`, you can't set the `version` parameter, as the version will be dynamically updated. If set to `false`, the extension won't be automatically upgraded, even for patch versions.
`--version`	Version of the extension to be installed (specific version to pin the extension instance to). Must not be supplied if `auto-upgrade-minor-version` is set to `true`.
`--configuration-settings`	Settings that can be passed into the extension to control its functionality. These are passed in as space-separated `key=value` pairs after the parameter name. If this parameter is used in the command, then `--configuration-settings-file` can't be used in the same command.
`--configuration-settings-file`	Path to a JSON file with `key=value` pairs to be used for passing configuration settings into the extension. If this parameter is used in the command, then `--configuration-settings` can't be used in the same command.
`--configuration-protected-settings`	Settings that aren't retrievable using `GET` API calls or `az k8s-extension show` commands. Typically used to pass in sensitive settings. These are passed in as space-separated `key=value` pairs after the parameter name. If this parameter is used in the command, then `--configuration-protected-settings-file` can't be used in the same command.
`--configuration-protected-settings-file`	Path to a JSON file with `key=value` pairs to be used for passing sensitive settings into the extension. If this parameter is used in the command, then `--configuration-protected-settings` can't be used in the same command.
`--release-namespace`	This parameter indicates the namespace within which the release will be created. Only relevant if `scope` is set to `cluster`.
`--release-train`	Extension authors can publish versions in different release trains such as `Stable`, `Preview`, etc. If this parameter isn't set explicitly, `Stable` is used as default. This parameter can't be used when `--auto-upgrade-minor-version` is set to `false`.
`--target-namespace`	Indicates the namespace within which the release will be created. Permission of the system account created for this extension instance will be restricted to this namespace. Only relevant if `scope` is set to `namespace`.

Show extension details

To view details of a currently installed extension instance, use k8s-extension show, passing in values for the mandatory parameters.

az k8s-extension show --name azuremonitor-containers --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

{
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": null,
  "configurationSettings": {
    "logAnalyticsWorkspaceResourceID": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/defaultresourcegroup-eus/providers/microsoft.operationalinsights/workspaces/defaultworkspace-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-eus"
  },
  "creationTime": "2021-04-02T12:13:06.7534628+00:00",
  "errorInfo": {
    "code": null,
    "message": null
  },
  "extensionType": "microsoft.azuremonitor.containers",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/demo/providers/Microsoft.Kubernetes/connectedClusters/demo/providers/Microsoft.KubernetesConfiguration/extensions/azuremonitor-containers",
  "identity": null,
  "installState": "Installed",
  "lastModifiedTime": "2021-04-02T12:13:06.753463+00:00",
  "lastStatusTime": "2021-04-02T12:13:49.636+00:00",
  "name": "azuremonitor-containers",
  "releaseTrain": "Stable",
  "resourceGroup": "demo",
  "scope": {
    "cluster": {
      "releaseNamespace": "azuremonitor-containers"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": null,
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "2.8.2"
}

List all extensions installed on the cluster

To view a list of all extensions installed on a cluster, use k8s-extension list, passing in values for the mandatory parameters.

az k8s-extension list --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

[
  {
    "autoUpgradeMinorVersion": true,
    "creationTime": "2020-09-15T02:26:03.5519523+00:00",
    "errorInfo": {
      "code": null,
      "message": null
    },
    "extensionType": "Microsoft.AzureMonitor.Containers",
    "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myRg/providers/Microsoft.Kubernetes/connectedClusters/myCluster/providers/Microsoft.KubernetesConfiguration/extensions/myExtInstanceName",
    "identity": null,
    "installState": "Pending",
    "lastModifiedTime": "2020-09-15T02:48:45.6469664+00:00",
    "lastStatusTime": null,
    "name": "myExtInstanceName",
    "releaseTrain": "Stable",
    "resourceGroup": "myRG",
    "scope": {
      "cluster": {
        "releaseNamespace": "myExtInstanceName1"
      }
    },
    "statuses": [],
    "type": "Microsoft.KubernetesConfiguration/extensions",
    "version": "0.1.0"
  },
  {
    "autoUpgradeMinorVersion": true,
    "creationTime": "2020-09-02T00:41:16.8005159+00:00",
    "errorInfo": {
      "code": null,
      "message": null
    },
    "extensionType": "microsoft.azuredefender.kubernetes",
    "id": "/subscriptions/0e849346-4343-582b-95a3-e40e6a648ae1/resourceGroups/myRg/providers/Microsoft.Kubernetes/connectedClusters/myCluster/providers/Microsoft.KubernetesConfiguration/extensions/defender",
    "identity": null,
    "installState": "Pending",
    "lastModifiedTime": "2020-09-02T00:41:16.8005162+00:00",
    "lastStatusTime": null,
    "name": "microsoft.azuredefender.kubernetes",
    "releaseTrain": "Stable",
    "resourceGroup": "myRg",
    "scope": {
      "cluster": {
        "releaseNamespace": "myExtInstanceName2"
      }
    },
    "type": "Microsoft.KubernetesConfiguration/extensions",
    "version": "0.1.0"
  }
]

Delete extension instance

To delete an extension instance on a cluster, use k8s-extension delete, passing in values for the mandatory parameters.

az k8s-extension delete --name azuremonitor-containers --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Note

The Azure resource representing this extension gets deleted immediately. The Helm release on the cluster associated with this extension is only deleted when the agents running on the Kubernetes cluster have network connectivity and can reach out to Azure services again to fetch the desired state.

Important

When working with AKS hybrid clusters provisioned from Azure, you must add --yes to the delete command. Installing Azure Arc extensions on AKS hybrid clusters provisioned from Azure is currently in preview.

AKS hybrid clusters provisioned from Azure (preview)

You can deploy extensions to AKS hybrid clusters provisioned from Azure. However, there are a few key differences to keep in mind in order to deploy successfully:

The value for the --cluster-type parameter must be provisionedClusters.
You must add --cluster-resource-provider microsoft.hybridcontainerservice to your commands.

When deleting an extension instance, you must add --yes to the command:

az k8s-extension delete --name azuremonitor-containers --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type provisionedClusters --cluster-resource-provider microsoft.hybridcontainerservice --yes

In addition, you must be using the latest version of the Azure CLI k8s-extension module (version >= 1.3.3). Use the following commands to add or update to the latest version:

# add if you do not have this installed
az extension add --name k8s-extension

# update if you do have the module installed
az extension update --name k8s-extension

Important

Installing Azure Arc extensions on AKS hybrid clusters provisioned from Azure is currently in preview.

Next steps

Review the az k8s-extension CLI reference for a comprehensive list of commands and parameters.
Learn more about how extensions work with Arc-enabled Kubernetes clusters.
Review the cluster extensions currently available for Azure Arc-enabled Kubernetes.

Deploy and manage Azure Arc-enabled Kubernetes cluster extensions

Prerequisites

Create extension instance

Required parameters

Optional parameters

Show extension details

List all extensions installed on the cluster

Delete extension instance

AKS hybrid clusters provisioned from Azure (preview)

Next steps

Feedback

Additional resources

Additional resources