Deploy Foundry Local as an Azure Arc extension

  • Applies to: Foundry Local on Azure Local

This article shows you how to set up Foundry Local as an extension on your Azure Kubernetes Service (AKS) cluster enabled by Azure Arc. Use the Azure portal or the Azure CLI to deploy Foundry Local as an extension on your Azure Arc-enabled Kubernetes cluster. Helm is also a supported deployment option, and installation instructions are provided during preview access onboarding.

If you plan to use models with Agentic Retrieval in Foundry Local, Entra ID authentication must remain enabled (the default) during this extension installation.

Important

  • Foundry Local is available in preview. Preview releases provide early access to features that are in active deployment.
  • Features, approaches, and processes can change or have limited capabilities before general availability (GA).

Prerequisites

Before you begin, make sure you have:

  • Access to Foundry Local preview: Foundry Local on Azure Local is available by request during preview. Submit an access request at aka.ms/FoundryLocalAzure_PreviewRequest.
  • A Kubernetes cluster (version 1.29 or later) connected to Azure Arc. For more information, see Azure Arc–enabled Kubernetes.
  • Your Azure Arc-enabled Kubernetes cluster is located in a supported region. For available regions, see Supported regions.
  • An app registration for enablement of authorization and authentication. See Configure authentication for Foundry Local enabled by Azure Arc.
  • kubectl installed and configured for your cluster.
  • Helm installed.
  • A Gateway API provider on the cluster. Foundry Local routes model traffic through the Kubernetes Gateway API instead of an Ingress controller. The supported provider is Istio (istio-base + istiod, version 1.29 or later) running as a Gateway API controller — sidecar injection and ambient mode are optional and not required by the inference operator.
  • Gateway API CRDs (v1.4.0 or later).
  • Gateway API Inference Extension CRDs (v1.5.0 or later)
  • (Optional) A namespace strategy if you plan to deploy models outside the default foundry-local-operator namespace. You must set namespace configuration during installation. For more information, see Namespace configuration for model deployments.

GPU prerequisites

If you plan to run GPU workloads, also make sure:

  • Your cluster has NVIDIA GPU nodes with CUDA drivers installed on the nodes.
  • The Kubernetes device plugin for NVIDIA is configured so the cluster can schedule GPU workloads.

For more information, see NVIDIA GPU Operator.

Step 1: Install Istio and the Gateway API CRDs

Foundry Local on Azure Local uses the Kubernetes Gateway API for in-cluster and external traffic routing. Install the Gateway API CRDs first, then install Istio as the Gateway API provider. Istio's istiod discovers the CRDs at startup and registers the istio GatewayClass once it sees them. Installing the CRDs and Istio in the reverse order forces an istiod restart and is reported as flaky on some clusters.

# 1. Install the Gateway API CRDs (standard channel). 
kubectl apply --server-side -f \ 
    https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml 
# 2. Install the Gateway API Inference Extension CRDs 
#    before istiod so istiod picks up InferencePool support at startup. 
kubectl apply --server-side -f \ 
    https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml 
# 3. Add the Istio Helm repository 
helm repo add istio https://istio-release.storage.googleapis.com/charts 
helm repo update 
# 4. Install Istio base (cluster-scoped CRDs) 
helm install istio-base istio/base \ 
    -n istio-system --create-namespace --wait 
# 5. Install the istiod control plane. 
#    Set ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true so istiod recognizes 
#    the InferencePool CRD. EPP routing is enabled by default for multi-replica 
#    vLLM ModelDeployments (and on demand for any deployment that explicitly 
#    sets spec.vllm.epp.enabled: true), so this flag is the recommended default. 
helm install istiod istio/istiod \ 
    -n istio-system \ 
    --set pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true \ 
    --wait

Step 2: Install cert-manager and trust-manager

Foundry Local on Azure Local requires cert-manager and trust-manager for automated certificate management.

Use the Azure CLI to create the cert-manager extension on your cluster. Choose the appropriate command for your shell environment:

az k8s-extension create \
    --cluster-name <your_arc_cluster_name> \
    --name "azure-cert-manager" \
    --resource-group <resource_group_of_the_arc_cluster> \
    --cluster-type connectedClusters \
    --extension-type Microsoft.CertManagement \
    --scope cluster \
    --release-train stable \
    --config config.enableGatewayAPI=true \
    --config cert-manager.crds.keep=true \
    --config trust-manager.defaultPackage.enabled=false \
    --config trust-manager.secretTargets.enabled=true \
    --config trust-manager.secretTargets.authorizedSecretsAll=true

Step 3: Install the Foundry Local extension

Install the Foundry Local extension by using the Azure portal or Azure CLI. Entra ID authentication is enabled by default. If you plan to use Agentic Retrieval in Foundry Local later, keep this default during installation and include the Entra application ID. If you disable Entra ID authentication, you prevent Agentic Retrieval from connecting to your deployed models.

Option 1: Command line

Use the Azure CLI in Bash or PowerShell to install the extension.

az k8s-extension create \
    --resource-group <resource_group_of_the_arc_cluster> \
    --cluster-name <arc_cluster_name> \
    --name "inference-operator" \
    --extension-type Microsoft.Foundry \
    --scope cluster \
    --release-namespace "foundry-local-operator" \
    --cluster-type connectedClusters \
    --auto-upgrade-minor-version true \
    --release-train stable \
    --config entraAuth.tenantId="<azure_tenant_id>" \
    --config entraAuth.clientId="<the_client_id_of_the_app_registration>"

Additional installation parameters

You can configure the following optional parameters during inference operator installation:

Parameter Description
entraAuth.enabled Boolean. When enabled, the Entra Auth SDK sidecar and msi-adapter sidecar are injected into inference pods for JWT validation and ARM RBAC authorization. When disabled, entraAuth.tenantId and entraAuth.clientId parameters are optional. Default: true. For more information, see Configure authentication for Foundry Local enabled by Azure Arc. If you intend to use Agentic Retrieval in Foundry Local later, you must enable Entra ID authentication for the Foundry Local extension.
watch.namespaces Array of strings. Configure this parameter if you want the operator to manage resources across multiple namespaces. By default, the operator manages the foundry-local-operator namespace where models and inference workloads are deployed. Pass the installation command as: --config watch.namespaces[0]="NS1" --config watch.namespaces[1]="NS2". For more information, see Namespace configuration for model deployments.
storeModel.cacheJob.resources The StoreModel cache job is configured with default memory values of 16Gi for requests and 32Gi for limits. These settings are established due to the typically large size of models, as downloading and caching them requires substantial memory to prevent out-of-memory (OOM) issues. In environments with limited hardware resources, or when working exclusively with smaller models, you can install the Foundry Local extension with reduced memory allocations for the model cache job: --config storeModel.cacheJob.resources.requests.memory="<value lower than 16Gi>" --config storeModel.cacheJob.resources.limits.memory="<value lower than 32Gi>". Ensure that the requests value doesn't exceed the limits value. For more information on StoreModel, see Model caching.

Configure Inference API exposure The api.exposure Helm value controls how the Foundry Local Inference API control-plane endpoint is exposed. This value is separate from spec.endpoint.exposure, which controls individual model data-plane endpoints. By default, api.exposure is set to internal, so the operator creates an HTTPRoute through the cluster-internal Gateway and the API is reachable inside the cluster at the /inference-api path. Set api.exposure: external when the Inference API must be reachable through the external LoadBalancer Gateway; this setting attaches the API route to the external Gateway and exposes it at https://<external-gateway-address>/inference-api. Set api.exposure: none to skip Gateway routing and keep the API reachable only through its ClusterIP service, for example by using kubectl port-forward.

Value Behavior
internal Default. Routes the Inference API through the internal Gateway only.
external Routes the Inference API through both the internal Gateway and external LoadBalancer Gateway.
none Creates no HTTPRoute; the API remains reachable only through the ClusterIP service.

Option 2: Azure portal

Use the Azure portal to install the Foundry Local extension and configure required settings for your Arc-enabled Kubernetes cluster.

  1. In the Azure portal, go to your Azure Arc–enabled Kubernetes cluster on Azure Local.

  2. Select Settings > Extensions > + Add.

  3. From the list of available extensions, select Foundry Local on Azure Local (Preview).

  4. Select Create.

  5. On the Basics tab, provide the following information:

    Field Value
    Subscription Select the subscription that contains your Arc–enabled Kubernetes cluster.
    Resource group Select the resource group that contains your Azure Arc cluster.
    Region Select the region where you want to deploy the extension.
    Connected K8S cluster Select your Arc–enabled Kubernetes cluster.
    Extension name Provide a name for the extension (for example, foundry).

    Screenshot of the Basics tab where you select the subscription, resource group, region, connected cluster, and extension name.

  6. Select Next.

  7. On the Configuration tab, provide the following information:

    Field Value
    Microsoft Entra ID Choose Enabled or Disabled. When enabled, you must provide the Entra application ID for authentication.
    Entra application ID Required only when Microsoft Entra ID is enabled. Enter the application ID from your enterprise application registration.
    Kubernetes namespaces Optional. Enter a comma-separated list of Kubernetes namespaces where you want to allow model deployments. If left empty, models deploy only to the foundry-local-operator namespace.

    Screenshot of the Configuration tab where you enable Microsoft Entra ID and optionally specify Kubernetes namespaces for model deployments.

  8. Select Review + Create.

  9. Review and validate the parameters you provided.

  10. Select Create to deploy the Foundry Local extension.

  11. After the deployment completes, under Extensions, verify that the extension state is Succeeded.

Step 4: Verify the operator

Verify that the inference operator extension is installed and that all pods are running. Use the following commands to check the operator status:

kubectl get pods -n foundry-local-operator
kubectl get crd | grep foundry

Wait until all pods show a Running status before you proceed.

The following screenshots show an example of the expected output:

Screenshot of terminal output from kubectl get pods command showing five pods in the foundry-local-operator namespace with Running or Completed status.

Screenshot of terminal output from kubectl get crd command showing four Foundry Local custom resource definitions registered in the cluster.

Troubleshoot your deployment

Use the following commands to troubleshoot issues with your deployment.

Check ModelDeployment status and events:

kubectl describe mdep <name>

Check operator logs:

kubectl logs -f deployment/inference-operator -n foundry-local-operator

Check pod status:

kubectl get pods -l app.kubernetes.io/managed-by=inference-operator
kubectl describe pod <pod-name>
kubectl logs <pod-name>

List all resources created by a deployment:

kubectl get deploy,svc,ing -l foundry.azure.com/deployment=<name>

Check the catalog ConfigMap:

kubectl get configmap foundry-local-catalog -n foundry-local-operator -o yaml

Verify a Model CR exists:

kubectl get models
kubectl describe model <name>

Next step

Deploy your first model and run inference

Related content

  • Last updated on