Error InvalidAuthenticationTokenTenant on az ml online-deployment create from BICEP

Question

I am getting this error when trying to create a model deployment with a BICEP file.

For the context, I am on a pipeline, running on ubuntu image on a service connection that has: owner and Azure AI Inference Deployment Operator roles on the resources group.

The pipeline has several tasks where it successfully runs:

        az ml workspace create --name $(workspaceName) --location $(location) --resource-group $(resourceGroupName)
        az ml online-endpoint create --name $(endPointName) --resource-group $(resourceGroupName) --workspace-name $(workspaceName) --auth-mode key

Then I have this task that fails, I added some debugging after reading a similar issue:

  - task: AzureCLI@2
    name: CreateDeployment
    inputs:
      azureSubscription: $(azureServiceConnection)
      scriptType: 'bash'
      scriptLocation: 'inlineScript'
      inlineScript: |
        echo "RUNNING: az extension add -n ml -y"
        az extension add -n ml -y
        echo "RUNNING: az account show"
        az account show
        echo "RUNNING: az ml online-deployment create"
        az ml online-deployment create --file ./deploy_model_to_endpoint.yml --name $(deploymentName) --endpoint-name $(endPointName) --resource-group $(resourceGroupName) --workspace-name $(workspaceName) --subscription *my_subscription*

It generated these logs:

	RUNNING: az extension add -n ml -y
	RUNNING: az account show 
	{
	  "environmentName": "AzureCloud",
	  "homeTenantId": "*my_tenant*",
	  "id": "*my_subscription*",
	  "isDefault": true,
	  "managedByTenants": [],
	  "name": "OECD DKI (Lab)",
	  "state": "Enabled",
	  "tenantId": "*my_tenant*",
	  "user": {
	    "name": "***",
	    "type": "servicePrincipal"
	  }
	}
	RUNNING: az ml online-deployment create
	Check: endpoint *my_endpoint* exists
	ERROR: (InvalidAuthenticationTokenTenant) The access token is from the wrong issuer 'https://sts.windows.net/*my_tenant*/'. 
	It must match the tenant 'https://sts.windows.net/*another_tenant_ending_with_5e33d*/' associated with this subscription. 
	Please use the authority (URL) 'https://login.windows.net/*another_tenant_ending_with_5e33d*' to get the token. 
	Note, if the subscription is transferred to another tenant there is no impact to the services, but information about new tenant could take time to propagate (up to an hour). 	If you just transferred your subscription and see this error message, please try back later.
	Code: InvalidAuthenticationTokenTenant

I am not in the 'transferred' case, I managed to create an endpoint and deployment from AI Studio and it worked, I need to do it from a pipeline, I found an article explaining that [another_tenant_ending_with_5e33d] has to do with private endpoints but couldn't find what to do with my issue.

Thanks for any help,

Claude

Answer 1

The error message (InvalidAuthenticationTokenTenant) you're encountering typically occurs when the access token being used to authenticate against Azure services is issued from a different tenant than the one associated with the subscription you are trying to operate in. Based on your logs, it looks like the tenant associated with your subscription (ending with 5e33d) is different from the tenant issuing the token (which is tied to your service connection).

Here are some steps you can take to troubleshoot and resolve this issue:

1. Check Azure Service Connection Configuration

• Ensure that the Azure service connection (defined by $(azureServiceConnection) in your YAML) is configured to use the correct tenant. If the service principal used in the pipeline is from a different tenant than the one linked to your subscription, you will see this error.
  • Go to Azure DevOps > Project Settings > Service Connections > Edit your service connection, and confirm that the correct tenant ID and subscription ID are used.
  • Make sure that the service principal has access to both the subscription and the resource group where you're deploying the model.

2. Switch to the Correct Tenant in the Pipeline

• Use az account tenant set to switch the tenant to the correct one before the deployment. You can add this step in your pipeline YAML script:

  
       - task: AzureCLI@2
  
         name: SwitchTenant
  
         inputs:
  
           azureSubscription: $(azureServiceConnection)
  
           scriptType: 'bash'
  
           scriptLocation: 'inlineScript'
  
           inlineScript: |
  
             echo "Switching to the correct tenant..."
  
             az account tenant set --tenant <correct_tenant_id>
  
  

  • This ensures that all subsequent operations in the pipeline use the correct tenant.

3. Use az login with the Correct Tenant

If the service connection cannot be reconfigured, you can force the az ml commands to authenticate with the correct tenant using az login or by specifying the --tenant option explicitly when creating the deployment. For example:

   az ml online-deployment create --file ./deploy_model_to_endpoint.yml --name $(deploymentName) --endpoint-name $(endPointName) --resource-group $(resourceGroupName) --workspace-name $(workspaceName) --subscription *my_subscription* --tenant <correct_tenant_id>

4. Ensure Correct Role Assignment

• Verify that the service principal used in the service connection has the necessary roles (Owner, Azure AI Inference Deployment Operator) on the correct tenant and subscription. If the service principal does not have access to the correct tenant, it will fail during the deployment step.
  • If needed, assign the correct role on the subscription in the target tenant by navigating to Azure Portal > Subscriptions > Access Control (IAM) > Add role assignment.

5. Check for Private Endpoint Configuration

If your deployment involves private endpoints, ensure that your network settings are correct. Sometimes, private endpoints require different configurations for accessing resources, and the wrong tenant error can be a symptom of an underlying network misconfiguration. Check Azure Private Link settings to ensure that you're routing correctly through the correct tenant and subscription.

By following these steps, you should be able to resolve the InvalidAuthenticationTokenTenant error. Let me know if you need further clarification or assistance!

Answer 2

Hello,

Thank you very much for your replies.

I tested the different suggestions and still have no success.

Here are the logs of my tests :

Setting AZURE_CONFIG_DIR env variable to: /home/vsts/work/_temp/.azclitask
Setting active cloud to: AzureCloud
/usr/bin/az cloud set -n AzureCloud
/usr/bin/az login --service-principal -u *** --password=*** --tenant ***471c --allow-no-subscriptions
[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "***471c",
    "id": "***0fde",
    "isDefault": true,
    "managedByTenants": [],
    "name": "***",
    "state": "Enabled",
    "tenantId": "***471c",
    "user": {
      "name": "***",
      "type": "servicePrincipal"
    }
  }
]
>az account tenant list
[
  {
    "id": "/tenants/***471c",
    "tenantId": "***471c"
  }
]
>az account clear
>az account set --name ***e086
ERROR: The subscription of '***' doesn't exist in cloud 'AzureCloud'
>az account tenant set --tenant ***471c
ERROR: 'tenant' is misspelled or not recognized by the system.
>az account show
{
  "environmentName": "AzureCloud",
  "homeTenantId": "***471c",
  "id": "***0fde",
  "isDefault": true,
  "managedByTenants": [],
  "name": "***",
  "state": "Enabled",
  "tenantId": "***471c",
  "user": {
    "name": "***",
    "type": "servicePrincipal"
  }
}

>az ml online-deployment create --file ./deploy_model_to_endpoint.yml --name $(deploymentName) --endpoint-name $(endPointName) --resource-group $(resourceGroupName) --workspace-name $(workspaceName) --subscription ***0fde --tenant ***471c
ERROR: unrecognized arguments: --tenant ***471c

I am not sure about what is 'AzureCloud', it is not something I called for but it seems to me that it

by default.

I added tested previously the Tenant options and already had those error messages.

This leaves your comment arond privage endpoint, I don't know much about this, I was requested to provide an URL in our resources group that enables us to provide a sentence, the URL must interrogate an AI Model and return a translation of the sentence, is a private endpoint really what I need ? In that case, would I need a VNET and specific network configurations ?

Thanks a lot for your guidance.