Deploy models as serverless API endpoints

In this article, you learn how to deploy a model from the model catalog as a serverless API with pay-as-you-go token-based billing.

Certain models in the model catalog can be deployed as a serverless API with pay-as-you-go billing. This kind of deployment provides a way to consume models as an API without hosting them on your subscription, while keeping the enterprise security and compliance that organizations need. This deployment option doesn't require quota from your subscription.

Prerequisites

• An Azure subscription with a valid payment method. Free or trial Azure subscriptions won't work. If you don't have an Azure subscription, create a paid Azure account to begin.

• An Azure Machine Learning workspace.

• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the Azure AI Developer role on the resource group. For more information on permissions, see Role-based access control in Azure Machine Learning.

• You need to install the following software to work with Azure Machine Learning:

  Studio

  You can use any compatible web browser to navigate Azure Machine Learning.

  Azure CLI

  The Azure CLI and the ml extension for Azure Machine Learning.

  az extension add -n ml
  

  If you already have the extension installed, ensure the latest version is installed.

  az extension update -n ml
  

  Once the extension is installed, configure it:

  az account set --subscription <subscription>
  az configure --defaults workspace=<workspace-name> group=<resource-group> location=<location>
  

  Python SDK

  Install the Azure Machine Learning SDK for Python.

  pip install -U azure-ai-ml
  

  Once installed, import necessary namespaces and create a client connected to your workspace:

  from azure.ai.ml import MLClient
  from azure.identity import InteractiveBrowserCredential
  from azure.ai.ml.entities import MarketplaceSubscription, ServerlessEndpoint
  
  client = MLClient(
      credential=InteractiveBrowserCredential(tenant_id="<tenant-id>"),
      subscription_id="<subscription-id>",
      resource_group_name="<resource-group>",
      workspace_name="<workspace-name>",
  )
  

  ARM

  You can use any compatible web browser to deploy ARM templates in the Microsoft Azure portal or using any of the deployment tools. This tutorial uses the Azure CLI.

Subscribe your workspace to the model offering

For models offered through the Azure Marketplace, you can deploy them to serverless API endpoints to consume their predictions. If it's your first time deploying the model in the workspace, you have to subscribe your workspace for the particular model offering from the Azure Marketplace. Each workspace has its own subscription to the particular Azure Marketplace offering of the model, which allows you to control and monitor spending.

Note

Models offered through the Azure Marketplace are available for deployment to serverless API endpoints in specific regions. Check Region availability for models in Serverless API endpoints to verify which regions are available. If the one you need is not listed, you can deploy to a workspace in a supported region and then consume serverless API endpoints from a different workspace.

1. Sign in to Azure Machine Learning studio

2. Ensure your account has the Azure AI Developer role permissions on the resource group, or that you meet the permissions required to subscribe to model offerings.

3. Go to your workspace.

4. Select Model catalog from the left sidebar and find the model card of the model you want to deploy. In this article, you select a Meta-Llama-3-8B-Instruct model.

   1. If you're deploying the model using Azure CLI, Python SDK, or ARM, copy the Model ID.

   Important

   Do not include the version when copying the Model ID. Serverless API endpoints always deploy the model's latest version available. For example, for the model ID azureml://registries/azureml-meta/models/Meta-Llama-3-8B-Instruct/versions/3, copy azureml://registries/azureml-meta/models/Meta-Llama-3-8B-Instruct.

   

5. Create the model's marketplace subscription. When you create a subscription, you accept the terms and conditions associated with the model offer.

   Studio

   1. On the model's Details page, select Deploy and then select Serverless API to open the deployment wizard.

   2. Select the checkbox to acknowledge the Microsoft purchase policy.

      

   3. If you see the note You already have an Azure Marketplace subscription for this workspace, you don't need to create the subscription since you already have one. You can proceed to Deploy the model to a serverless API endpoint.

   4. In the deployment wizard, select the link to Azure Marketplace Terms to learn more about the terms of use. You can also select the Pricing and terms tab to learn about pricing for the selected model.

   5. On the deployment wizard, select the link to Azure Marketplace Terms to learn more about the terms of use. You can also select the Marketplace offer details tab to learn about pricing for the selected model.

   6. Select Subscribe and Deploy.

   Azure CLI

   subscription.yml

   name: meta-llama3-8b-qwerty
   model_id: azureml://registries/azureml-meta/models/Meta-Llama-3-8B-Instruct
   

   Use the subscription.yml file to create the subscription:

   az ml marketplace-subscription create -f subscription.yml
   

   Python SDK

   model_id="azureml://registries/azureml-meta/models/Meta-Llama-3-8B-Instruct"
   subscription_name="Meta-Llama-3-8B-Instruct"
   
   marketplace_subscription = MarketplaceSubscription(
       model_id=model_id,
       name=subscription_name,
   )
   
   marketplace_subscription = client.marketplace_subscriptions.begin_create_or_update(
       marketplace_subscription
   ).result()
   

   ARM

   Use the following template to create a model subscription:

   template.json

   {
       "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
       "contentVersion": "1.0.0.0",
       "parameters": {
           "workspace_name": {
               "defaultValue": "my-workspace",
               "type": "String"
           },
           "subscription_name": {
               "defaultValue": "Meta-Llama-3-8B-Instruct",
               "type": "String"
           },
           "model_id": {
               "defaultValue": "azureml://registries/azureml-meta/models/Meta-Llama-3-8B-Instruct",
               "type": "String"
           }
       },
       "variables": {},
       "resources": [
           {
               "type": "Microsoft.MachineLearningServices/workspaces/marketplaceSubscriptions",
               "apiVersion": "2024-04-01",
               "name": "[concat(parameters('workspace_name'), '/', parameters('subscription_name'))]",
               "location": "[parameters('location')]",
               "properties": {
                   "modelId": "[parameters('model_id')]"
               }
           }
       ]
   }
   

6. Once you sign up the workspace for the particular Azure Marketplace offering, subsequent deployments of the same offering in the same workspace don't require subscribing again.

7. At any point, you can see the model offers to which your workspace is currently subscribed:

   Studio

   1. Go to the Azure portal

   2. Navigate to the resource group where the workspace belongs.

   3. On the Type filter, select SaaS.

   4. You see all the offerings to which you're currently subscribed.

   5. Select any resource to see the details.

   Azure CLI

   az ml marketplace-subscription list
   

   Python SDK

   marketplace_sub_list = client.marketplace_subscriptions.list()
   
   for sub in marketplace_sub_list:
       print(sub.as_dict())
   

   ARM

   You can use the resource management tools to query the resources. The following code uses Azure CLI:

   az resource list \
       --query "[?type=='Microsoft.SaaS']"
   

Deploy the model to a serverless API endpoint

Once you've created a model's subscription, you can deploy the associated model to a serverless API endpoint. The serverless API endpoint provides a way to consume models as an API without hosting them on your subscription, while keeping the enterprise security and compliance organizations need. This deployment option doesn't require quota from your subscription.

In this article, you create an endpoint with name meta-llama3-8b-qwerty.

1. Create the serverless endpoint

   Studio

   1. From the previous wizard, select Deploy (if you've just subscribed the workspace to the model offer in the previous section), or select Continue to deploy (if your deployment wizard had the note You already have an Azure Marketplace subscription for this workspace).

      

   2. Give the deployment a name. This name becomes part of the deployment API URL. This URL must be unique in each Azure region.

      

   3. Select Deploy. Wait until the deployment is ready and you're redirected to the Deployments page.

   Azure CLI

   endpoint.yml

   name: meta-llama3-8b-qwerty
   model_id: azureml://registries/azureml-meta/models/Meta-Llama-3-8B-Instruct
   

   Use the endpoint.yml file to create the endpoint:

   az ml serverless-endpoint create -f endpoint.yml
   

   Python SDK

   endpoint_name="meta-llama3-8b-qwerty"
   
   serverless_endpoint = ServerlessEndpoint(
       name=endpoint_name,
       model_id=model_id
   )
   
   created_endpoint = client.serverless_endpoints.begin_create_or_update(
       serverless_endpoint
   ).result()
   

   ARM

   Use the following template to create an endpoint:

   template.json

   {
       "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
       "contentVersion": "1.0.0.0",
       "parameters": {
           "workspace_name": {
               "defaultValue": "my-workspace",
               "type": "String"
           },
           "endpoint_name": {
               "defaultValue": "meta-llama3-8b-qwerty",
               "type": "String"
           },
           "location": {
               "defaultValue": "eastus2",
               "type": "String"
           },
           "model_id": {
               "defaultValue": "azureml://registries/azureml-meta/models/Meta-Llama-3-8B-Instruct",
               "type": "String"
           }
       },
       "variables": {},
       "resources": [
           {
               "type": "Microsoft.MachineLearningServices/workspaces/serverlessEndpoints",
               "apiVersion": "2024-04-01",
               "name": "[concat(parameters('workspace_name'), '/', parameters('endpoint_name'))]",
               "location": "[parameters('location')]",
               "sku": {
                   "name": "Consumption"
               },
               "properties": {
                   "modelSettings": {
                       "modelId": "[parameters('model_id')]"
                   }
               }
           }
       ]
   }
   

   Then create the deployment:

   az deployment group create \
       --name model-subscription-deployment \
       --resource-group <resource-group> \
       --template-file template.json
   

   The Azure deployment template can take a few minutes to complete. When it finishes, you see a message that includes the result:

   "provisioningState": "Succeeded",
   

2. At any point, you can see the endpoints deployed to your workspace:

   Studio

   1. Go to your workspace.

   2. Select Endpoints.

   3. Select the Serverless endpoints tab to display the serverless API endpoints.

   Azure CLI

   az ml serverless-endpoint list
   

   Python SDK

   endpoint_name="meta-llama3-8b-qwerty"
   
   serverless_endpoint = ServerlessEndpoint(
       name=endpoint_name,
       model_id=model_id
   )
   
   created_endpoint = client.serverless_endpoints.begin_create_or_update(
       serverless_endpoint
   ).result()
   

   ARM

   You can use the resource management tools to query the resources. The following code uses Azure CLI:

   az resource list \
       --query "[?type=='Microsoft.MachineLearningServices/workspaces/serverlessEndpoints']"
   

3. The created endpoint uses key authentication for authorization. Use the following steps to get the keys associated with a given endpoint.

   Studio

   1. To return to the deployment's page, select the endpoint's name from the list of serverless endpoints.
   2. Note the endpoint's Target URI and Key. Use them to call the deployment and generate predictions.

   Note

   When using the Azure portal, serverless API endpoints aren't displayed by default on the resource group. Use the Show hidden types option to display them on the resource group.

   Azure CLI

   az ml serverless-endpoint get-credentials -n meta-llama3-8b-qwerty
   

   Python SDK

   endpoint_keys = client.serverless_endpoints.get_keys(endpoint_name)
   print(endpoint_keys.primary_key)
   print(endpoint_keys.secondary_key)
   

   ARM

   Use REST APIs to query this information.

4. At this point, your endpoint is ready to be used.

5. If you need to consume this deployment from a different workspace, or you plan to use prompt flow to build intelligent applications, you need to create a connection to the serverless API deployment. To learn how to configure an existing serverless API endpoint on a new project or hub, see Consume deployed serverless API endpoints from a different workspace or from Prompt flow.

   Tip

   If you're using prompt flow in the same workspace where the deployment was deployed, you still need to create the connection.

Using the serverless API endpoint

Models deployed in Azure Machine Learning and Azure AI studio in Serverless API endpoints support the Azure AI Model Inference API that exposes a common set of capabilities for foundational models and that can be used by developers to consume predictions from a diverse set of models in a uniform and consistent way.

Read more about the capabilities of this API and how you can leverage it when building applications.

Delete endpoints and subscriptions

You can delete model subscriptions and endpoints. Deleting a model subscription makes any associated endpoint to become Unhealthy and unusable.

Studio

To delete a serverless API endpoint:

1. Go to Azure Machine Learning studio.

2. Select Endpoints from the left sidebar.

3. Select the Serverless endpoints tab to display the serverless API endpoints.

4. Open the endpoint you want to delete.

5. Select Delete.

To delete the associated model subscription:

1. Go to the Azure portal

2. Navigate to the resource group where the workspace belongs.

3. On the Type filter, select SaaS.

4. Select the subscription you want to delete.

5. Select Delete.

Azure CLI

To delete a serverless API endpoint:

az ml serverless-endpoint delete \
    --name "meta-llama3-8b-qwerty"

To delete the associated model subscription:

az ml marketplace-subscription delete \
    --name "Meta-Llama-3-8B-Instruct"

Python SDK

To delete a serverless API endpoint:

client.serverless_endpoints.begin_delete(endpoint_name).wait()

To delete the associated model subscription:

client.marketplace_subscriptions.begin_delete(subscription_name).wait()

ARM

You can use the resource management tools to manage the resources. The following code uses Azure CLI:

az resource delete --name <resource-name>

Cost and quota considerations for models deployed as serverless API endpoints

Models deployed as a serverless API endpoint are offered through the Azure Marketplace and integrated with Azure Machine Learning for use. You can find the Azure Marketplace pricing when deploying or fine-tuning the models.

Each time a workspace subscribes to a given offer from the Azure Marketplace, a new resource is created to track the costs associated with its consumption. The same resource is used to track costs associated with inference and fine-tuning; however, multiple meters are available to track each scenario independently.

For more information on how to track costs, see Monitor costs for models offered through the Azure Marketplace.

Quota is managed per deployment. Each deployment has a rate limit of 200,000 tokens per minute and 1,000 API requests per minute. However, we currently limit one deployment per model per workspace. Contact Microsoft Azure Support if the current rate limits aren't sufficient for your scenarios.

Permissions required to subscribe to model offerings

Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the Owner, Contributor, or Azure AI Developer role for the Azure subscription. Alternatively, your account can be assigned a custom role that has the following permissions:

• On the Azure subscription—to subscribe the workspace to the Azure Marketplace offering, once for each workspace, per offering:

  • Microsoft.MarketplaceOrdering/agreements/offers/plans/read
  • Microsoft.MarketplaceOrdering/agreements/offers/plans/sign/action
  • Microsoft.MarketplaceOrdering/offerTypes/publishers/offers/plans/agreements/read
  • Microsoft.Marketplace/offerTypes/publishers/offers/plans/agreements/read
  • Microsoft.SaaS/register/action

• On the resource group—to create and use the SaaS resource:

  • Microsoft.SaaS/resources/read
  • Microsoft.SaaS/resources/write

• On the workspace—to deploy endpoints (the Azure Machine Learning data scientist role contains these permissions already):

  • Microsoft.MachineLearningServices/workspaces/marketplaceModelSubscriptions/*
  • Microsoft.MachineLearningServices/workspaces/serverlessEndpoints/*

For more information on permissions, see Manage access to an Azure Machine Learning workspace.

Related content

• Model Catalog and Collections
• Consume deployed serverless API endpoints from a different workspace