Deploy an application that uses OpenAI on Azure Kubernetes Service (AKS)
Article
In this article, you learn how to deploy an application that uses Azure OpenAI or OpenAI on AKS. With OpenAI, you can easily adapt different AI models, such as content generation, summarization, semantic search, and natural language to code generation, for your specific tasks. You start by deploying an AKS cluster in your Azure subscription. Then, you deploy your OpenAI service and the sample application.
The sample cloud native application is representative of real-world implementations. The multi-container application is comprised of applications written in multiple languages and frameworks, including:
Golang with Gin
Rust with Actix-Web
JavaScript with Vue.js and Fastify
Python with FastAPI
These applications provide front ends for customers and store admins, REST APIs for sending data to RabbitMQ message queue and MongoDB database, and console apps to simulate traffic.
Note
We don't recommend running stateful containers, such as MongoDB and Rabbit MQ, without persistent storage for production. We use them here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.
To access the GitHub codebase for the sample application, see AKS Store Demo.
Before you begin
You need an Azure account with an active subscription. If you don't have one, create an account for free.
For this demo, you can either use Azure OpenAI service or OpenAI service.
If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.
If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.
When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.
Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.
Create a resource group
An Azure resource group is a logical group in which you deploy and manage Azure resources. When you create a resource group, you're prompted to specify a location. This location is the storage location of your resource group metadata and where your resources run in Azure if you don't specify another region during resource creation.
The following example creates a resource group named myResourceGroup in the eastus location.
If your Linux-based system requires elevated permissions, you can use the sudo az aks install-cli command.
Configure kubectl to connect to your Kubernetes cluster using the az aks get-credentials command.
This command executes the following operations:
Downloads credentials and configures the Kubernetes CLI to use them.
Uses ~/.kube/config, the default location for the Kubernetes configuration file. Specify a different location for your Kubernetes configuration file using --file argument.
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
Verify the connection to your cluster using the kubectl get command. This command returns a list of the cluster nodes.
kubectl get nodes
The following example output shows the nodes created in the previous steps. Make sure the node status is Ready.
NAME STATUS ROLES AGE VERSION
aks-nodepool1-31469198-vmss000000 Ready agent 3h29m v1.25.6
aks-nodepool1-31469198-vmss000001 Ready agent 3h29m v1.25.6
aks-nodepool1-31469198-vmss000002 Ready agent 3h29m v1.25.6
Note
For private clusters, the nodes might be unreachable if you try to connect to them through the public IP address. In order to fix this, you need to create an endpoint within the same VNET as the cluster to connect from. Follow the instructions to Create a private AKS cluster and then connect to it.
Deploy the application
The AKS Store application manifest includes the following Kubernetes deployments and services:
Product service: Shows product information.
Order service: Places orders.
Makeline service: Processes orders from the queue and completes the orders.
Store front: Web application for customers to view products and place orders.
Store admin: Web application for store employees to view orders in the queue and manage product information.
Virtual customer: Simulates order creation on a scheduled basis.
Virtual worker: Simulates order completion on a scheduled basis.
Mongo DB: NoSQL instance for persisted data.
Rabbit MQ: Message queue for an order queue.
Note
We don't recommend running stateful containers, such as MongoDB and Rabbit MQ, without persistent storage for production. We use them here for simplicity, but we recommend using managed services, such as Azure CosmosDB or Azure Service Bus.
The following example output shows the successfully created deployments and services:
deployment.apps/mongodb created
service/mongodb created
deployment.apps/rabbitmq created
service/rabbitmq created
deployment.apps/order-service created
service/order-service created
deployment.apps/makeline-service created
service/makeline-service created
deployment.apps/product-service created
service/product-service created
deployment.apps/store-front created
service/store-front created
deployment.apps/store-admin created
service/store-admin created
deployment.apps/virtual-customer created
deployment.apps/virtual-worker created
Deploy OpenAI
You can either use Azure OpenAI or OpenAI and run your application on AKS.
Now that the application is deployed, you can deploy the Python-based microservice that uses OpenAI to automatically generate descriptions for new products being added to the store's catalog.
Set the environment variable USE_AZURE_OPENAI to "True".
Get your Azure OpenAI deployment name from Azure OpenAI studio and fill in the AZURE_OPENAI_DEPLOYMENT_NAME value.
Get your Azure OpenAI endpoint and Azure OpenAI API key from the Azure portal by selecting Keys and Endpoint in the left blade of the resource. Update the AZURE_OPENAI_ENDPOINT and OPENAI_API_KEY in the YAML accordingly.
Deploy the application using the kubectl apply command and specify the name of your YAML manifest.
kubectl apply -f ai-service.yaml
The following example output shows the successfully created deployments and services:
deployment.apps/ai-service created
service/ai-service created
Create a file named ai-service.yaml and copy in the following manifest:
Deploy the application using the kubectl apply command and specify the name of your YAML manifest.
kubectl apply -f ai-service.yaml
The following example output shows the successfully created deployments and services:
deployment.apps/ai-service created
service/ai-service created
Note
Directly adding sensitive information, such as API keys, to your Kubernetes manifest files isn't secure and may accidentally get committed to code repositories. We added it here for simplicity. For production workloads, use Managed Identity to authenticate to Azure OpenAI service instead or store your secrets in Azure Key Vault.
Test the application
Check the status of the deployed pods using the kubectl get pods command.
kubectl get pods
Make sure all the pods are Running before continuing to the next step.
Get the IP of the store admin web application and store front web application using the kubectl get service command.
kubectl get service store-admin
The application exposes the Store Admin site to the internet via a public load balancer provisioned by the Kubernetes service. This process can take a few minutes to complete. EXTERNAL IP initially shows pending until the service comes up and shows the IP address.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
store-admin LoadBalancer 10.0.142.228 40.64.86.161 80:32494/TCP 50m
Repeat the same step for the service named `store-front``.
Open a web browser and browse to the external IP address of your service. In the example shown here, open 40.64.86.161 to see Store Admin in the browser. Repeat the same step for Store Front.
In store admin, select the products tab, then select Add Products.
When the `ai-service`` is running successfully, you should see the Ask OpenAI button next to the description field. Fill in the name, price, and keywords, then generate a product description by selecting Ask OpenAI > Save product.
You can now see the new product you created on Store Admin used by sellers. In the picture, you can see Jungle Monkey Chew Toy is added.
You can also see the new product you created on Store Front used by buyers. In the picture, you can see Jungle Monkey Chew Toy is added. Remember to get the IP address of store front using the kubectl get service command.
The source for this content can be found on GitHub, where you can also create and review issues and pull requests. For more information, see our contributor guide.
Azure Kubernetes Service feedback
Azure Kubernetes Service is an open source project. Select a link to provide feedback: