Create and manage prompt flow runtimes in Azure AI Studio

Article
03/28/2024

Note

Azure AI Studio is currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

In Azure AI Studio, you can create and manage prompt flow runtimes. You need a runtime to use prompt flow.

A prompt flow runtime has computing resources that are required for the application to run, including a Docker image that contains all necessary dependency packages. In addition to flow execution, Azure AI Studio uses the runtime to ensure the accuracy and functionality of the tools incorporated within the flow when you make updates to the prompt or code content.

Azure AI Studio supports the following types of runtimes:

Runtime type	Underlying compute type	Life cycle management	Customize environment
Automatic runtime (preview)	Serverless compute and Compute instance	Automatic	Easily customize packages
Compute instance runtime	Compute instance	Manual	Manually customize via Azure Machine Learning environment

If you're a new user, we recommend that you use the automatic runtime (preview). You can easily customize the environment by adding packages in the requirements.txt file in flow.dag.yaml in the flow folder.

If you want manage compute resource by your self, you can use compute instance as compute type in automatic runtime or use compute instance runtime.

Create a runtime

Create an automatic runtime on a flow page

Automatic is the default option for a runtime. You can start an automatic runtime by selecting an option from the runtime dropdown list on a flow page:

Select Start. Start creating an automatic runtime by using the environment defined in flow.dag.yaml in the flow folder, it runs on the virtual machine (VM) size of serverless compute which you have enough quota in the workspace.
Select Start with advanced settings. In the advanced settings, you can:
- Select compute type. You can choose between serverless compute and compute instance.
  - If you choose serverless compute, you can set following settings:
    - Customize the VM size that the runtime uses.
    - Customize the idle time, which saves code by deleting the runtime automatically if it isn't in use.
    - Set the user-assigned managed identity. The automatic runtime uses this identity to pull a base image and install packages. Make sure that the user-assigned managed identity has Azure Container Registry pull permission.
      
      If you don't set this identity, we use the user identity by default. Learn more about how to create and update user-assigned identities for a workspace.
  - If you choose compute instance, you can only set idle shutdown time.
    - As it is running on an existing compute instance the VM size is fixed and cannot change in runtime side.
    - Identity used for this runtime also is defined in compute instance, by default it uses the user identity. Learn more about how to assign identity to compute instance
    - For the idle shutdown time it is used to define life cycle of the runtime, if the runtime is idle for the time you set, it will be deleted automatically. And of you have idle shut down enabled on compute instance, then it will continue

Create a compute instance runtime on a runtime page

Sign in to Azure AI Studio and select your project from the Build page. If you don't have a project, create one.
On the collapsible left menu, select AI project settings.
In the Compute instances section, select View all.
Make sure that a compute instance is available and running. If you don't have a compute instance, you can create one in Azure AI Studio.
Select the Prompt flow runtimes tab.
Select Create.
Select the compute instance for the runtime, and then select Create.
Acknowledge the warning that the compute instance will be restarted by selecting Confirm.
On the page for runtime details, monitor the status of the runtime. The runtime has a status of Not available until it's ready. This process can take a few minutes.
When the runtime is ready, the status changes to Running. You might need to select Refresh to see the updated status.
Select the runtime on the Prompt flow runtimes tab to see its details.

Update a runtime on the UI

Update an automatic runtime on a flow page

On a flow page, you can use the following options to manage an automatic runtime:

Install packages Open requirements.txt in prompt flow UI, you can add packages in it.
View installed packages shows the packages that are installed in the runtime. It includes the packages baked to base image and packages specify in the requirements.txt file in the flow folder.
Reset deletes the current runtime and creates a new one with the same environment. If you encounter a package conflict, you can try this option.
Edit opens the runtime configuration page, where you can define the VM side and the idle time for the runtime.
Stop deletes the current runtime. If there's no active runtime on the underlying compute, the compute resource is also deleted.

You can also customize the environment that you use to run this flow by adding packages in the requirements.txt file in the flow folder. After you add more packages in this file, you can choose either of these options:

Save and install triggers pip install -r requirements.txt in the flow folder. The process can take a few minutes, depending on the packages that you install.
Save only just saves the requirements.txt file. You can install the packages later yourself.

Note

You can change the location and even the file name of requirements.txt, but be sure to also change it in the flow.dag.yaml file in the flow folder.

Don't pin the version of promptflow and promptflow-tools in requirements.txt, because we already include them in the runtime base image.

Add packages in a private feed in Azure DevOps

If you want to use a private feed in Azure DevOps, follow these steps:

Create a user-assigned managed identity and add this identity in the Azure DevOps organization. To learn more, see Use service principals and managed identities.

Note

If the Add Users button isn't visible, you probably don't have the necessary permissions to perform this action.
Add or update user-assigned identities to your project.
Add {private} to your private feed URL. For example, if you want to install test_package from test_feed in Azure devops, add -i https://{private}@{test_feed_url_in_azure_devops} in requirements.txt:
```
-i https://{private}@{test_feed_url_in_azure_devops}
test_package
```
Specify the user-assigned managed identity in Start with advanced settings if automatic runtime isn't running, or use the Edit button if automatic runtime is running.

Change the base image for automatic runtime (preview)

By default, we use the latest prompt flow image as the base image. If you want to use a different base image, you need build your own base image, this docker image should be built from prompt flow base image that is mcr.microsoft.com/azureml/promptflow/promptflow-runtime:<newest_version>. If possible use the latest version of the base image. To use the new base image, you need to reset the runtime via the reset command. This process takes several minutes as it pulls the new base image and reinstalls packages.

environment:
    image: <your-custom-image>
    python_requirements_txt: requirements.txt

Update a compute instance runtime on a runtime page

Azure AI Studio gets regular updates to the base image (mcr.microsoft.com/azureml/promptflow/promptflow-runtime-stable) to include the latest features and bug fixes. To get the best experience and performance, periodically update your runtime to the latest version.

Go to the page for runtime details and select Update. On the Edit compute instance runtime pane, you can update the runtime environment. If you select Use default environment, the system tries to update your runtime to the latest version.

Every time you open the page for runtime details, AI Studio checks whether there are new versions of the runtime. If new versions are available, a notification appears at the top of the page. You can also manually check the latest version by selecting the Check version button.

Switch compute instance runtime to automatic runtime

Automatic runtime has following advantages over compute instance runtime:

Automatic manage lifecycle of runtime and underlying compute. You don't need to manually create and managed them anymore.
Easily customize packages by adding packages in the requirements.txt file in the flow folder, instead of creating a custom environment.

We would recommend you to switch to automatic runtime, if you're using compute instance runtime. You can switch it to an automatic runtime by using the following steps:

Prepare your requirements.txt file in the flow folder. Make sure that you don't pin the version of promptflow and promptflow-tools in requirements.txt, because we already include them in the runtime base image. Automatic runtimewill install the packages in requirements.txt file when it starts.
If you create custom environment to create compute instance runtime, you can also use get the image from environment detail page, and specify it in flow.dag.yaml file in the flow folder. To learn more, see Change the base image for automatic runtime. Make sure you have acr pull permission for the image.
For compute resource, you can continue to use the existing compute instance if you would like to manually manage the lifecycle of compute resource or you can try serverless compute which lifecycle is managed by system.