Edit

Get values for environment variables

The Microsoft 365 Copilot Agent Evaluations CLI (@microsoft/m365-copilot-eval) reads connection and authentication settings from environment variables. This article shows you where to obtain each value.

For the list of variables and where to put them, see Agent Evaluations CLI overview and Quickstart: Use the Agent Evaluations CLI.

Note

The Agent Evaluations CLI is currently in preview. Features and functionality are subject to change.

Before you begin

Gather the following items before you collect values:

  • Access to the Azure subscription that hosts your Azure OpenAI in Foundry Models resource.
  • Sign-in access to the Microsoft 365 tenant where your Copilot agent is deployed.
  • The Azure role needed to view keys on your Azure OpenAI resource (for example, Cognitive Services Contributor or Owner). If you can't see keys, ask your subscription administrator.

Get your tenant ID (TENANT_ID)

TENANT_ID is the Microsoft Entra tenant ID where your agent is deployed. It's a GUID in the form xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

Use any of the following methods.

Option 1: Microsoft Entra admin center

  1. Sign in to the Microsoft Entra admin center with an account in the target tenant.
  2. Select Identity > Overview.
  3. Copy the Tenant ID value.

Option 2: Azure CLI

If you have the Azure CLI installed:

az account show --query tenantId --output tsv

If you belong to multiple tenants, sign in to the correct one first:

az login --tenant <tenant-domain-or-id>

Option 3: Azure portal

  1. Sign in to the Azure portal.
  2. Select Microsoft Entra ID from the portal menu.
  3. Copy the Tenant ID from the Overview page.

Tip

If runevals connects but reports that your agent can't be found, the most common cause is a TENANT_ID that doesn't match the tenant your agent is deployed to. See Tenant mismatch.

Get your Azure OpenAI endpoint and API key

The evaluation tool uses an Azure OpenAI resource to score responses. You need both the endpoint URL and API key from your Azure OpenAI resource for evaluations. The Azure OpenAI endpoint can be in any tenant or account. You configure the evaluations tool by using AZURE_AI_OPENAI_ENDPOINT and AZURE_AI_API_KEY.

Your Azure OpenAI resource must have a chat model deployed. By default, the CLI uses gpt-4o-mini; to use a different deployment, see Set the API version and model.

If you don't already have an Azure OpenAI resource with a deployed model, follow these steps to create one:

  1. Sign in to the Azure portal.

  2. In the search bar, enter OpenAI and select Azure OpenAI from the results.

    Screenshot of the Azure portal search bar showing the Azure OpenAI service.

  3. After you select Azure OpenAI, select Create to create an AI Microsoft Foundry resource.

    Screenshot of the Azure OpenAI service page with the Create AI Foundry resource button.

  4. On the Create AI Foundry resource form, fill in the details, and then select Review + create.

    Screenshot of the Create AI Foundry resource form with the Review + create button.

  5. After the resource is deployed, open the Azure AI Foundry portal.

    Screenshot of the resource deployment complete page with a link to the Azure AI Foundry portal.

  6. In the left navigation, select Models + endpoints.

    Screenshot of the AI Foundry portal left navigation with Models + endpoints selected.

  7. Select Deploy model > Deploy base model. Use the gpt-4o-mini model.

    Screenshot of the Deploy model dropdown showing the Deploy base model option.

  8. Select Confirm, and then select Customize.

    Screenshot of the model deployment confirmation dialog with the Customize button.

  9. Change the capacity to 50K tokens per minute.

    Screenshot of the model deployment customization page showing the token capacity setting.

    Screenshot of the token capacity set to 50K tokens per minute.

  10. Select Deploy and wait a few minutes for the model to deploy.

  11. After deployment finishes, you're redirected to the API endpoint and API key page. Copy the endpoint and key values shown.

    Screenshot of the API endpoint and API key values on the model deployment page.

  12. Add these values to your env file. For details, see Step 3 of the Quickstart.

Warning

Treat API keys as secrets. Store them in .env.local.user or env/.env.<environment> - never in source control. See Configure environment variables for the correct file for your project type.

Get your agent ID (M365_AGENT_ID)

M365_AGENT_ID identifies the Copilot agent that runevals sends test prompts to. The CLI accepts two formats:

  • User-scoped: U_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  • Tenant-scoped: T_<agent-name>.declarativeAgent

Microsoft 365 Agents Toolkit projects

If you built your agent by using the Microsoft 365 Agents Toolkit, you don't need to set M365_AGENT_ID directly. The CLI auto-detects the agent ID from M365_TITLE_ID, which the Agents Toolkit writes to .env.local during provisioning.

To confirm, open .env.local in your project root and look for a line similar to:

M365_TITLE_ID="U_0dc4a8a2-b95f-edac-91c8-d802023ec2d4"

Non-Microsoft 365 Agents Toolkit projects

Set M365_AGENT_ID explicitly in env/.env.<environment>. Get the value from the location where your agent was published - for example, the Microsoft Teams admin center, the Microsoft 365 admin center, or the deployment output from your publishing pipeline.

Override the agent at run time

To evaluate a different agent without editing env files, pass --m365-agent-id on the command line:

runevals --m365-agent-id "U_0dc4a8a2-b95f-edac-91c8-d802023ec2d4"

For details, see --m365-agent-id in the CLI reference.

Set the API version and model

Two variables control which Azure OpenAI REST API version and which model deployment the CLI uses for scoring. They have working defaults, so you can accept the defaults or set them explicitly in your env file:

AZURE_AI_API_VERSION="2024-12-01-preview"   # version
AZURE_AI_MODEL_NAME="gpt-4o-mini"           # actual name in Foundry
Variable Default When to change
AZURE_AI_API_VERSION 2024-12-01-preview Your resource requires a different Azure OpenAI REST API version.
AZURE_AI_MODEL_NAME gpt-4o-mini You want to score with a different deployment in your Azure OpenAI resource.

AZURE_AI_MODEL_NAME must match the deployment name exactly as it appears in your Azure OpenAI resource.

Get AZURE_AI_MODEL_NAME from Foundry

  1. Sign in to the Azure portal and open your Azure OpenAI resource.
  2. Select Resource Management > Model deployments (or open the resource in Foundry).
  3. Copy the Deployment name of the model you want to use for scoring.

Validate your values

After you gather the values and add them to the appropriate env file (see Step 3 of the Quickstart), run:

runevals --init-only --log-level debug

This command loads your environment, initializes the Python runtime, and surfaces configuration errors without running evaluations. If it completes without error, your values are loaded correctly.

Warning

--log-level debug might print raw API payloads. Review the output before sharing it. Redaction is pattern-based and might not catch every key or identifier.

Troubleshooting

Use the following table to troubleshoot issues you might encounter.

Symptom Likely cause Next step
Authentication error on startup Signed in to the wrong Microsoft 365 account, or TENANT_ID doesn't match your agent's tenant. See Authentication errors.
Tool connects but no agent is found TENANT_ID mismatch, or the agent isn't deployed to that tenant. See Tenant mismatch.
Scoring fails with 401 or 403 Wrong or expired AZURE_AI_API_KEY, or endpoint/key from different resources. See Azure OpenAI API key errors.
Value in env file is ignored Another env file earlier in the precedence order sets the same variable. See Environment file precedence.

Related content

  • Last updated on