Share via

Test agents using the Microsoft Agent 365 SDK

Important

You need to be part of the Frontier preview program to get early access to Microsoft Agent 365. Frontier connects you directly with Microsoft’s latest AI innovations. Frontier previews are subject to the existing preview terms of your customer agreements. As these features are still in development, their availability and capabilities may change over time.

Test your agent locally using Agents Playground before deployment. This guide covers setting up your development environment, configuring authentication, and validating your agent's functionality with the Agents Playground testing tool.

Once your agent is working locally, you can deploy and publish it to test in Microsoft 365 applications like Teams.

Prerequisites

Before you begin testing your agent, ensure you have the following prerequisites installed:

Common prerequisites

Language-specific prerequisites

Configure agent testing environment

This section covers setting environment variables, authenticating your development environment, and preparing your Agent 365 powered agent for testing.

Setting up your agent testing environment follows a sequential workflow:

  1. Configure your environment - Create or update your environment configuration file

  2. LLM configuration - Get API keys and configure OpenAI or Azure OpenAI settings

  3. Configure authentication - Set up agentic authentication

  4. Environment variables reference - Configure required environment variables:

    1. Authentication variables
    2. MCP endpoint configuration
    3. Observability variables
    4. Agent application server configuration

After completing these steps, you're ready to start testing your agent in Agents Playground.

Step 1: Configure your environment

Set up your configuration file:

cp .env.template .env

Note

Refer to the Microsoft Agent 365 SDK samples to find configuration templates showing required fields.

Step 2: LLM configuration

Configure OpenAI or Azure OpenAI settings for local testing. Add your API keys and service endpoints obtained from the prerequisites to your configuration file along with any model parameters.

Add to your .env file:

# Replace with your actual OpenAI API key
OPENAI_API_KEY=

# Azure OpenAI Configuration
AZURE_OPENAI_API_KEY=
AZURE_OPENAI_ENDPOINT=
AZURE_OPENAI_DEPLOYMENT=
AZURE_OPENAI_API_VERSION=

Python LLM environment variables

Variable Description Required Example
OPENAI_API_KEY API key for OpenAI service For OpenAI sk-proj-...
AZURE_OPENAI_API_KEY API key for Azure OpenAI service For Azure OpenAI a1b2c3d4e5f6...
AZURE_OPENAI_ENDPOINT Azure OpenAI service endpoint URL For Azure OpenAI https://your-resource.openai.azure.com/
AZURE_OPENAI_DEPLOYMENT Deployment name in Azure OpenAI For Azure OpenAI gpt-4
AZURE_OPENAI_API_VERSION API version for Azure OpenAI For Azure OpenAI 2024-02-15-preview

Step 3: Configure authentication for your agent

Choose one of the following authentication methods for your agent:

Agentic authentication

Use the A365 CLI a365 config display command to retrieve your agent blueprint credentials.

a365 config display -g

This command displays your agent blueprint configuration. Copy the following values:

Value Description
agentBlueprintId Your agent's client ID
agentBlueprintClientSecret Your agent's client secret
tenantId Your Microsoft Entra tenant ID

Use these values to configure agentic authentication in your agent:

Add the following settings to your .env file, replacing the placeholder values with your actual credentials:

USE_AGENTIC_AUTH=true
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__CLIENTID=<agentBlueprintId>
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__CLIENTSECRET=<agentBlueprintClientSecret>
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__TENANTID=<your-tenant-id>
Variable Description Required Example
USE_AGENTIC_AUTH Enable agentic authentication mode Yes true
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__CLIENTID Agent blueprint client ID from a365 config display -g Yes 12345678-1234-1234-1234-123456789abc
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__CLIENTSECRET Agent blueprint client secret from a365 config display -g Yes abc~123...
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__TENANTID Microsoft Entra tenant ID from a365 config display -g Yes adfa4542-3e1e-46f5-9c70-3df0b15b3f6c

Bearer token authentication

For early development and testing scenarios when agent users aren't available, you can use bearer token authentication to test your agent.

First, use a365 develop add-permissions to add the required MCP server permissions to your application:

a365 develop add-permissions

Then, use a365 develop get-token to retrieve and configure bearer tokens:

a365 develop get-token

The get-token command automatically:

  • Retrieves bearer tokens for all MCP servers configured in your ToolingManifest.json file
  • Updates your project configuration files with the BEARER_TOKEN environment variable

Note

Bearer tokens expire after approximately 1 hour. Use a365 develop get-token to refresh expired tokens.

Step 4: Environment variables reference

Complete your environment setup by configuring the following required environment variables:

Authentication variables

Configure the authentication handler settings required for agentic authentication to work properly.

Add to your .env file:

# Agentic Authentication Settings
AGENTAPPLICATION__USERAUTHORIZATION__HANDLERS__AGENTIC__SETTINGS__TYPE=AgenticUserAuthorization
AGENTAPPLICATION__USERAUTHORIZATION__HANDLERS__AGENTIC__SETTINGS__SCOPES=https://graph.microsoft.com/.default
AGENTAPPLICATION__USERAUTHORIZATION__HANDLERS__AGENTIC__SETTINGS__ALTERNATEBLUEPRINTCONNECTIONNAME=service_connection

# Connection Mapping
CONNECTIONSMAP_0_SERVICEURL=*
CONNECTIONSMAP_0_CONNECTION=SERVICE_CONNECTION
Variable Description Required
AGENTAPPLICATION__USERAUTHORIZATION__HANDLERS__AGENTIC__SETTINGS__TYPE Authentication handler type Yes
AGENTAPPLICATION__USERAUTHORIZATION__HANDLERS__AGENTIC__SETTINGS__SCOPES Authentication scopes for Microsoft Graph Yes
AGENTAPPLICATION__USERAUTHORIZATION__HANDLERS__AGENTIC__SETTINGS__ALTERNATEBLUEPRINTCONNECTIONNAME Alternate blueprint connection name Yes
CONNECTIONSMAP_0_SERVICEURL Service URL pattern for connection mapping Yes
CONNECTIONSMAP_0_CONNECTION Connection name for mapping Yes

MCP endpoint configuration

The MCP (Model Context Protocol) endpoint configuration is required to specify which Agent 365 platform endpoint your agent should connect to. When you generate the tooling manifest that defines the tooling servers for your agent, you must specify the MCP platform endpoint. This endpoint determines which environment (preprod, test, or production) the MCP tool servers connect to for Microsoft 365 integration capabilities.

Add to your .env file:

# MCP Server Configuration
MCP_PLATFORM_ENDPOINT=<MCP endpoint>
Variable Description Required Default Example
MCP_PLATFORM_ENDPOINT MCP platform endpoint URL (preprod, test, or prod) No Production endpoint

Important: If MCP_PLATFORM_ENDPOINT isn't specified, it defaults to the production endpoint.

Observability variables

Configure these required variables to enable logging and distributed tracing for your agent. Learn more about about observability features and best practices

Note

The observability configuration is the same across all languages.

Variable Description Default Example
ENABLE_A365_OBSERVABILITY Enable/disable observability false true
ENABLE_A365_OBSERVABILITY_EXPORTER Export traces to observability service false true
OBSERVABILITY_SERVICE_NAME Service name for tracing Agent name my-agent-service
OBSERVABILITY_SERVICE_NAMESPACE Service namespace agent365-samples my-company-agents

Agent application server configuration

Configure the port where your agent application server runs. This setting is optional and applies to Python and JavaScript agents.

Add to your .env file:

# Server Configuration
PORT=3978
Variable Description Required Default Example
PORT Port number where the agent server runs No 3978 3978

Install dependencies and start the agent application server

Once your environment is configured, you need to install the required dependencies and start your agent application server locally for testing.

Install dependencies

uv pip install -e .

This command reads the package dependencies defined in pyproject.toml and installs them from PyPI. When creating an agent application from scratch, you need to create a pyproject.toml file to define your dependencies. Sample agents from the samples repository already have these packages defined. You can add or update them as needed.

Start the agent application server

python <main.py>

Replace <main.py> with the name of your main Python file that contains the entry point for your agent application (for example, start_with_generic_host.py, app.py, or main.py).

Or using uv:

uv run python <main.py>

Your agent server should now be running and ready to receive requests from Agents Playground or Microsoft 365 applications.

Test agent in Agents Playground

Agents Playground is a local testing tool that simulates the Microsoft 365 environment without requiring a full tenant setup. It's the fastest way to validate your agent's logic and tool invocations. For more information, see Test with Agents Playground.

Note

When you use agentic authentication, direct messaging in Agents Playground isn't currently supported. You must test through custom activities instead. See Test with notification activities for details.

Open a new terminal (PowerShell on Windows) and start Agents Playground:

agentsplayground

This command opens a web browser with the Agents Playground interface. The tool displays a chat interface where you can send messages to your agent.

Basic test

Start by verifying your agent is properly configured. Send a message to the agent:

What can you do?

The agent replies with the instructions it's configured with, based on your agent's system prompt and capabilities. This reply confirms that:

  • Your agent is running correctly
  • The agent can process messages and respond
  • Communication between Agents Playground and your agent is working

Test tool invocations

After configuring your MCP tool servers in toolingManifest.json (see Tooling for setup instructions), test tool invocations with examples like these:

First, verify which tools are available:

List all tools I have access to

Then test specific tool invocations:

Mail tools

Send email to your-email@example.com with subject "Test" and message "Hello from my agent"

Expected response: The agent sends an email using the Mail MCP server and confirms the message was sent.

Calendar tools

List my calendar events for today

Expected response: The agent retrieves and displays your calendar events for the current day.

SharePoint tools

List all SharePoint sites I have access to

Expected response: The agent queries SharePoint and return a list of sites you have access to.

You can view the tool invocations in:

  • The chat window - see the agent's response and any tool calls
  • The Log panel - see detailed activity information including tool parameters and responses

Test with notification activities

During local development, you can test notification scenarios by simulating custom activities in Agents Playground. This simulation allows you to verify your agent's notification handling before deploying it to production.

Before testing notification activities, ensure you have:

Notifications require both proper tool configuration and notification setup to work correctly. You can test scenarios such as email notifications or Word comments using the custom activity feature.

To send custom activities:

  1. Start your agent and Agents Playground
  2. In Agents Playground, navigate to Mock an Activity > Custom activity
  3. Copy the conversationId from the activity (the conversation ID changes each time Agents Playground is restarted)
  4. Paste your custom activity JSON and update the personal-chat-id field with the conversation ID you copied. Refer to the Email notification example
  5. Select Send activity
  6. View the result in both the chat conversation and the log panel

Email notification

This configuration simulates an email sent to the agent. Replace placeholder values with your actual agent details:

{
  "type": "message",
  "id": "aaaaaaaa-0000-1111-2222-bbbbbbbbbbbb",
  "timestamp": "2025-09-24T17:40:19+00:00",
  "serviceUrl": "http://localhost:56150/_connector",
  "channelId": "agents",
  "name": "emailNotification",
  "from": {
    "id": "manager@contoso.com",
    "name": "Agent Manager",
    "role": "user"
  },
  "recipient": {
    "id": "agent@contoso.com",
    "name": "Agent",
    "agenticUserId": "<your-agentic-user-id>",
    "agenticAppId": "<your-agent-app-id>",
    "tenantId": "<your-tenant-id>"
  },
  "conversation": {
    "conversationType": "personal",
    "tenantId": "aaaabbbb-0000-cccc-1111-dddd2222eeee",
    "id": "personal-chat-id"
  },
  "membersAdded": [],
  "membersRemoved": [],
  "reactionsAdded": [],
  "reactionsRemoved": [],
  "locale": "en-US",
  "attachments": [],
  "entities": [
    {
      "id": "email",
      "type": "productInfo"
    },
    {
      "type": "clientInfo",
      "locale": "en-US",
      "timezone": null
    },
    {
      "type": "emailNotification",
      "id": "bbbbbbbb-1111-2222-3333-cccccccccccc",
      "conversationId": "personal-chat-id",
      "htmlBody": "<body dir=\"ltr\">\n<div class=\"elementToProof\" style=\"font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);\">\nYour email message content here</div>\n\n\n</body>"
    }
  ],
  "channelData": {
    "tenant": {
      "id": "aaaabbbb-0000-cccc-1111-dddd2222eeee"
    }
  },
  "listenFor": [],
  "textHighlights": []
}

View observability logs

To view observability logs during local development, instrument your agent with observability code (see Observability for code examples) and configure the environment variables as described in Observability variables. Once configured, real-time traces appear in the console showing:

  • Agent invocation traces
  • Tool execution details
  • LLM inference calls
  • Input and output messages
  • Token usage
  • Response times
  • Error information

These logs help you debug issues, understand agent behavior, and optimize performance.

Troubleshooting

This section provides solutions to common issues you might encounter when testing your agent locally.

Connection and environment issues

These issues relate to network connectivity, port conflicts, and environment setup problems that can prevent your agent from communicating properly.

Agents Playground connection issues

Symptom: Agents Playground can't connect to your agent

Solutions:

  • Verify your agent server is running
  • Check that the port numbers match between your agent and Agents Playground
  • Ensure there are no firewall rules blocking local connections
  • Try restarting both the agent and Agents Playground

Outdated Agents Playground version

Symptom: Unexpected errors or missing features in Agents Playground

Solution: Uninstall and reinstall Agents Playground:

winget uninstall agentsplayground
winget install agentsplayground

Port conflicts

Symptom: Error indicating port is already in use

Solution:

  • Stop any other instances of your agent
  • Change the port in your configuration
  • Kill any processes using the port:
# Windows PowerShell
Get-Process -Id (Get-NetTCPConnection -LocalPort <port>).OwningProcess | Stop-Process

Can't add DeveloperMCPServer

Symptom: Error when trying to add DeveloperMCPServer in VS Code

Solution: Close and reopen Visual Studio Code, then try adding the server again.

Authentication issues

These issues occur when your agent can't authenticate properly with Microsoft 365 services or when credentials are expired or misconfigured.

Bearer token expired

Symptom: Authentication errors or 401 Unauthorized responses

Solution: Bearer tokens expire after approximately 1 hour. Acquire a new token and update your configuration.

Agentic authentication errors in Python

Symptom: Error acquiring agentic instance token

Solution: Verify the ALT_BLUEPRINT_NAME setting in your .env:

# Change from:
AGENTAPPLICATION__USERAUTHORIZATION__HANDLERS__AGENTIC__SETTINGS__ALT_BLUEPRINT_NAME=ServiceConnection

# To:
AGENTAPPLICATION__USERAUTHORIZATION__HANDLERS__AGENTIC__SETTINGS__ALT_BLUEPRINT_NAME=SERVICE_CONNECTION

Tool and notification issues

These issues involve problems with tool invocations, MCP server interactions, and notification delivery.

Email not received

Symptom: Agent indicates email was sent, but you don't receive it

Solutions:

  • Check your Junk/Spam folder
  • Email delivery can be delayed by a few minutes - wait up to 5 minutes
  • Verify the recipient email address is correct
  • Check agent logs for any errors during email sending

Word comment responses not working

Known issue: The notification service currently can't respond directly to Word comments. This functionality is being developed.

Messages not reaching agent

Symptom: Messages sent to the agent in Teams are not received by your agent application

Possible causes:

  • Agent blueprint not configured in Developer Portal
  • Azure Web App issues (deployment failures, app not running, configuration errors)
  • Agent instance not created properly in Teams

Solutions:

  1. Verify Developer Portal configuration:

    Ensure you've completed the agent blueprint configuration in Developer Portal. See Configure agent blueprint in Developer Portal for detailed steps.

  2. Check Azure Web App health:

    If your agent is deployed to Azure, verify the Web App is running properly:

    1. Navigate to Azure Portal
    2. Go to your Web App resource
    3. Check Overview > Status (should show "Running")
    4. Check Log stream under Monitoring for runtime errors
    5. Review Deployment Center logs to verify deployment succeeded
    6. Verify Configuration > Application settings contain all required environment variables

  3. Verify agent instance creation:

    Ensure you created the agent instance correctly in Microsoft Teams:

    1. Open Microsoft Teams
    2. Go to Apps and search for your agent
    3. Verify the agent appears in search results
    4. If not found, verify it's published in Microsoft 365 Admin Center - Agents
    5. Create a new instance by selecting Add on your agent
    6. For detailed instructions, see Onboard agents

Getting help

If you encounter issues not covered in this troubleshooting section, explore these resources:

Microsoft Agent 365 SDK repositories

More support

Next steps

After you successfully test your agent locally, you're ready to deploy it to Azure and publish it to Microsoft 365:

  • Deploy and publish agents: Learn how to deploy your agent to Azure Web App and publish it to Microsoft Admin Center, making it available for your organization to discover and create agent instances in Microsoft 365.
  • Last updated on