Share via

Automate web and desktop apps with computer use (preview)

[This article is prerelease documentation and is subject to change.]

Computer use is a tool in Copilot Studio that lets your agent interact with and automate tasks on a Windows computer. It works with websites and desktop apps by selecting buttons, choosing menus, and entering text into fields on the screen. Describe in natural language what you want computer use to do, and it performs the task on a computer you set up by using a virtual mouse and keyboard. With computer use, agents can complete tasks even when there's no API to connect directly to the system. If a person can use an app or website, computer use can too. You can use computer use for tasks like automated data entry, invoice processing, and data extraction.

Important

This article contains Microsoft Copilot Studio preview documentation and is subject to change.

Preview features aren't meant for production use and may have restricted functionality. These features are available before an official release so that you can get early access and provide feedback.

If you're building a production-ready agent, see Microsoft Copilot Studio Overview.

Computer use is powered by Computer-Using Agents (CUA), an AI model that combines vision capabilities with advanced reasoning to interact with graphical user interfaces (GUIs). Because it's AI-powered, it adapts to interface changes. For example, when buttons or screens change, the tool continues working without breaking your flow. It's simple to use. Just describe what you want in natural language. You don't need to write code.

Watch this video to learn how the computer use tool lets an agent interact with a web application.

Requirements

  • The feature is available for environments where the region is set to United States.

  • Make sure your agent has the generative orchestrator enabled to use computer use.

Add computer use to your agent

On a new agent or an existing agent in Copilot Studio, add computer use as a tool by following these steps:

  1. Go to the Tools section in your agent and select Add tool.

  2. In the Add tool dialog, select New tool.

  3. Select Computer use.

  4. Provide the instructions that describe the task the tool should perform on the computer. You see some instruction templates to get started with. To learn more about how to best write instructions for computer use, check best practices for instructions for computer use.

  5. Choose the machine where computer use runs. You can either use the hosted browser, a ready-to-use machine for automating tasks on public-facing websites, or select a Windows machine that you can configure to use for computer use. Learn more in Configure where computer use runs.

  6. Select Add and configure.

  7. On the configuration page, configure these three fields:

    • Name: Enter the display name for the computer use tool. This name helps you differentiate it from other tools you add to your agent.

    • Description: Enter a short description of what this tool does and when to use it. This text lets your agent know when to use this tool.

    • Instructions: List the steps the tool should perform, including URLs and application names. For tips, see Best practices for instructions for computer use.

  8. Review these other fields and settings on the configuration page that might be relevant:

    • Inputs: Use Inputs to define dynamic values that change each time computer use runs. For example, if you want to fill out a form with a different value on every run, create an input for that field. At execution time, computer use combines your instructions with the input values to complete the task.

    • Machine: Select the target machine that the agent uses to run computer use. Refer to Configure where computer use runs for more details on choosing the right machine type based on your requirements.

      • Use Refresh to update the machine list.

      • Choose Manage machines to open the machine management page in Power Automate portal.

      • Select See machine details to open the machine details page in Power Automate portal.

      • Connection: Connection used for this tool. Update or create a new connection to change the credentials used.

      • Credentials to use: Specify how computer use authenticates during execution:

        • Maker-provided credentials (default): This option uses the maker’s credentials and is suitable for autonomous agents.

          Warning

          If you share an agent with this setting, anyone using it can act with the original author’s access on the configured machine.

        • End user credentials: This option uses the credentials of the person interacting with the agent. Each user must have access credentials to the machine.

    • Human supervision: Specify who should be contacted via email (Outlook) if the computer-use agent detects potentially harmful instructions that could alter model behavior. Each agent run and its activity is tied to the user who initiated it. If you choose a reviewer other than the person running the computer-use agent, they likely don't see the activity because they didn’t initiate the run. Therefore, they can't properly verify or act on the request. Also, ensure the recipient is authorized and has the necessary context to handle such requests. The response time limit defines how long the request remains active. After this period, the request expires, and the computer-use run stops if no response is received. See Human supervision for more information.

    • Stored credentials: Define the credentials that computer use uses to sign in to websites and applications. During execution, if a sign-in prompt appears, computer use securely uses any credentials you defined in this section for that site or application. Password values of these credentials are stored in an Azure Key Vault that you provide. Learn how to create an Azure Key Vault in Create a key vault using the Azure portal.

      To configure credentials in the tool, first enter the subscription ID, resource group name, and Key Vault name. All this information is available from the Overview page of your Key Vault.

      To use Azure Key Vault secrets with Power Platform:

      • The Azure subscription that has the vault must have the "PowerPlatform resource provider" registered.
      • The user who creates the environment variable must have appropriate permissions to the Azure Key Vault resource.

      If you haven't already, follow the steps in Configure Azure Key Vault. Then provide the following details:

      • Name: the display name for the credential.

      • Username: the username used to sign in to the target website or application.

      • Login domain: the domain where you enter the credentials (for example, login.microsoft.com). Make sure to verify this domain, as it might differ from the main site URL.

      • Azure secret name: the name of the secret in the Key Vault that stores the password for the website or application.

        Note

        Password fields are supported on all websites and most Windows applications (WinForms, WPF, UWP, WinUI, Win32), which covers most customer scenarios. Some app types, such as Electron, Java, Unity, games, command-line interfaces, Citrix, or other virtualized environments, might not be supported.

    • Access control: By default, computer use can operate on any website or application. If you want to restrict this access, enable access control to define the specific URLs and desktop applications that computer use should be limited to. You can configure both websites and applications:

      • Websites: Enter the main website address (for example, example.com). All pages on that website are included automatically. You can also use wildcards (*) for subdomains.
        • Examples: www.contoso.com, *.contoso.com, contoso.com
      • Desktop applications: Enter the application product name or process name. To find it, press Ctrl+Shift+Esc to open Task Manager. Then check the Processes tab.
        • Examples: Microsoft Edge, msedge, Notepad.

      Note

      Access control only prevents the model from taking actions on websites or applications that aren't in the allowlist. It doesn't stop the model from opening them. For example, if only microsoft.com and Microsoft Edge are in the allowlist, the model can still use the Edge search bar to open Bing. However, once Bing is opened, any attempt to interact with it fails because it isn't in the allowlist.

  9. Select Save.

Test computer use

Testing computer use is a key step in the authoring journey. After you enter a name, description, and instructions, and save the tool, select Test to start the test experience.

After a short loading period, the test experience appears:

  • The left panel shows your instructions and a step-by-step log of the tool’s reasoning and actions.
  • The right panel shows a preview of the actions on the machine you set up for computer use.

When the task finishes, you see a Test completed message. While the test is in progress, you can select Stop testing to immediately stop all actions on the machine.

If the result isn't what you expect, go to the configuration page and refine your instructions. Add more detail to improve accuracy. For guidance, see best practices for writing effective instructions.

Publish an agent with computer use

Set up computer use, then publish your agent. How your agent runs depends on the scenario. It can be autonomous or conversational:

  • Autonomous agents run automatically and perform tasks in the background.

  • Conversational agents let users interact through channels like Microsoft Teams.

Computer use works best for autonomous agents, performing tasks in the background without user interaction.

You can also apply computer use in conversational experiences, but keep these considerations in mind:

  1. If you select User authentication as the authentication setting, each user interacting with the agent in a conversation needs valid credentials for the machine used by computer use.
  2. When the tool runs, it shares reasoning messages and screenshots of the machine's activity in the chat.

Best practices

To stay productive and safe in today's digital environments, follow best practices for computer use, especially when writing clear instructions and securing machines.

Best practices for securing machines

When setting up machines for computer use that lets AI perform tasks by using natural language, consider these security recommendations:

Security recommendation Additional information
Use dedicated machines for computer use Assign specific, isolated machines exclusively for tasks involving computer use. This approach reduces the risk of cross-contamination from unrelated software, malware, or unauthorized access. It lets you control configurations, updates, and monitoring more effectively.
Limit permissions to the user account that you're using for computer use Set up the user account for computer use to follow the principle of least privilege—grant only the permissions needed to run the required tools.
Limit web access to an allowlist of specific trusted websites only Allow web access only to a predefined list of vetted and trusted domains. For example, you can configure Microsoft Edge policy settings with Microsoft Intune that target machines used for computer use.
Limit specific desktop apps to be available Only install and allow execution of applications that are essential for the intended AI workflows. Remove or disable access to unnecessary software. For example, you can configure Application Control to limit what applications are allowed to run on the machine.

Best practices for instructions for computer use

The instructions you write determine how well computer use works. Specific, detailed instructions help computer use complete tasks accurately. Think of it as explaining a task to a colleague. Clear, step-by-step guidance helps ensure success.

Tips for writing effective instructions:

  • Be specific about websites and applications. Always include the full URL of any website and the exact name of any application the tool should use. Example: "Open https://www.microsoft.com and go to 'Company news'."

  • Clearly state relevant actions. If you want something done, say it explicitly - especially actions like submitting a form or sending an email. Example: "Once you fill in the form, select Submit. No need to ask for permission."

  • Break down complex interactions. For areas where the UI might be more complex to navigate, explain each step in detail. Example: "Select the More icon in the top right corner. A dropdown appears. Once it opens, select the last item in the list."

  • Use step-by-step formatting for longer tasks. Long instructions are easier to follow when you format them as a list.

Sample instructions

Explore these sample instructions to try out computer use or use them as a reference for writing your own.

Scenario Name Description Instructions
Invoice processing Transfer and submit invoice details Transfer invoice data from a PDF and submit it to another form. 1. Go to https://computerusedemos.blob.core.windows.net/web/Contoso/invoice-manager.html, set the Date filter to Last 24 hours, and open the invoice PDF.
2. In a new tab, open https://computerusedemos.blob.core.windows.net/web/Contoso/index.html and fill out the form with the data from that PDF. Submit the invoice form, no confirmation needed.
Data entry Submit inventory items Add products to the inventory system. 1. Go to https://computerusedemos.blob.core.windows.net/web/Adventure/index.html.
2. Submit a new entry for each of the following items:
Rear Derailleur, RD-4821, 50, 42.75, Tailspin Toys
Pedal Set, PD-1738, 80, 19.99, Northwind Traders
Brake Lever, BL-2975, 35, 14.50, Trey Research
Chainring Bolt Set, CB-6640, 100, 5.25, VanArsdel, Ltd.
Bottom Bracket, BB-9320, 60, 24.90, Tailwind Traders
Data extraction Look up portfolio manager and value Get the manager name and value for a portfolio. 1. Go to https://computerusedemos.blob.core.windows.net/web/Portfolio/index.html.
2. Find the row for Fourth Coffee and record the Portfolio Manager name and the current Portfolio Value exactly as shown.
3. Return those two values as the final output.

Best practices for extracting data

You can use computer use to extract data from websites or applications and then pass that data to your agent or to other tools in the agent. To do this, just describe what information you want to extract directly in the instructions for computer use.

If you want to use the extracted data with another tool (for example, sending it by email), specify that need in the agent instructions and make sure both tools are added to the agent.

Example: "Use computer use to extract the client information for the financial portfolio dashboard and email the extracted data." In this case, your agent must include both the computer use tool and the email tool.

Extract values as text

You can instruct computer use to extract specific values and return them as plain text. For example, the following prompt extracts the portfolio manager and portfolio value for the client Fourth Coffee:

1. Go to https://computerusedemos.blob.core.windows.net/web/Portfolio/index.html.
2. Find the row for Fourth Coffee and record the Portfolio Manager name and the current Portfolio Value exactly as shown.
3. Return those two values as the final output.

The agent returns only the two requested values as text.

Extract values as JSON

You can also ask computer use to return extracted information in JSON format. This format is useful when you want to pass structured data to another tool. The following example extracts all portfolio rows related to Contoso and returns them as properly formatted JSON:

Navigate to https://computerusedemos.blob.core.windows.net/web/Portfolio/index.html, retrieve the portfolio details for all Contoso entities, and return the results as a valid JSON object.

Structure the output so that:
•	Each top-level key is the client name
•	Each value contains the client's portfolio ID, portfolio value, portfolio manager, and last updated date (format: YYYY-MM-DD)
•	Return only the JSON, with no additional text.

This approach ensures the output is structured.

Licensing

While computer use is in preview, it's billed using the Agent action feature with a billing rate of five Copilot Credits. For more information, see Microsoft Copilot Studio billing rate and management.

Each computer use run can perform several actions, and each action costs five Copilot Credits.

For example, if you set up computer use to fill out a web-based timesheet form, it performs these actions each time it's triggered:

  1. Launch browser.

  2. Navigate to the timesheet web portal.

  3. Select Create new timesheet.

  4. Fill the Start Time form field.

  5. Fill the End Time form field.

  6. Fill the Project Code form field.

  7. Select the Submit button.

In this example, computer use executes seven actions, consuming a total of 35 messages.

Share your feedback

Do you have feedback about computer use? Let us know at computeruse-feedback@microsoft.com.

FAQ for the computer use tool

  • Last updated on