Responsible AI FAQ for Copilot in Customer Service

Applies to: Dynamics 365 Contact Center—embedded, Dynamics 365 Contact Center—standalone, and Dynamics 365 Customer Service

This FAQ article helps answer the questions around the responsible use of AI in copilot features in Customer Service.

What is Copilot in Dynamics 365 Customer Service?

Copilot is an AI-powered tool that transforms the agent experience in Dynamics 365 Customer Service. It provides real-time AI powered assistance that will help agents resolve issues faster, handle cases more efficiently, and automate time-consuming tasks. Then agents can focus on delivering high-quality service to their customers.

What are the systems capabilities?

Copilot provides the following main features:

  • Ask a question: Is the first tab that agents see when they activate the Copilot help pane. It's a conversational interface with Copilot, which helps provide contextual responses to the agents’ questions. Copilot’s responses are based on both internal and external knowledge sources provided by your organization during setup.

  • Write an email: Is the second tab on the Copilot help pane helps agents quickly create email responses based on the context of the case, reducing the time users need to spend creating emails.

  • Draft a chat response: Enables agents to create a response in a single click to the ongoing digital messaging conversation from knowledge sources configured by your organization.

  • Summarize a case: Copilot provides agents with a summary of a case right on the case form, so they can quickly catch up on the important details of a case.

  • Summarize a conversation: Copilot provides agents with a summary of a conversation at key points throughout the customer journey such as virtual agent handoffs, transfers and on demand.

  • Generate knowledge draft from case (preview): Copilot generates a knowledge article draft as a proposal that's based on information from the case. Agents can review and refine the draft by giving revision instructions to Copilot and then save it.

  • Summarize a custom record: Copilot provides agents with a summary of a record based on fields that the administrator configures for the custom table so that agents can quickly catch up on the important details of a support record.

What is the system’s intended use?

Copilot in Customer Service is intended to help customer service representatives work more efficiently and effectively. Customer service representatives can use Copilot’s knowledge-based responses to save time from searching knowledge articles and drafting responses. Copilot summaries are designed to support agents in quickly ramping up on cases and conversations. Content generated by Copilot in Customer Service isn't intended to be used without human review or supervision.

How is Copilot in Customer Service evaluated? What metrics are used to measure performance?

Copilot in Customer Service has been evaluated against real world scenarios with customers around the world through each phase of its design, development, and release. Using a combination of research and business impact studies, we’ve evaluated various quantitative and qualitative metrics about Copilot, including its accuracy, usefulness, and agent-trust. Learn more in Responsible AI Transparency Report.

What are the limitations of Copilot in Customer Service? How can users minimize the impact of Copilot limitations?

Copilot’s knowledge-based capabilities like ask a question, write an email, and draft a chat response, are dependent on high-quality and up-to-date knowledge articles for grounding. Without these knowledge articles, users are more likely to encounter Copilot responses that aren't factual.

To minimize the likelihood of seeing non-factual responses from Copilot, it’s important that Microsoft employs robust knowledge management practices to ensure the business knowledge that connects to Copilot is of high-quality and up-to-date.

What operational factors and settings allow for effective and responsible use of the system?

Always review results from Copilot

Copilot is built on large language model technology, which is probabilistic in nature. When presented with a piece of input text, the model calculates the probability of each word in that text given the words that came before it. The model then chooses the word that is most likely to follow. However, since the model is based on probabilities, it can't say with absolute certainty what the correct next word is. Instead, it gives us its best guess based on the probability distribution it has learned from the data it was trained on. Copilot uses an approach called grounding, which involves adding additional information to the input to contextualize the output to your organization. It uses semantic search to understand the input and retrieve relevant internal organizational documents and trusted public web search results, and guides the language model to respond based on that content. While this is helpful in ensuring Copilot responses adhere to organizational data, it's important to always review results produced by Copilot prior to using them.

Get the best out of Copilot

When you're interacting with Copilot, it's important to keep in mind that the structure of the questions can greatly affect the response that Copilot gives. To interact with Copilot effectively, it's crucial to ask clear and specific questions, provide context to help the AI better understand your intent, ask one question at a time, and avoid technical terms for clarity and accessibility.

Ask clear and specific questions

Clear intent is essential when asking questions, as it directly impacts the quality of the response. For instance, asking a broad question like “Why is the customer’s coffee machine not starting up?” is less likely to yield a useful response compared to a more specific question, such as “What steps can I take to determine why the customer’s coffee machine isn't starting up?”.

However, asking an even more detailed question like “What steps can I take to determine why a Contoso 900 coffee machine with a 5-bar pressure rating isn't starting up?” narrows down the scope of the problem and provides more context, leading to more accurate and targeted responses.

Add Context

Adding context helps the conversational AI system better understand the user's intent and provide more accurate and relevant responses. Without context, the system might misunderstand the user's question or provide generic or irrelevant responses.

For example, "Why is the coffee machine not starting up?" will result in a generic response when compared to a question with more context like, "Recently, the customer initiated descaling mode on their coffee machine and completed descaling successfully. They even received three flashes from the power light at the end to confirm that descaling was complete. Why are they unable to start the coffee machine anymore?"

Adding context in this manner is important because it helps Copilot better understand the user's intent and provide more accurate and relevant responses.

Avoid technical terms if possible

We recommend that you avoid using extremely technical terms and resource names when interacting with Copilot because the system may not always understand it accurately or appropriately. The use of simpler, natural language helps ensure that the system can understand the user's intent correctly and provide clear, useful responses. For example –

"The customer can't SSH into the VM after having changed the firewall config."

Instead, you can rephrase as –

“The customer changed the firewall rules on their virtual machine. However, they can no longer connect using Secure Shell (SSH). Can you help?”

By following the suggestions, agents can enhance their interactions with Copilot and increase the likelihood of receiving accurate and confident responses from it.

Summarizing or expanding a response

Sometimes the response from Copilot can be longer than expected. This could be the case when the agent is in a live chat conversation with a customer and needs to send concise responses when compared with sending a response over email. In such cases, asking Copilot to “summarize the response” will result in a concise answer to the question. Similarly, if there's a need for more detail, asking Copilot to “Provide more details” will result in a more detailed answer to your question. If the response is truncated, typing “continue” will display the remaining part of the response.

How can I influence the responses generated by copilot? Can I fine tune the underlying LLM?

It's not possible to customize the large language model (LLM) directly. Copilot responses can be influenced by updating the source documentation. All the feedback content from Copilot responses is stored. Reports can be created using this data to determine the data sources that need to be updated. It’s a good idea to have processes in place to periodically review the feedback data and ensure knowledge articles are providing the best and most up-to-date information to Copilot.

What's the data security model for Copilot?

Copilot enforces the role-based access (RBAC) controls defined and adheres to all the existing security constructs. Therefore, agents cannot view data that they do not have access to. Additionally, only data sources that the agent has access to are used for copilot response generation.

How does Copilot determine whether content is offensive or harmful?

Copilot determines whether content is harmful through a severity rating system based on distinct categories of objectionable content. You can learn more at Harm categories in Azure AI Content Safety.

Where does data processing and retrieval occur to generate copilot responses?

Copilot is not calling the public OpenAI service that powers ChatGPT. Copilot in Customer Service uses the Microsoft Azure OpenAI Service in a Microsoft managed tenant. All data processing and retrieval occurs within Microsoft managed tenants. Additionally, customer’s data is not shared and is not fed back into public models.

What are the language limitations for summaries that Copilot generates from cases and conversations?

Many languages are supported in Copilot-generated summaries from cases and conversations. The quality of these summaries is expected to be the highest in English, while in the other languages, the quality is expected to improve over time. Learn about supported languages in Language support for Copilot features

Is the model tested and monitored on an ongoing basis? If so, how frequently? What testing is performed?

The model is tested for quality and harmful content, every time there's a change to the model interaction or version. Learn more in the Responsible AI Transparency Report.

How frequently is the model monitored to detect performance degradation?

The generative AI GPT Model is hosted and managed by Azure OpenAI. The use of the model in customer service scenarios are bound by responsible AI practices and Deployment Safety Board checks. Any changes to model versions or underlying prompts are validated for quality and harmful content. Learn more in the Responsible AI Transparency Report.

Does the product or service employ more than one model or a system of interdependent models?

Different features in the system could be using different versions of Azure OpenAI Service models. Learn more in Azure OpenAI Service models.

Does Copilot use a non-Microsoft model product or service, and is documentation for this model available?

Copilot is built using Azure OpenAI, a fully managed AI service that integrates OpenAI with Microsoft-developed content filtering and abuse detection models. Learn more in Transparency note for Azure OpenAI.

Is there a set process to communicate any changes in models, upstream models or outputs that are used from other AI/ML or model solutions?

Any planned changes to the Copilot features are communicated via public documentation. However changes with respect to model version or prompts, are governed by the internal responsible AI processes. These changes are not communicated, as they are incremental and ongoing functional improvements.

Is user verbatim feedback available to Microsoft to improve the product?

No.

Does Microsoft have policies and procedures that define and differentiate the various human roles and responsibilities when they interact with or monitor AI systems?

Yes. In the responsible AI process, all stakeholders and users involved are considered, and their use or unintended use of the system is discussed. Based on the identified scenarios, required mitigations are brought in within the product or via documentation. Learn more in Responsible AI Transparency Report.

Does Microsoft identify and document approaches to engage, capture, and incorporate input from other end users and key stakeholders to assist with the continuous monitoring for potential impacts and emerging risks?

Yes. Learn more in Responsible AI Transparency Report.

Does Microsoft document, practice, and measure incident response plans for AI system incidents including measuring response and down times?

Yes. The responsible AI process requires that team have a incident response plan for AI issues, similar to what's done for functional issues. The feature teams continuously monitor the performance and reliability of the system. If any issues are detected, the teams react to first investigate and then mitigate. Learn more in Responsible AI Transparency Report.

Does Microsoft establish procedures to share information about error incidences and negative impacts with relevant stakeholders, operators, practitioners, users and affected parties?

Yes. In case of high severity issues, feature teams must communicate the outage with the affected customers.

Does Microsoft measure and monitor system performance in real time to enable rapid response when an AI system incident is detected?

Yes. The feature teams continuously monitor the performance and reliability of the system. If any issues are detected, the teams react to first investigate and then mitigate.

Does Microsoft test the quality of systems explanations with end users and other stakeholders?

Yes. Learn more in Responsible AI Transparency Report.

Does Microsoft have established policies and procedures for monitoring and addressing model system performance and trustworthiness, including bias and security problems across the lifecycle of the model, when evaluating AI systems for negative risks and benefits?

Yes. Learn more in Responsible AI Transparency Report.

Does Microsoft conduct fairness assessments to manage computational and statistical forms of bias?

Yes. Learn more in Responsible AI Transparency Report.

Does Microsoft monitor system outputs for performance or bias issues?

Yes. Moderation filters are applied in multiple layers, including on output to make sure that there's no harmful content in the response. Learn more in Responsible AI Transparency Report.

What is the level of resiliency in the model's operation? For example, is there a disaster recovery and contingency plan for instances when the model is unavailable?

Similar to all Azure services, backup and recovery is supported via multiple data centers for high availability.

Is the model dependent upon, embedded within third-party tools or solutions that makes it difficult to migrate the model to a different environment (including variables such as hosting provider, hardware, software systems) that would impede the model's explainability?

No.

Is there an established model governance policy?

Yes, there's an established governance policy supported by Azure OpenAI. Learn more in Responsible AI Transparency Report.

Are there established and documented protocols (authorization, duration, type) and access controls for training or production data sets containing PII in accordance with privacy and data governance policies?

Currently, there's no model training, hence no requirement surrounding the data set. However when a customer service representative engages with Copilot, depending on the feature, context data (case or chat) is used to generate a response.

Are PII disclosures and inference of sensitive or legally protected attributes monitored?

Yes, privacy review is done for every feature.

Yes. Legal review is done for every feature to help with regulatory requirements and other legal matters.

Use copilot features
Use Copilot to generate knowledge drafts from cases
Region availability of Copilot
FAQ for Copilot data security and privacy in Microsoft Power Platform