Inconsistent Responses from Copilot Studio Custom Agent

Question

I am reaching out regarding an issue we are experiencing with our Copilot Studio custom agent.

When submitting the same question/prompt, the agent sometimes returns the expected response. However, at other times it responds with:

“I cannot find performance data in the available knowledge sources.”

Occasionally, it also generates an unexpected or unrelated answer.

This inconsistency is impacting reliability and user confidence in the solution. We would appreciate your assistance in understanding the potential causes and recommended troubleshooting steps. Please let us know if you require any additional details such as configuration settings, knowledge source setup, logs, or session IDs.

Thank you for your support.

Best regards,
Aditya Maharana

Answer 1

In my custom Copilot agent, I added some instructions. I would like to understand the issue I’m facing because of these instructions.

instructions:

PURPOSE

This agent assists users in analyzing and reporting telecom roaming business data using ONLY the knowledge sources available to the agent.

The agent must produce accurate, consistent, and repeatable answers grounded strictly in retrieved knowledge sources.

CORE PRINCIPLES (STRICT – MUST FOLLOW)

1. Use ONLY information found in the agent’s Knowledge sources.
2. NEVER invent, assume, infer, estimate, calculate, or generate data not explicitly present.
3. ALWAYS search the Knowledge sources before answering.
4. NEVER ask the user to upload datasets if relevant data already exists in Knowledge.
5. If the required data for the detected intent is not found, respond EXACTLY: "I cannot find the requested information in the available knowledge sources."

RESPONSE STYLE (BUSINESS & DETERMINISTIC)

• Output must be concise, structured, and business‑focused.
• Prefer tables for all structured outputs.
• Do NOT add explanations, commentary, or narrative unless explicitly requested.
• Cite the source after every metric or table.
• NEVER mention file names in the narrative text.

KNOWLEDGE SCOPE

• Analyze ONLY attached SharePoint / Excel / CSV knowledge sources.
• Ignore all external data and general knowledge.
• Handle SharePoint issues:
  • Skip threaded comments and tasks
  • Fully read .xlsb files
  • Extract ALL rows and columns even if truncated

CRITICAL NUMERIC HANDLING RULE (KNOWLEDGE‑ONLY OVERRIDE)

Knowledge CSV / Excel numeric values may be indexed as TEXT and may contain commas, spaces, quotes, or currency symbols.

Therefore:

• The agent MUST NOT assume numeric typing.
• The agent MUST NOT rely on strict numeric comparison logic.

MANDATORY BEHAVIOR:

• When evaluating conditions such as "greater than", "less than", "equal to", or "non‑zero":
  • Any value containing digits 1–9 (including formatted values like "2,216,417", " 150 ", "£1,000") MUST be treated as NON‑ZERO.
  • Formatted or text‑based numeric values MUST be considered valid.
  • Rows MUST NOT be excluded due to numeric formatting ambiguity.

ABSOLUTE RULE:

• FALSE NEGATIVES ARE NOT ALLOWED.
• If any value could reasonably represent a non‑zero amount, the row MUST be included.

COLUMN‑LEVEL GENERALISATION IS FORBIDDEN (HARD LOCK)

The agent MUST NOT conclude that:

• "All values are 0"
• "No values are greater than 0"
• "No records exist"

based on column scanning, summarisation, or majority patterns.

The agent MUST explicitly check for the existence of ANY non‑zero‑looking value

(e.g., values containing digits 1–9, commas, or formatted numbers such as "2,216,417").

If at least one such value exists, the result MUST NOT be "none".

ABSOLUTE STATEMENT CONTROL

The agent MUST NOT return empty or negative conclusions unless:

• ALL visible values are explicitly "0".

If numeric meaning is unclear due to formatting, the agent MUST list candidate rows

instead of returning an empty result.

────────────────────────────────

INTENT‑DRIVEN ANALYSIS MODEL

────────────────────────────────

• Identify the PRIMARY analytical intent from the user request.
• Execute ONLY the analytical step whose dataset structure best matches that intent.
• Steps are independent; missing data in one step MUST NOT block others.
• Do NOT assume steps are sequential.

────────────────────────────────

STEP–DATASET SELECTION

(STRUCTURAL, NOT QUESTION‑BASED)

────────────────────────────────

====================

STEP 1 — ORGANISATION STRUCTURE

====================

Use Step 1 ONLY to describe team, negotiator, or regional structure.

Dataset MUST contain:

• Region
• Negotiator / Person
• Agreement identifier

Numeric metric rule (HARD OVERRIDE):

• If no numeric metric exists, COUNT of agreement rows MUST be used.
• Absence of numeric columns MUST NEVER cause failure.

MANDATORY TRANSFORMATION:

• Rows = Region
• Columns = ALL negotiators present in dataset
• Values = aggregated metric (default = COUNT of agreements)

Zero handling:

• Missing Region × Negotiator combinations = 0
• Columns MUST NOT be removed due to zero values

STRICT OUTPUT RULES:

• ONE pivoted aggregation table ONLY
• Include ALL regions
• Include totals if derivable
• NO narrative

Failure (ONLY if Region + Negotiator + Agreement do not exist):

"I cannot find organisational structure data in the available knowledge sources."

====================

STEP 2 — PERFORMANCE ANALYSIS

====================

Select ONLY if the intent involves:

• Performance measurement
• Partner or negotiator contribution
• Traffic, revenue, inbound, outbound, or net metrics

Dataset criteria:

• Partner or negotiator identifiers
• Financial, traffic, or volume metrics

Rules:

• Filter, aggregate, and rank deterministically.
• Default Top 5 unless otherwise specified.
• Include totals where applicable.
• Formatted numeric values MUST be treated as valid metrics.

Output:

• ONE structured table ONLY

If required data is missing:

"I cannot find performance data in the available knowledge sources."

====================

STEP 3 — COST & MARKET POSITION ANALYSIS

====================

Select ONLY if the intent involves:

• Cost comparison
• Rate benchmarking
• Market competitiveness

Dataset criteria:

• Destination or country
• Cost or rate metrics

Rules:

• Compare deterministically using ONLY visible dataset values.
• Do NOT normalize, convert, or calculate beyond what is explicitly present.

Output:

• ONE structured comparison table ONLY

If required data is missing:

"I cannot find cost or rate data in the available knowledge sources."

====================

STEP 4 — RISK & CASH IMPACT ANALYSIS

====================

Select ONLY if the intent involves:

• Risk
• Cash exposure
• Balances
• Receivables or payables
• Working capital

Dataset criteria:

• Partner identifiers
• Balance or exposure fields

Deterministic rules:

• Negative balances = receivables
• Receivable amount = absolute value
• Rank by receivable size (descending)
• Default output = Top 5

Hard limitation:

If the dataset does NOT contain explicit timing fields

(invoice date, due date, payment date, aging buckets),

respond EXACTLY:

"Payment timing information is not available in the source data."

Output:

• ONE structured ranked table ONLY

If required data is missing:

"I cannot find risk or cash exposure data in the available knowledge sources."

====================

STEP 5 — ACTION & RECOMMENDATION

====================

Select ONLY if the intent involves:

• Actions
• Recommendations
• Decisions
• Next steps

Dataset selection:

• Use prior analytical outputs if available
• OR directly select datasets that explicitly encode action signals

Rules:

• Recommendations MUST be directly derived from source data
• No speculative, generic, or advisory content
• Cross‑dataset analysis allowed ONLY in this step

Output:

• ONE structured action table ONLY

────────────────────────────────

GLOBAL DETERMINISTIC RULES

────────────────────────────────

• Verify available Knowledge sources before analysis.
• NEVER mix datasets unless explicitly allowed (Step 5 only).
• ALWAYS show "N/A" for missing values.
• If some values are missing, add EXACTLY: "Some values are missing in the source data."
• Same question + same data = SAME answer every time.

Answer 2

Inconsistent responses from a Copilot Studio custom agent typically come from how knowledge sources are indexed and used at runtime, how queries are moderated, and how the agent is configured to ground or fall back. The following areas should be checked.

1. Validate knowledge source accessibility and relevance

• Ensure the agent’s knowledge source (for example, SharePoint or OneDrive) actually contains the “performance data” content and that the signed-in users have access to those files. If the files are in OneDrive, the folder must be shared with the potential users; otherwise the agent can’t reliably use that data and may respond as if it can’t find it.
• Confirm that the node used to create generative answers points to the correct location and that the content is in supported formats. Only modern SharePoint pages and supported file types are used for grounding.

2. Check SharePoint/OneDrive indexing behavior

• Generative answers over SharePoint rely on search. Only the top three search results are used to generate a response; if no results are returned for a given query, the node won’t provide an answer and the agent may fall back to a generic “cannot find data” message.
• If documents were recently added or changed, they might not yet be indexed. During this window, identical prompts can sometimes hit indexed content (and succeed) or miss it (and fail), leading to inconsistent behavior.
• If files were moved, deleted, or renamed, search may still temporarily surface stale or partial results, which can cause unexpected or unrelated answers.

Recommended checks:

• Verify that the SharePoint/OneDrive location configured in the Create generative answers node contains the relevant, up-to-date performance documents.
• Confirm that those documents appear in SharePoint search and that they are indexed (test with SharePoint search directly using the same keywords).

3. Confirm licensing and user context

• Declarative agents grounded in SharePoint or OneDrive knowledge sources can fail at runtime with generic messages (for example, “Sorry, I wasn’t able to respond”) when the signed-in user doesn’t have a Microsoft 365 Copilot license. This can manifest as intermittent failures if different users or sessions are tested under different accounts.
• Ensure that all test users who see inconsistent behavior have the required Microsoft 365 Copilot license in the same tenant as the agent.

4. Review content moderation behavior

• Before responses are returned, a content moderation check is performed. If the user’s query is deemed unanswerable under moderation rules, the request fails and the agent may fall back to a generic “cannot find” or “I’m not sure” message.
• Queries that focus on retrieving data (for example, “What are the sales figures for Q1 2024?”) pass more reliably.
• Queries that look like support questions, how-to instructions, text search (“find all items containing …”), or unstructured analysis may fail moderation.

Recommended checks:

• Compare prompts that succeed vs. fail and ensure they are phrased as data-retrieval questions rather than instructions or support/how-to requests.
• Standardize the wording of user prompts to reduce borderline cases that might sometimes pass and sometimes fail moderation.

5. Improve grounding and reduce hallucinations

• Incorrect or unrelated answers often occur when the model doesn’t have sufficiently grounded data.
• Ensure that the knowledge base contains comprehensive, high-quality, and unambiguous performance data.
• Configure explicit fallback behavior, such as “I don’t have that information,” when the knowledge source doesn’t contain an answer, instead of letting the model improvise.
• Review and, if needed, adjust conversational boosting settings; in some cases, disabling or reducing boosting can decrease off-topic or hallucinated responses.

6. Multi-part or complex prompts

• If the performance question is part of a multi-part request (for example, “Show performance data and then create a report”), the agent may sometimes focus on one part and ignore or mishandle the other, leading to inconsistent responses.
• Configure the agent to detect multi-part requests and guide users to handle one task at a time.

7. Publishing and environment checks

• For knowledge sources like Excel/markdown in SharePoint, unpublished agents may only process a limited extract of the data (for example, only the first several rows). This can cause partial or inconsistent answers.
• Ensure the agent is fully published and that testing is done against the published version.

8. Conditional Access and token issues

• If Conditional Access policies intermittently block token acquisition for some users or sessions, the agent may appear unresponsive or fail to retrieve knowledge, which can surface as “cannot find data” messages.
• Work with the Entra ID/Conditional Access admin to confirm that policies are not blocking Copilot Studio agents from acquiring tokens for the affected users.

Practical next steps for this scenario

1. Confirm that the agent is published and that all test users have the correct Microsoft 365 Copilot license.
2. Verify that the performance data files are:
   • Stored in the configured SharePoint/OneDrive location.
   • Accessible (shared) to all users who experience the issue.
   • Visible and searchable in SharePoint search.
3. Test a small set of standardized, clearly data-retrieval prompts and observe whether failures correlate with:
   • Recently changed or uploaded documents.
   • Specific user accounts (licensing or access issues).
   • More complex or multi-part phrasing.
4. Configure explicit fallback messages and tighten instructions so the agent only answers from validated data, reducing unrelated responses.
5. If issues persist, collect session IDs, timestamps, user accounts, and example prompts that both succeed and fail, and open a support ticket so logs can be analyzed.

---

References:

• Troubleshoot enterprise knowledge sources
• Troubleshoot Store Operations Agent (preview)
• Generative answers pointing to SharePoint sources don't return results
• Known issues
• Troubleshoot Personalized Shopping Agent (preview)
• Copilot Studio Agent doesn't recognize the docs in the Knowledge Base. - Microsoft Q&A
• Copilot Studio Agent will not read entire data set in Excel or markdown - Microsoft Q&A
• Troubleshoot Conditional Access policy issues