Copilot Studio Questions

Question

Copilot Studio Questions

Eric S 0

I'm building a ChatBot for a public transportation agency. I'm facing few challenges with Copilot Studio:

I would like to extract entities from a text. It seems that the only option to extract entities is to ask a question. Any other alternatives?
I would like to web crawl the agency web site for the knowledge. I have some knowledge located at the third or four level down in the web site. How I can make sure the pages at the level will be crawled? If not any other alternatives?
I would like to have a more deterministic approach for the web crawl. How can I verify that child web pages will be crawled or skipped? Any way to accomplish that?

Prashanth Veeragoni 4,930 Reputation points Microsoft External Staff Moderator

2025-03-26T04:43:34.84+00:00

Hi Eric S,

I understand that you're working on building a chatbot using Microsoft Copilot Studio for a public transportation agency, and you have challenges related to entity extraction and web crawling for knowledge retrieval. Below are detailed explanations and possible solutions for each query:

1.Extracting Entities Without Asking a Question

Issue:

In Copilot Studio, entities (like dates, locations, names) are typically extracted using a pre-built entity recognition feature that works when the bot asks a direct question.

You need a way to extract entities from user input without requiring a direct question.

Solution:

There are a few workarounds to achieve this:

A. Use System Entities (Pre-Built)

Copilot Studio includes pre-built system entities for common data types like:

datetimeV2 → Extracts dates and times.

geographyV2 → Extracts locations.

number → Extracts numerical values.

If the user provides a sentence, you can use system entities to extract relevant parts of the input dynamically.

B. Use Custom Entities

You can create custom entities to recognize specific words/phrases in user input.

Define patterns or lists of keywords related to transportation (e.g., bus stops, route names).

Implement a trigger phrase model that maps user input to extracted entities without requiring explicit questions.

C. Use Power Automate for Advanced Extraction

If system/custom entities don’t fully cover your needs, use Power Automate to call:

Azure AI Language Services → Extract named entities (NER) using AI.

Regex-based filtering → Identify structured patterns in input.

Custom ML models (via Azure or external services) → Detect specific terms dynamically.

2.Web Crawling the Agency’s Website for Knowledge

Issue:

You want the bot to crawl deep web pages (third or fourth level) on the agency’s website for knowledge.

By default, Copilot Studio’s website crawling may not index deeply nested pages.

Solution:

To ensure deeper web crawling, follow these methods:

A. Use Website Authentication & Sitemap Strategy

Ensure the website allows bots to crawl deep pages:

Check robots.txt to see if the agency has blocked certain pages.

If possible, request permission to whitelist your chatbot for deeper crawling.

Manually Provide URLs to Crawl:

If deep links aren't being discovered automatically, add the specific URLs manually in Copilot Studio’s knowledge sources.

Provide a sitemap.xml file containing all pages you want to crawl.

B. Use Azure Cognitive Search for Better Indexing

Instead of relying on Copilot Studio’s default crawl, index the website content with Azure Cognitive Search.

Steps:

Configure Azure Cognitive Search to crawl deep pages.

Use indexing rules to ensure pages at the 3rd or 4th level are included.

Connect Copilot Studio to Azure Search as a knowledge source.

C. Use Power Automate for Custom Web Crawling

If Copilot Studio's built-in crawl misses deep pages, build a custom web scraper using:

Power Automate → Automate requests to extract content from deeper pages.

Azure Functions → Use Python-based web scraping (BeautifulSoup/Scrapy).

External APIs (Google Custom Search API or Scrapy Cloud) → Fetch deep content.

3.Verifying Whether Child Pages Are Crawled

Issue:

You need a deterministic approach to verify which web pages were crawled or skipped.

Solution:

A. Use Copilot Studio’s Web Crawler Reports

Enable crawler logs in Copilot Studio to see which pages were indexed.

If some URLs are missing, manually add them.

B. Use Azure Cognitive Search Debugging

Check Indexed Content:

If using Azure Cognitive Search, inspect the index to confirm which pages were captured.

Run Queries on the Search Index:

Perform a test query in Azure Search Explorer to see retrieved results.

C. Implement Power Automate for Logging

Use Power Automate to trigger after crawling and log the fetched URLs into SharePoint/Excel.

If expected pages are missing, automate a retry mechanism.

Hope the above steps addresses all your queries, if you still have any queries, please do reach out to me.

Thank you.
Nivedipa-MSFT 3,646 Reputation points Microsoft External Staff Moderator

2025-03-27T06:20:19.09+00:00

@Eric S - Could you please confirm if your issue is resolved with the above suggestion or are you still looking for any help?
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-04-23T20:56:50.3733333+00:00

@Eric S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-04-24T13:15:23.9766667+00:00

@Eric S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Your answer

Nivedipa-MSFT 3,646 Reputation points Microsoft External Staff Moderator

2025-03-27T06:20:19.09+00:00

@Eric S - Could you please confirm if your issue is resolved with the above suggestion or are you still looking for any help?
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-04-23T20:56:50.3733333+00:00

@Eric S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-04-24T13:15:23.9766667+00:00

@Eric S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Copilot Studio Questions

Your answer