This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 68%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EES%3C/text%3E%3C/svg%3E)
I'm building a ChatBot for a public transportation agency. I'm facing few challenges with Copilot Studio:
Hi Eric S,
I understand that you're working on building a chatbot using Microsoft Copilot Studio for a public transportation agency, and you have challenges related to entity extraction and web crawling for knowledge retrieval. Below are detailed explanations and possible solutions for each query:
1.Extracting Entities Without Asking a Question
Issue:
In Copilot Studio, entities (like dates, locations, names) are typically extracted using a pre-built entity recognition feature that works when the bot asks a direct question.
You need a way to extract entities from user input without requiring a direct question.
Solution:
There are a few workarounds to achieve this:
A. Use System Entities (Pre-Built)
Copilot Studio includes pre-built system entities for common data types like:
datetimeV2 → Extracts dates and times.
geographyV2 → Extracts locations.
number → Extracts numerical values.
If the user provides a sentence, you can use system entities to extract relevant parts of the input dynamically.
B. Use Custom Entities
You can create custom entities to recognize specific words/phrases in user input.
Define patterns or lists of keywords related to transportation (e.g., bus stops, route names).
Implement a trigger phrase model that maps user input to extracted entities without requiring explicit questions.
C. Use Power Automate for Advanced Extraction
If system/custom entities don’t fully cover your needs, use Power Automate to call:
Azure AI Language Services → Extract named entities (NER) using AI.
Regex-based filtering → Identify structured patterns in input.
Custom ML models (via Azure or external services) → Detect specific terms dynamically.
2.Web Crawling the Agency’s Website for Knowledge
Issue:
You want the bot to crawl deep web pages (third or fourth level) on the agency’s website for knowledge.
By default, Copilot Studio’s website crawling may not index deeply nested pages.
Solution:
To ensure deeper web crawling, follow these methods:
A. Use Website Authentication & Sitemap Strategy
Ensure the website allows bots to crawl deep pages:
Check robots.txt to see if the agency has blocked certain pages.
If possible, request permission to whitelist your chatbot for deeper crawling.
Manually Provide URLs to Crawl:
If deep links aren't being discovered automatically, add the specific URLs manually in Copilot Studio’s knowledge sources.
Provide a sitemap.xml file containing all pages you want to crawl.
B. Use Azure Cognitive Search for Better Indexing
Instead of relying on Copilot Studio’s default crawl, index the website content with Azure Cognitive Search.
Steps:
Configure Azure Cognitive Search to crawl deep pages.
Use indexing rules to ensure pages at the 3rd or 4th level are included.
Connect Copilot Studio to Azure Search as a knowledge source.
C. Use Power Automate for Custom Web Crawling
If Copilot Studio's built-in crawl misses deep pages, build a custom web scraper using:
Power Automate → Automate requests to extract content from deeper pages.
Azure Functions → Use Python-based web scraping (BeautifulSoup/Scrapy).
External APIs (Google Custom Search API or Scrapy Cloud) → Fetch deep content.
3.Verifying Whether Child Pages Are Crawled
Issue:
You need a deterministic approach to verify which web pages were crawled or skipped.
Solution:
A. Use Copilot Studio’s Web Crawler Reports
Enable crawler logs in Copilot Studio to see which pages were indexed.
If some URLs are missing, manually add them.
B. Use Azure Cognitive Search Debugging
Check Indexed Content:
If using Azure Cognitive Search, inspect the index to confirm which pages were captured.
Run Queries on the Search Index:
Perform a test query in Azure Search Explorer to see retrieved results.
C. Implement Power Automate for Logging
Use Power Automate to trigger after crawling and log the fetched URLs into SharePoint/Excel.
If expected pages are missing, automate a retry mechanism.
Hope the above steps addresses all your queries, if you still have any queries, please do reach out to me.
Thank you.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 85%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENM%3C/text%3E%3C/svg%3E)
@Eric S - Could you please confirm if your issue is resolved with the above suggestion or are you still looking for any help?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
@Eric S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
@Eric S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.