How do I create a Web Crawling Solution using Azure AI Foundry?

Question

I am working on a Proof of Concept (POC) for a client where I need to web crawl a website and use the extracted data in Azure AI Foundry to be build an Agent that will offload majority of their Call Center workload. However, I realized that inside the Chat Playground in Azure AI Foundry, web crawling is not directly supported. What is the best approach to implement this capability within the Azure ecosystem? Are there recommended integrations or alternative workflows for achieving this?

Answer 1

Hi @Eza Ngam
Thanks for reaching out to Microsoft Q&A.
To get web-crawled data into Azure AI Foundry, you can use various Azure services for easy data extraction, processing, and integration. Start by setting up a web crawler with Azure Functions or Azure Container Apps using tools like Scrapy (Python) or Puppeteer (Node.js) to grab website content. Store the data in Azure Blob Storage or Azure Cosmos DB for structured and scalable storage.

Then, index and make the data searchable with Azure AI Search for quick retrieval. Hook up the indexed data to Azure AI Foundry using Connections so your AI agents can access the latest info dynamically. More details on setting up Connections in AI Foundry can be found here: Azure AI Studio - Connections. To keep things running smoothly, use Azure Monitor and Application Insights to track web crawling activities and ensure your AI system is responsive. Check out the Azure Monitoring Monitor Applications in Azure AI Studio. This method helps Azure AI Foundry use real-time crawled data, making the AI system more dynamic and effective for handling customer queries.
If the answer is helpful, please click Accept Answer and kindly upvote it so that other people who faces similar issue may get benefitted from it.
Please let us know if you need further assistance.