Azure Cognitive Search indexing and external website

Question

Is it possible to index an external website using Azure Cognitive Search?

I'm looking for some guides on how this should be setup but most of them relate to setting up Cognitive Search indexes to index Azure services.

Any pointers welcome.

Answer 1

Hi @Ireland, Robert (IT Services) , Yes I believe it is possible to index an external website using Azure Cognitive Search. You can use the "Custom Skill" feature of Cognitive Search to extract data from external websites and add it to your search index.

Here's how you can set it up:

1. Create a Cognitive Search service in the Azure portal if you haven't already done so.
2. Create a search index and define the schema for the data you want to index.
3. Create a data source for the external website you want to index. You can use the "Web" data source type and specify the URL of the website.
4. Create a skillset that defines the custom skill you want to use to extract data from the website. You can use an open-source library like BeautifulSoup or Scrapy to extract data from the HTML of the website.
5. Add the custom skill to the skillset and configure it to extract the data you want to index.
6. Create an indexer that uses the data source, skillset, and search index you created earlier. The indexer will automatically extract data from the website and add it to your search index.

Here are some resources that can help you get started:

• Indexers in Azure Cognitive Search
• Pulling data into an index
• Example: Create a custom skill using the Bing Entity Search API
• Add a custom skill to an Azure Cognitive Search enrichment pipeline

Another way to achieve this is by using Azure Cognitive Search + Azure Functions + Azure Blob Storage. Please see more details in this similar Stack Overflow thread for more details.

Hope that helps.

Answer 2

Hello @Ireland, Robert (IT Services),

You can't directly, these are the Supported data sources for now:

Indexers crawl data stores on Azure and outside of Azure.

• Azure Blob Storage
• Azure Cosmos DB
• Azure Data Lake Storage Gen2
• Azure SQL Database
• Azure Table Storage
• Azure SQL Managed Instance
• SQL Server on Azure Virtual Machines
• Azure Files (in preview)
• Azure MySQL (in preview)
• SharePoint in Microsoft 365 (in preview)
• Azure Cosmos DB for MongoDB (in preview)
• Azure Cosmos DB for Apache Gremlin (in preview)

In the other hand, you have Data Sources from partners: https://learn.microsoft.com/en-us/azure/search/search-data-sources-gallery

Maybe this project might be interesting to you: https://github.com/thomas11/AzureSearchCrawler

Good Luck!!