Ingesting webpage URL for the open AI web app in Azure

Question

Hi there. In the Azure open AI studio, there is an option for defining webpage URL when you add data for the app but based on the requirements in the Microsoft website, it can only extract text up tp 20 sublinks and also I can only put one URL in it. After deploying the open AI Web app, what is the best way to ingest or define the webpages for the app. I do not want to extract the text myself and put the extracted data in the blob storage. Is there any other way that the app/search automatically extracts info from URLs and I put the URLs that I need somewhere?

Answer 1

Hi @Jalali, Hadi , I have received the confirmation that increasing the 20 URL/web address limit when using "on your data" from a URL/web address is not supported. This feature is in backlog but there no ETA for the same. You may have to create a custom solution for your scenario. You can try implementing a solution which uses a web crawler like "Beautiful Soup" which extracts contents from any web page. Then you can upload the extracted content on a Blob Storage Container and use Azure Search Documents SDK for indexing your documents programmatically. Please refer to the below documents for additional details -

• Tutorial: Optimize indexing with the push API
• Azure.Search.Documents Samples - Indexing Documents

Please let me know if you have any other questions.

Thanks

Saurabh

Please 'Accept as answer' and Upvote if it helped so that it can help others in the community looking for help on similar topics.