Help building my first test app

Question

Hi, I am currently building a test app using the OpenAI Studio. Eventually I want to build a search and retrieve functionality but first I must build up a great dataset with all the files I want the applicaiton to base its answers from.

The data I want to be used are currently being accessible /downloadable from a public website. But I figure that they are not readable on the internet rather you need to download each file to read as a user today. In total there should be around 10,000+ files so not a task to be done manually.

My question is...

What is the easiest/quickest way to setup my applicaiton so that I can get all the data readily available in the app..

The automated workflow I had in mind (probably not how it works)

step 1) read url and go to website

step 2) download all the files from the website library that is the starting page the initial url goes to

step 3) store all the data in some clever way in azure

step 4) access all the files and data/text stored in them as a input when I build my openAI app.

step 5) profit??

Answer 1

Hello @Wallin, Isak ,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Regarding your questions, I understand that you would like to build app named test app using Azure OpenAI Studio with your specific data and workflow.

Most of all, this sound like a project not that you stumble into an error during your development which is the purpose of this platform. Microsoft Q&A is to answer questions you may have or encounter while using Microsoft product, technology and services.

I can see you have a burning desire and ready to build your application, so in this answer I will provide basic guides to start your app development, also you can utilize "additional resources" at the right side of this page. Meanwhile, whenever you encounter error or difficult challenges in your work, we would be very happy to assist you.

Let's get started to creating a test app using Azure OpenAI Studio. By following this steps:

Prerequisites:

• Get Azure Subscription, if you don't have one you can create a free Azure subscription.
• Get Access to Azure OpenAI, you can apply for access by completing the form at this link.
• Get Azure OpenAI Resource, ensure you have an Azure OpenAI resource with a deployed model. You can read more https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource

Be Conversant to use Azure OpenAI Studio:

• Select the appropriate directory, Azure subscription, and your Azure OpenAI resource.
• Explore the Playground to experiment.
• Explore the platform for building, deploying, and managing AI models.

Understand Content Moderation:

• Azure OpenAI performs content moderation on prompt inputs and generated outputs. Harmful content may be filtered. For more details, you can read from content filter article.

Create a new Project:

• In Azure OpenAI Studio, create a new project for your application and give it an appropriate name and description.

Set up Data Storage:

• Azure Data Lake Storage, or Azure SQL Database and Azure Blob Storage, you can choose the one that fits your requirements best.

Automate Web Scraping:

• You have option for the tools as well as databases we discuss above. Azure Functions or Azure Logic Apps to automate the web scraping process will be two main options for you. You can also use tools like Selenium or Beautiful Soup for scraping web pages.
• After you determine your tool, then you can write a script or workflow that :

1. reads the URL,
2. navigates to the website, and
3. downloads the files from the library.

Store Data in Azure:

• Once the files are downloaded, store them in your chosen Azure data storage solution. Organize the data in a way that makes it easy to access later.

Build OpenAI App:

• Develop your OpenAI application using Azure OpenAI services such as Azure Cognitive Services or Azure Machine Learning.
• Access the data stored in Azure storage as input for your application.

Deploy and Run:

• Once your application is built, deploy it within Azure OpenAI Studio. Test it to ensure everything works as expected.

Monitor and Manage:

• Monitor the performance of your application and manage it as needed. Azure provides tools for monitoring resource usage, scaling, and debugging.

If you read the above carefully, you will find out how it simplifies your challenges into steps you can take to build your desire app. In following the above steps, you might need some YouTube video/online training, documentations and more research to achieve them.

If you carefully follow the steps, you will definitely achieve your dream app.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam

Answer 2

@Wallin, Isak At this point of time, the option that is available for your scenario is to use URL/web address option of BYOD feature but it has some limitations. Please see this page and snapshot below.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.