It is possible to bring external data processed in the Functions App to Cognitive Search?

Question

I have an external API from which I retrieve my knowledge base in JSON format. The approach I have been working on is as follows:

First, I create an HTTP function in Azure that sends a request to my API to fetch the data.

Once the data is received within the HTTP function, I invoke my Blob Trigger function at that point, passing the context and the data (from the JSON) as parameters. The Blob Trigger function is responsible for validating whether the content of the JSON file is identical to the content of the JSON file already existing in the container. If the content is the same, no changes are made. However, if it differs, the file is rewritten to update the data in "real-time".

I am seeking an approach where I can have a separate function dedicated to calling the external API and retrieving the JSON data. Within that function, after obtaining the data, I can manually invoke the blob trigger function and pass the fetched data as an argument. The blob trigger function will then be triggered and process the provided data, performing the necessary operations.

Once the data is processed and stored in the container, I intend to invoke it and create an index in Cognitive Search to facilitate advanced information searches.

Using the results obtained from Cognitive Search, my goal is to utilize Azure OpenAI services to generate GPT responses based on my up-to-date knowledge base.

Now, I have some questions:

1. Is this approach sustainable and scalable?
2. Can I truly maintain real-time data updates with this architecture?
3. In terms of pricing, is it advisable if I aim to achieve a good cost/benefit ratio?

Thanks in advance for your helps.

Answer 1

Azure OpenAI now supports bringing your own data sources. Instead of using the function & blob & all the process you are doing, do take a look at the latest BUILD announcement where you can bring in your own data to Azure OpenAI & Azure OpenAI can now provide citation from you own data.

Infact you can create a custom model & fine tune it based on your data source. More info here: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/fine-tuning?pivots=programming-language-studio#choose-your-training-data