Implementing RAG Application with Dynamic Index Refresh on Azure Storage Blob Changes

Chandresh Maniya 0 Reputation points
2024-06-06T13:46:26.6133333+00:00

Hello,

I am working on creating a Retrieval-Augmented Generation (RAG) application and I need assistance with dynamically refreshing the index based on changes in Azure Storage Blob. Specifically, I want to trigger reindexing when new documents are added or existing documents are deleted from the blob storage.

Scenario:

  • The RAG application initially indexes documents stored in Azure Storage Blob.
  • When new documents are added or existing documents are deleted, the index should automatically refresh to include/exclude these documents.
  • The updated index should be used for future queries to ensure the latest documents are considered.

Requirements:

  1. Detecting Changes: How can I detect changes (additions or deletions) in Azure Storage Blob?
  2. Triggering Reindexing: What is the best way to trigger the reindexing process upon detecting changes?
  3. Sample Code: Any example code or guidance on implementing this would be highly appreciated.

Current Progress:

I am familiar with setting up a RAG application and indexing documents, but I am unsure about the best approach to monitor and handle changes in Azure Storage Blob to dynamically refresh the index.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,471 questions
Azure Storage Explorer
Azure Storage Explorer
An Azure tool that is used to manage cloud storage resources on Windows, macOS, and Linux.
237 questions
Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
791 questions
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,417 questions
Azure AI services
Azure AI services
A group of Azure services, SDKs, and APIs designed to make apps more intelligent, engaging, and discoverable.
2,506 questions
0 comments No comments
{count} votes