What Machine Type is Recommended for a 5-10MB Dataset Using RAG System in Azure ML?

Question

Hello Azure Community,

I am working on a project where I plan to implement a Retrieval-Augmented Generation (RAG) system using Azure Cognitive Search and Azure OpenAI Service. The dataset I will be using is relatively small, around 5-10MB, consisting of text documents such as FAQs, policies, and internal manuals.

Since I need to run the model efficiently within the Azure Machine Learning environment, I would like advice on:

1. Which machine type (VM SKU) would be suitable for this workload, considering the dataset size and the fact that it will involve both retrieval and generative inference?
2. Do I need GPU support for the inferencing process with OpenAI, or will a CPU-based machine suffice given the dataset size and query volume?
3. Any suggestions for scalability or optimization if the workload or query frequency increases over time?

I want to ensure that I use the most cost-effective machine that provides good performance without over-provisioning resources.

Thank you in advance for your help and suggestions!

Answer 1

@ADITYA RAJPUROHIT With respect to the size of your data set I think you can use the least available VM type for your subscription. I would recommend CPU type Standard_DS11_v2 for light weight testing or Standard_E4ds_v4 if you want more cores and RAM.

Also, since you are planning to use OpenAI deploying an endpoint of a base model should automatically take care of compute behind the endpoint as it is managed by Azure. Creation of VMs in Azure ML might only be required for data manipulation tasks or processing. If you create a compute instance ensure you shutdown your instance when not in use or schedule automatic shutdown when idle to reduce costs. I hope this helps!!

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.