Choose the right language model for Edge RAG Preview enabled by Azure Arc

Review available model options and understand model requirements to choose the right language model for your Edge RAG deployment. This article is part of the deployment prerequisites checklist.

Important

Edge RAG Preview, enabled by Azure Arc is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Select a language model

Decide which language model your organization wants to deploy. You can use your own language model or use one of the Microsoft provided language models.

After Edge RAG extension is deployed, you can't change the language model. Therefore, work with your application development team to decide which is the right model for your organization's use case.

You can refer to some of these resources from Microsoft to choose the right model for your use case:

• Blog: How to Choose the Right Models for Your Apps | Azure AI
• Video: How to Choose the Right Models for Your Apps | Azure AI - YouTube
• Azure AI Foundry also provides tooling such as model benchmarks to choose the right model.

Microsoft provided language models

If you don't have your own language model to use with Edge RAG, select one of the following Microsoft provided language models when you deploy the Edge RAG extension:

• Microsoft Phi 3.5 Mini
• Mistral 7B

Bring your own language model

Edge RAG works with small language models (SLM) or large language models (LLM) that expose endpoints that support the OpenAI inference API. Set up these models locally using Kubernetes AI toolchain operator (KAITO) or similar mechanisms. Edge RAG can also work with OpenAI models in Azure that need API Key-based authentication.

If you plan to use your own language model with Edge RAG, you must complete the steps in the following articles:

• Before you deploy Edge RAG, create an endpoint to use for Edge RAG deployment.
• After you deploy the Edge RAG extension, configure "BYOM" endpoint authentication for Edge RAG.

Next step

If you choose to:

• Use a Microsoft provided language model, see Verify NFS server access for Edge RAG.
• Bring use your own language model, see Create an endpoint to use for Edge RAG.