How to create custom content filter for serverless Llama 3.1. deployment

Question

Hi,

Could you tell me if it's possible to create and apply a custom content filter to a Llama 3.1 405B Instruct model deployed in serverless mode, and if so, how, please?

I tried to create it in Azure AI Foundry, in the Safety+security page of the project the model is deployed in, but I can't select a Connection, the list is empty.

I'm not completely sure if this is necessary, but I have added an Azure AI Content Safety resource and have added it as a connected resource to the project but that hasn't resolved the issue. I can experiment with Content filtering in the AI Services -> Content Safety -> Moderate text context section of the project using the Azure AI Content Safety resource I have created but it's not clear to me how I can save the filter I create there and apply it to the model.

Thanks,

Elizabeth

Answer 1

Hi Elizabeth Nandorfi

By Default, Model deployed through serverless endpoint use content filters with medium threshold for text models and lower threshold for image models

Reference- https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/model-catalog-content-safety

For Llama, It is not showing any option to configure custom content filter or set it as Non.

For other Azure AI infernece models ,We normally create custom content filters with higher severity threshold from safety and security tab and connect to a model deployment at "Deployment(Optional)" or update the deployments with custom content filter as shown in below screenshots. (Limited supported for Azure AI inference models)

You can also set Content filter as None (instead of custom content filter) for Chat completions and other models from Azure AI inference models aside text embedding, time series models

Custom Content filters

But all the model under serverless inference other than Azure AI inference is not supported, (specially timeseries and embedding models), You have to create an Azure content safety resource and pass the text inputs before/after passing it to the Serverless endpoint (llama 3.1 endpoint) to moderate it further.

Content Safety response

When a content filter is triggered, a 400-error code is returned with the description of the rule triggered.

You can use the sample code from "consume" tab of serverless endpoint and chain it with content safety codes mentioned here in content safety documentation.

Hope it addresses your queries.

Thank you.