Accessing a Llama 3 model deployed on Azure OpenAI returns 403 "key_auth_bad_header_forbidden"

Question

I have deployed a Llama 3 model to Azure OpenAI:

I then try to access it with the following Java code:

AzureOpenAiChatModel model = AzureOpenAiChatModel.builder()
  .apiKey(AZURE_OPENAI_KEY)
  .endpoint(AZURE_OPENAI_ENDPOINT)
  .deploymentName("llama-3-2-1b-instruct-1")
  .logRequestsAndResponses(true)
  .build();

System.out.println(model.generate("When was the first Rolling Stones album released?"));

And I get a Status code 403, "key_auth_bad_header_forbidden".

The same code works fine when I deploy an OpenAI mode (eg. GPT-4o) but doesn't with non-OpenAI models such as Llama.

I am trying to solve this issue: https://github.com/langchain4j/langchain4j/discussions/1580

Any pointers? Any idea of what could be wrong ? Do I need something else than just the API Key ? Any network config I need to setup ?

Thanks

Answer 1

Hi, Antonio

My feeling is that it is unlikely to work. The llama-3-2-1b-instruct-1 (any open source LLM) is deployed to *.inference.ml.azure.com that is different to the GPT models deployment as a part of Azure OpenAI service.

Guess you used the target uri from the deployment endpoint in the format of: https://xxxxx.eastus2.inference.ml.azure.com. if you compare this with your working GPT-4o's endpoint url, you will notice the difference.

On the other hand, the GPT-4o's request structure would be slightly different to llama 3.2's. the error you see is likely related to the authentication. If you go to Consume tab, and look at the example python code, the required auth header is a bearer header, but the AzureOpenAiChatModel would be using a api-key header.

In short, dont believe the AzureOpenAiChatModel support llama-3-2-1b-instruct-1 model out of the box yet.