How to deploy Llama 3.1 in AzureML job using transformers

Question

How to deploy Llama 3.1 in AzureML job using transformers

matsuo_basho 30

I run pipeline jobs programatically using AzureML. Is there a way to finetune the Llama 3.1 models? I see this tutorial, but I'm not interested in just a deployment. I need to finetune.

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-models-llama?view=azureml-api-2&tabs=llama-three

I know I can go the route of authenticating with HuggingFace from within my script detailed here, but I don't want to give Meta my contact info, birthdate, etc to use llama.

I also see an option to deploy the model from the Model Catalog. This includes finetuning, but it's for text generation. I have a more complex finetuning task - masked language model followed by a sequence-to-sequence task.

User's image

dupammi 8,615 Reputation points Microsoft External Staff

2024-08-14T04:44:10.9+00:00
Hi @matsuo_basho

Thank you for your question.

To fine-tune Llama 3.1 models on AzureML, start by setting up your Azure environment, ensuring you have an active Azure subscription and an Azure Machine Learning (AzureML) workspace, and provision a compute instance or cluster within your workspace for running fine-tuning jobs. To prepare the Llama 3.1 model, you can leverage the Azure AI Model Catalog to access Llama models, including Llama 3.1, if you want to avoid using Hugging Face and Meta’s authentication. For fine-tuning with Transformers in AzureML, install the required libraries using pip install transformers datasets and use the transformers library to fine-tune the model on tasks like masked language modeling followed by sequence-to-sequence tasks. Finally, create a Python script from the sample code provided and run it as a job in AzureML, using the azureml-sdk to submit the job programmatically.

For more details, please refer -

How to deploy Meta Llama 3.1 models with Azure Machine Learning studio

examples/pytorch/language-modeling/run_mlm.py

azure-ml-py
matsuo_basho 30 Reputation points

2024-08-14T13:23:01.7+00:00

What you wrote makes sense halfway through. However, if I select finetune Llama 3.1 from the model catalog, the only task type is text generation:
matsuo_basho 30 Reputation points

2024-08-14T15:06:49.8166667+00:00
dupammi 8,615 Reputation points Microsoft External Staff

2024-08-14T15:51:18.4466667+00:00
Hi @matsuo_basho

Sorry for the trouble you are facing. I recommend reporting this issue to the Azure support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure support team will review your request and provide assistance as soon as possible.
matsuo_basho 30 Reputation points

2024-08-15T13:18:09.1266667+00:00

It's not my responsibility as a customer to help Microsoft fix issues with your services. This isn't an open-source framework where the price we pay for free use is that we participate in improving the product. If the support would be free I'd do it, but support requires a paid plan. And even then from my experience, it takes Microsoft 2+ weeks to pass the issue to the right team get any sort of help.
dupammi 8,615 Reputation points Microsoft External Staff

2024-08-16T07:32:15.33+00:00

Hi @matsuo_basho

Thank you for your follow-up and sorry for the inconvenience.

Azure is continuously improving, and troubleshooting steps are essential to identify the root cause of the issue. To assist further, please share the steps you've taken and provide detailed logs for better investigation. This will help Azure support team to expedite the resolution.

I hope you understand. Thank you.

Your answer

dupammi 8,615 Reputation points Microsoft External Staff

2024-08-14T04:44:10.9+00:00

Hi @matsuo_basho

Thank you for your question.

To fine-tune Llama 3.1 models on AzureML, start by setting up your Azure environment, ensuring you have an active Azure subscription and an Azure Machine Learning (AzureML) workspace, and provision a compute instance or cluster within your workspace for running fine-tuning jobs. To prepare the Llama 3.1 model, you can leverage the Azure AI Model Catalog to access Llama models, including Llama 3.1, if you want to avoid using Hugging Face and Meta’s authentication. For fine-tuning with Transformers in AzureML, install the required libraries using pip install transformers datasets and use the transformers library to fine-tune the model on tasks like masked language modeling followed by sequence-to-sequence tasks. Finally, create a Python script from the sample code provided and run it as a job in AzureML, using the azureml-sdk to submit the job programmatically.

For more details, please refer -

How to deploy Meta Llama 3.1 models with Azure Machine Learning studio

examples/pytorch/language-modeling/run_mlm.py

azure-ml-py
matsuo_basho 30 Reputation points

2024-08-14T13:23:01.7+00:00

What you wrote makes sense halfway through. However, if I select finetune Llama 3.1 from the model catalog, the only task type is text generation:
matsuo_basho 30 Reputation points

2024-08-14T15:06:49.8166667+00:00
dupammi 8,615 Reputation points Microsoft External Staff

2024-08-14T15:51:18.4466667+00:00

Hi @matsuo_basho

Sorry for the trouble you are facing. I recommend reporting this issue to the Azure support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure support team will review your request and provide assistance as soon as possible.
matsuo_basho 30 Reputation points

2024-08-15T13:18:09.1266667+00:00

It's not my responsibility as a customer to help Microsoft fix issues with your services. This isn't an open-source framework where the price we pay for free use is that we participate in improving the product. If the support would be free I'd do it, but support requires a paid plan. And even then from my experience, it takes Microsoft 2+ weeks to pass the issue to the right team get any sort of help.
dupammi 8,615 Reputation points Microsoft External Staff

2024-08-16T07:32:15.33+00:00

Hi @matsuo_basho

Thank you for your follow-up and sorry for the inconvenience.

Azure is continuously improving, and troubleshooting steps are essential to identify the root cause of the issue. To assist further, please share the steps you've taken and provide detailed logs for better investigation. This will help Azure support team to expedite the resolution.

I hope you understand. Thank you.

Share via

How to deploy Llama 3.1 in AzureML job using transformers

Your answer