How to deploy Falcon-40b-instruct on Azure ML?

Question

How to deploy Falcon-40b-instruct on Azure ML?

jjkam 25

Hello community,

I'm seeking assistance regarding the deployment of Falcon-40b-instruct on Azure ML. My PC lacks the necessary power to download and containerize the model for later deployment on Azure.

Interestingly, I've noticed that Amazon Sagemaker has already supported the deployment of Falcon-40b for a week now. This raises the question: Why hasn't Microsoft added this capability to their portal, particularly with Hugging Face integrations?

For reference, here is the link to the Amazon Sagemaker blog post discussing the deployment of Falcon-40b with large model inference DLCs:

https://aws.amazon.com/blogs/machine-learning/deploy-falcon-40b-with-large-model-inference-dlcs-on-amazon-sagemaker/

Any insights or alternative methods for deploying Falcon-40b-instruct on Azure ML would be greatly appreciated. Thank you!

YutongTie-MSFT 54,011 Reputation points Moderator

2023-06-21T02:51:19.8666667+00:00

Hello @jjkam

Thanks for reaching out to us, we are checking internally on this point with product team. Will let you know once we get any response.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
jjkam 25 Reputation points

2023-06-23T19:48:50.4133333+00:00

@YutongTie-MSFT any updates?
Mohamed Bilel Hasni 0 Reputation points

2023-06-27T14:56:42.4733333+00:00

Any updates??
jjkam 25 Reputation points

2023-06-27T15:31:00.3533333+00:00

@Mohamed Bilel Hasni none. Didn't hear from @YutongTie-MSFT so I switched to Amazon. Can't wait forever.
Mohamed Bilel Hasni 0 Reputation points

2023-06-27T20:08:45.5933333+00:00

Thanks @jjkam , I think I'll do the same!
Nico Döring 0 Reputation points

2023-06-27T21:27:25+00:00
I'm interested in an easy to use solution as well.

Attempts for deployment of a batch endpoint through the Azure ML Package client library for Python, LangChains HuggingFacePipeline and a scoring script results in the ValueError shown below. As far as the logs show, this is a result from the transformers from_pretrained function which needs the trust_remote_code parameter set to True.

Any tips or fixes for this issues are appreciated.

ValueError: signal only works in main thread of the main interpreter
jjkam 25 Reputation points

2023-06-27T21:58:40.5566667+00:00

@Nico Döring try Hugging Face inference endpoints.
Nico Döring 0 Reputation points

2023-06-28T11:16:11.3666667+00:00

@jjkam thanks for the suggestion. Unfortunately my project requirements state that everything needs to be deployed directly through Azure/Azure ML. So a solution through the model catalog with possibility for deployment from SDK or CLI would be ideal.
jjkam 25 Reputation points

2023-06-28T11:24:29.76+00:00

@Nico Döring depending on the model you're willing to deploy, if it's not heavy and your machine can handle the download, you can deploy your Hugging Face model to ACI as a Docker container. This would allow you to control the threading model of the deployment.
Nico Döring 0 Reputation points

2023-06-29T08:55:56.1633333+00:00

@jjkam I'll have a look into that, thanks!
Sarang Brahme 1 Reputation point

2023-08-18T19:23:42.5633333+00:00

I haven't tried fully yet, but found this on Medium:
https://medium.com/microsoftazure/running-the-falcon-40b-instruct-model-on-azure-kubernetes-service-2c3c2fb82b5
YutongTie-MSFT 54,011 Reputation points Moderator

2023-08-19T02:10:34.19+00:00

Hello @Sarang Brahme @Ananthakrishnan @Nico Döring @jjkam

Sorry for the late response, there is a preview feature which you can leverage the model directly in Azure Machine Learning Studio, please see the blog for a quick introduce and find it in the screenshot -

Falcon is a new family of language models comprising two base models: Falcon-40B and Falcon-7B. Falcon-40B tops the charts of the Open LLM Leaderboard, while Falcon-7B is the best in its weight class. Falcon family also has instructive versions of the models, Falcon-7B-Instruct and Falcon-40B-Instruct, which are finetuned on instructions and conversational data making them better suited for assistant-style tasks.

https://techcommunity.microsoft.com/t5/ai-machine-learning-blog/falcon-llms-in-azure-machine-learning/ba-p/3876847

I hope this helps, and apologize for the late response again.

Regards, Yutong

1 answer

Your answer

YutongTie-MSFT 54,011 Reputation points Moderator

2023-06-21T02:51:19.8666667+00:00

Hello @jjkam

Thanks for reaching out to us, we are checking internally on this point with product team. Will let you know once we get any response.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
jjkam 25 Reputation points

2023-06-23T19:48:50.4133333+00:00

@YutongTie-MSFT any updates?
Mohamed Bilel Hasni 0 Reputation points

2023-06-27T14:56:42.4733333+00:00

Any updates??
jjkam 25 Reputation points

2023-06-27T15:31:00.3533333+00:00

@Mohamed Bilel Hasni none. Didn't hear from @YutongTie-MSFT so I switched to Amazon. Can't wait forever.
Mohamed Bilel Hasni 0 Reputation points

2023-06-27T20:08:45.5933333+00:00

Thanks @jjkam , I think I'll do the same!
Nico Döring 0 Reputation points

2023-06-27T21:27:25+00:00

I'm interested in an easy to use solution as well.

Attempts for deployment of a batch endpoint through the Azure ML Package client library for Python, LangChains HuggingFacePipeline and a scoring script results in the ValueError shown below. As far as the logs show, this is a result from the transformers from_pretrained function which needs the trust_remote_code parameter set to True.

Any tips or fixes for this issues are appreciated.

ValueError: signal only works in main thread of the main interpreter
jjkam 25 Reputation points

2023-06-27T21:58:40.5566667+00:00

@Nico Döring try Hugging Face inference endpoints.
Nico Döring 0 Reputation points

2023-06-28T11:16:11.3666667+00:00

@jjkam thanks for the suggestion. Unfortunately my project requirements state that everything needs to be deployed directly through Azure/Azure ML. So a solution through the model catalog with possibility for deployment from SDK or CLI would be ideal.
jjkam 25 Reputation points

2023-06-28T11:24:29.76+00:00

@Nico Döring depending on the model you're willing to deploy, if it's not heavy and your machine can handle the download, you can deploy your Hugging Face model to ACI as a Docker container. This would allow you to control the threading model of the deployment.
Nico Döring 0 Reputation points

2023-06-29T08:55:56.1633333+00:00

@jjkam I'll have a look into that, thanks!
Sarang Brahme 1 Reputation point

2023-08-18T19:23:42.5633333+00:00

I haven't tried fully yet, but found this on Medium:
https://medium.com/microsoftazure/running-the-falcon-40b-instruct-model-on-azure-kubernetes-service-2c3c2fb82b5
YutongTie-MSFT 54,011 Reputation points Moderator

2023-08-19T02:10:34.19+00:00

Hello @Sarang Brahme @Ananthakrishnan @Nico Döring @jjkam

Sorry for the late response, there is a preview feature which you can leverage the model directly in Azure Machine Learning Studio, please see the blog for a quick introduce and find it in the screenshot -

Falcon is a new family of language models comprising two base models: Falcon-40B and Falcon-7B. Falcon-40B tops the charts of the Open LLM Leaderboard, while Falcon-7B is the best in its weight class. Falcon family also has instructive versions of the models, Falcon-7B-Instruct and Falcon-40B-Instruct, which are finetuned on instructions and conversational data making them better suited for assistant-style tasks.

https://techcommunity.microsoft.com/t5/ai-machine-learning-blog/falcon-llms-in-azure-machine-learning/ba-p/3876847

I hope this helps, and apologize for the late response again.

Regards, Yutong

Answer 1

Ananthakrishnan 0

@YutongTie-MSFT Any updates on the same. Me too looking for a resolution on the same.

Share via

How to deploy Falcon-40b-instruct on Azure ML?

1 answer

Your answer