How to fix "Python Fatal Error: Invalid Instruction" when deploying llm using AML endpoint.

Question

How to fix "Python Fatal Error: Invalid Instruction" when deploying llm using AML endpoint.

Dmytro Dronov 0

I'm deploying a large language model (llm) using the llama-cpp-python library using Azure Machine Learning endpoints.

During provisioning deployment I get this error:

2023-07-12 16:13:33,906 I [1042] azmlinfsrv - Starting up app insights client
2023-07-12 16:13:35,513 I [1042] azmlinfsrv.user_script - Found user script at /var/azureml-app/230712175720-2263799543/score.py
2023-07-12 16:13:35,513 I [1042] azmlinfsrv.user_script - run() is not decorated. Server will invoke it with the input in JSON string.
2023-07-12 16:13:35,513 I [1042] azmlinfsrv.user_script - Invoking user's init function
WARNING:entry_module:LOG_LEVEL =

Ramr-msft 17,826 Reputation points

2023-07-13T09:51:16.5366667+00:00

@Dmytro Dronov Thanks for the question. Can you please add more error details and the steps you performed to deploy the large language model using the llama-cpp-python library and Azure Machine Learning endpoints.

Dmytro Dronov 0

@Ramr-msft sure.
To deploy a model to the AML endpoint I used two ways:

Using portal.
Using CLI.

score.py

import os
import json
import logging
from llama_cpp import Llama


log = logging.getLogger(__name__)


def init():
    global llm
    try:
        model_path = os.path.join(
            os.getenv("AZUREML_MODEL_DIR"), "model.bin"
        )
        llm = Llama(model_path=model_path, n_ctx=2048)
    except Exception as e:
        log.error(f"Failed to load model. Work termination. {e}")
        raise e


def run(data) -> dict:
    try:
        prompt = data.values['prompt']
    except:
        prompt = json.loads(data)["prompt"]
    answer = llm(prompt)
    return answer

env

llama-cpp-python==0.1.71

P.S. I also noticed a strange behavior that this error does not always occur. For example, with 3/5 attempts there may be no error. I couldn't figure out the pattern, but it's a fact.

Ramr-msft 17,826 Reputation points

2023-08-03T09:54:15.8766667+00:00

Thanks for the Details, We would recommend raising an issue in the following Repo.

Your answer

Ramr-msft 17,826 Reputation points

2023-07-13T09:51:16.5366667+00:00

@Dmytro Dronov Thanks for the question. Can you please add more error details and the steps you performed to deploy the large language model using the llama-cpp-python library and Azure Machine Learning endpoints.
Dmytro Dronov 0 Reputation points

2023-07-25T08:44:55.38+00:00

@Ramr-msft sure.
To deploy a model to the AML endpoint I used two ways:

Using portal.

Using CLI.

score.py

import os import json import logging from llama_cpp import Llama log = logging.getLogger(__name__) def init(): global llm try: model_path = os.path.join( os.getenv("AZUREML_MODEL_DIR"), "model.bin" ) llm = Llama(model_path=model_path, n_ctx=2048) except Exception as e: log.error(f"Failed to load model. Work termination. {e}") raise e def run(data) -> dict: try: prompt = data.values['prompt'] except: prompt = json.loads(data)["prompt"] answer = llm(prompt) return answer

env

llama-cpp-python==0.1.71

P.S. I also noticed a strange behavior that this error does not always occur. For example, with 3/5 attempts there may be no error. I couldn't figure out the pattern, but it's a fact.
Ramr-msft 17,826 Reputation points

2023-08-03T09:54:15.8766667+00:00

Thanks for the Details, We would recommend raising an issue in the following Repo.

Share via

How to fix "Python Fatal Error: Invalid Instruction" when deploying llm using AML endpoint.

Your answer