Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,571 questions
Error code 400: AciDeploymentFailedin Azureml
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 34%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
ramsundar
0
Reputation points
Hi Everyone,
I have been trying for last 3 days but couldn't able to fix this issue. I have checked the model is also registered.
Below code is my score.py
#score.py
import json
import joblib
from azureml.core.model import Model
def init():
global model
model_path = Model.get_model_path(model_name = 'model_estimator_500.pkl')
model = joblib.load(model_path)
def run(raw_data):
try:
data = np.array(json.loads(raw_data)['data'])
y_hat = model.predict(data)
return y_hat.to_list()
except Exception as e:
result = str(e)
return json.dump({"error": result})
from azureml.core.webservice import Webservice
from azureml.core.webservice import AciWebservice
from azureml.core.image import ContainerImage
from azureml.exceptions import WebserviceException
from azureml.core.image import Image
from azureml.core.conda_dependencies import CondaDependencies
myenv = Environment(name="myenv")
env = CondaDependencies()
env = CondaDependencies.create(
pip_packages=['azureml-defaults','scikit-learn', 'joblib']
)
myenv.python.conda_dependencies=env
#myenv.python.conda_dependencies.save("myenv.yml")
from azureml.core.webservice import Webservice
from azureml.core.model import InferenceConfig
from azureml.core.webservice import AciWebservice
from azureml.core.environment import Environment
service_name = 'my-custom-env-service'
aciconfig = AciWebservice.deploy_configuration(cpu_cores=1, memory_gb=1)
inference_config = InferenceConfig(entry_script="score.py", environment=myenv)
service = Model.deploy(workspace=ws,
name=service_name,
models=[model],
inference_config=inference_config,
deployment_config=aciconfig,
overwrite=True)
service.wait_for_deployment(show_output=True)
print(service.scoring_uri)
#--------------------------------xx--------------------------------#
Now here this is the o/p I am getting:
/tmp/ipykernel_5105/4290996075.py:70: FutureWarning: azureml.core.model:
To leverage new model deployment capabilities, AzureML recommends using CLI/SDK v2 to deploy models as online endpoint,
please refer to respective documentations
https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoints /
https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoint-sdk-v2 /
https://docs.microsoft.com/azure/machine-learning/how-to-attach-kubernetes-anywhere
For more information on migration, see https://aka.ms/acimoemigration.
To disable CLI/SDK v1 deprecation warning set AZUREML_LOG_DEPRECATION_WARNING_ENABLED to 'False'
service = Model.deploy(workspace=ws,
Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 9d233765-e0f1-43e2-a02e-d92fc5d9fb31
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.
1. Please check the logs for your container instance: my-custom-env-service. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.
2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_c471f026f58655f5b5c96f38d8bd754b locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.
1. Please check the logs for your container instance: my-custom-env-service. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.
2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_c471f026f58655f5b5c96f38d8bd754b locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information."
},
{
"code": "AciDeploymentFailed",
"message": "Your container application crashed. Please follow the steps to debug:
1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.
2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.
3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
4. View the diagnostic events to check status of container, it may help you to debug the issue.
"RestartCount": 3
"CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"}
"PreviousState": {"state":"Terminated","startTime":"2023-01-27T03:31:58.463Z","exitCode":111,"finishTime":"2023-01-27T03:32:11.52Z","detailStatus":"Error"}
"Events": null
"
}
]
}
Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes.
Running
2023-01-27 03:28:38+00:00 Creating Container Registry if not exists.
2023-01-27 03:28:38+00:00 Registering the environment.
2023-01-27 03:28:39+00:00 Use the existing image.
2023-01-27 03:28:39+00:00 Generating deployment configuration.
2023-01-27 03:28:40+00:00 Submitting deployment to compute.
2023-01-27 03:28:44+00:00 Checking the status of deployment my-custom-env-service..
2023-01-27 03:30:36+00:00 Checking the status of inference endpoint my-custom-env-service.
Failed
---------------------------------------------------------------------------
WebserviceException Traceback (most recent call last)
Cell In [6], line 77
68 inference_config = InferenceConfig(entry_script="score.py", environment=myenv)
70 service = Model.deploy(workspace=ws,
71 name=service_name,
72 models=[model],
73 inference_config=inference_config,
74 deployment_config=aciconfig,
75 overwrite=True)
---> 77 service.wait_for_deployment(show_output=True)
79 print(service.scoring_uri)
82 #--------------------------------xx--------------------------------#
File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/webservice/webservice.py:918, in Webservice.wait_for_deployment(self, show_output, timeout_sec)
915 if not logs_response:
916 logs_response = 'Current sub-operation type not known, more logs unavailable.'
--> 918 raise WebserviceException('Service deployment polling reached non-successful terminal state, current '
919 'service state: {}\n'
920 'Operation ID: {}\n'
921 '{}\n'
922 'Error:\n'
923 '{}'.format(self.state, self._operation_endpoint.split('/')[-1],
924 logs_response, format_error_response), logger=module_logger)
925 print('{} service creation operation finished, operation "{}"'.format(self._webservice_type,
926 operation_state))
927 except WebserviceException as e:
WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 9d233765-e0f1-43e2-a02e-d92fc5d9fb31
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.
1. Please check the logs for your container instance: my-custom-env-service. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.
2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_c471f026f58655f5b5c96f38d8bd754b locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.
1. Please check the logs for your container instance: my-custom-env-service. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.
2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_c471f026f58655f5b5c96f38d8bd754b locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information."
},
{
"code": "AciDeploymentFailed",
"message": "Your container application crashed. Please follow the steps to debug:
1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.
2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.
3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.
4. View the diagnostic events to check status of container, it may help you to debug the issue.
"RestartCount": 3
"CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"}
"PreviousState": {"state":"Terminated","startTime":"2023-01-27T03:31:58.463Z","exitCode":111,"finishTime":"2023-01-27T03:32:11.52Z","detailStatus":"Error"}
"Events": null
"
}
]
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 9d233765-e0f1-43e2-a02e-d92fc5d9fb31\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"AciDeploymentFailed\",\n \"statusCode\": 400,\n \"message\": \"Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\n\t1. Please check the logs for your container instance: my-custom-env-service. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.\n\t2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_c471f026f58655f5b5c96f38d8bd754b locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.\",\n \"details\": [\n {\n \"code\": \"CrashLoopBackOff\",\n \"message\": \"Your container application crashed. This may be caused by errors in your scoring file's init() function.\n\t1. Please check the logs for your container instance: my-custom-env-service. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.\n\t2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_c471f026f58655f5b5c96f38d8bd754b locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.\"\n },\n {\n \"code\": \"AciDeploymentFailed\",\n \"message\": \"Your container application crashed. Please follow the steps to debug:\n\t1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n\t2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n\t3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t4. View the diagnostic events to check status of container, it may help you to debug the issue.\n\"RestartCount\": 3\n\"CurrentState\": {\"state\":\"Waiting\",\"startTime\":null,\"exitCode\":null,\"finishTime\":null,\"detailStatus\":\"CrashLoopBackOff: Back-off restarting failed\"}\n\"PreviousState\": {\"state\":\"Terminated\",\"startTime\":\"2023-01-27T03:31:58.463Z\",\"exitCode\":111,\"finishTime\":\"2023-01-27T03:32:11.52Z\",\"detailStatus\":\"Error\"}\n\"Events\": null\n\"\n }\n ]\n}"
}
}
print(service.get_logs())
Also when I am checking service logs its, returning me None
Thanks,
{count} votes
-
romungi-MSFT 42,206 Reputation points Microsoft Employee2023-01-27T08:53:39.2666667+00:00 ramsundar Have you tried to increase the ACI configuration to higher numbers? You currently have only 1 core and 1 GB memory, Try atleast, 2 cores and 4 GB or higher and check if this goes through. Thanks!!
aciconfig = AciWebservice.deploy_configuration(cpu_cores=2, memory_gb=4) -
romungi-MSFT 42,206 Reputation points Microsoft Employee
2023-01-30T14:03:20.4433333+00:00 ramsundar Did you get a chance to check the same with increased capacity?
-
ramsundar 0 Reputation points
2023-02-03T05:32:44.41+00:00 Hi @romungi-MSFT thanks for responding, sorry for late reply. I did check and the error is still same
I tried both combination 2core with 4 gig ram as well as 2 core with 8 gig ram. Pls help it still stuck during endpoint...
/tmp/ipykernel_20386/1551616474.py:59: FutureWarning: azureml.core.model: To leverage new model deployment capabilities, AzureML recommends using CLI/SDK v2 to deploy models as online endpoint, please refer to respective documentations https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoints / https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoint-sdk-v2 / https://docs.microsoft.com/azure/machine-learning/how-to-attach-kubernetes-anywhere For more information on migration, see https://aka.ms/acimoemigration. To disable CLI/SDK v1 deprecation warning set AZUREML_LOG_DEPRECATION_WARNING_ENABLED to 'False' service = Model.deploy(ws, aci_service_name, [model], inference_config, deployment_config, overwrite=True) Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 400cd7af-33f4-4513-912f-f8127ca63592 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function. 1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. 2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.", "details": [ { "code": "CrashLoopBackOff", "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function. 1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. 2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information." }, { "code": "AciDeploymentFailed", "message": "Your container application crashed. Please follow the steps to debug: 1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information. 2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information. 3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 4. View the diagnostic events to check status of container, it may help you to debug the issue. "RestartCount": 3 "CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"} "PreviousState": {"state":"Terminated","startTime":"2023-01-29T22:08:22.99Z","exitCode":111,"finishTime":"2023-01-29T22:08:33.053Z","detailStatus":"Error"} "Events": null " } ] } Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes. Running 2023-01-29 22:05:24+00:00 Creating Container Registry if not exists. 2023-01-29 22:05:24+00:00 Registering the environment. 2023-01-29 22:05:24+00:00 Use the existing image. 2023-01-29 22:05:24+00:00 Generating deployment configuration. 2023-01-29 22:05:25+00:00 Submitting deployment to compute. 2023-01-29 22:05:29+00:00 Checking the status of deployment aci-diabetes-model.. 2023-01-29 22:06:51+00:00 Checking the status of inference endpoint aci-diabetes-model. Failed --------------------------------------------------------------------------- WebserviceException Traceback (most recent call last) Cell In [14], line 60 57 deployment_config = AciWebservice.deploy_configuration(cpu_cores=2, memory_gb=8) 59 service = Model.deploy(ws, aci_service_name, [model], inference_config, deployment_config, overwrite=True) ---> 60 service.wait_for_deployment(True) 62 print(service.state) File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/webservice/webservice.py:918, in Webservice.wait_for_deployment(self, show_output, timeout_sec) 915 if not logs_response: 916 logs_response = 'Current sub-operation type not known, more logs unavailable.' --> 918 raise WebserviceException('Service deployment polling reached non-successful terminal state, current ' 919 'service state: {}\n' 920 'Operation ID: {}\n' 921 '{}\n' 922 'Error:\n' 923 '{}'.format(self.state, self._operation_endpoint.split('/')[-1], 924 logs_response, format_error_response), logger=module_logger) 925 print('{} service creation operation finished, operation "{}"'.format(self._webservice_type, 926 operation_state)) 927 except WebserviceException as e: WebserviceException: WebserviceException: Message: Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 400cd7af-33f4-4513-912f-f8127ca63592 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function. 1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. 2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.", "details": [ { "code": "CrashLoopBackOff", "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function. 1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. 2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information." }, { "code": "AciDeploymentFailed", "message": "Your container application crashed. Please follow the steps to debug: 1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information. 2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information. 3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 4. View the diagnostic events to check status of container, it may help you to debug the issue. "RestartCount": 3 "CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"} "PreviousState": {"state":"Terminated","startTime":"2023-01-29T22:08:22.99Z","exitCode":111,"finishTime":"2023-01-29T22:08:33.053Z","detailStatus":"Error"} "Events": null " } ] } InnerException None ErrorResponse { "error": { "message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 400cd7af-33f4-4513-912f-f8127ca63592\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"AciDeploymentFailed\",\n \"statusCode\": 400,\n \"message\": \"Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\n\t1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.\n\t2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.\",\n \"details\": [\n {\n \"code\": \"CrashLoopBackOff\",\n \"message\": \"Your container application crashed. This may be caused by errors in your scoring file's init() function.\n\t1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.\n\t2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.\"\n },\n {\n \"code\": \"AciDeploymentFailed\",\n \"message\": \"Your container application crashed. Please follow the steps to debug:\n\t1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n\t2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n\t3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t4. View the diagnostic events to check status of container, it may help you to debug the issue.\n\"RestartCount\": 3\n\"CurrentState\": {\"state\":\"Waiting\",\"startTime\":null,\"exitCode\":null,\"finishTime\":null,\"detailStatus\":\"CrashLoopBackOff: Back-off restarting failed\"}\n\"PreviousState\": {\"state\":\"Terminated\",\"startTime\":\"2023-01-29T22:08:22.99Z\",\"exitCode\":111,\"finishTime\":\"2023-01-29T22:08:33.053Z\",\"detailStatus\":\"Error\"}\n\"Events\": null\n\"\n }\n ]\n}" } } -
ramsundar 0 Reputation points
2023-02-03T05:33:07.2633333+00:00 Hi @romungi-MSFT thanks for responding, sorry for late reply. I did check and the error is still same
I tried both combination 2core with 4 gig ram as well as 2 core with 8 gig ram. Pls help it still stuck during endpoint.../tmp/ipykernel_20386/1551616474.py:59: FutureWarning: azureml.core.model: To leverage new model deployment capabilities, AzureML recommends using CLI/SDK v2 to deploy models as online endpoint, please refer to respective documentations https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoints / https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoint-sdk-v2 / https://docs.microsoft.com/azure/machine-learning/how-to-attach-kubernetes-anywhere For more information on migration, see https://aka.ms/acimoemigration. To disable CLI/SDK v1 deprecation warning set AZUREML_LOG_DEPRECATION_WARNING_ENABLED to 'False' service = Model.deploy(ws, aci_service_name, [model], inference_config, deployment_config, overwrite=True) Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 400cd7af-33f4-4513-912f-f8127ca63592 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function. 1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. 2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.", "details": [ { "code": "CrashLoopBackOff", "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function. 1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. 2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information." }, { "code": "AciDeploymentFailed", "message": "Your container application crashed. Please follow the steps to debug: 1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information. 2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information. 3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 4. View the diagnostic events to check status of container, it may help you to debug the issue. "RestartCount": 3 "CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"} "PreviousState": {"state":"Terminated","startTime":"2023-01-29T22:08:22.99Z","exitCode":111,"finishTime":"2023-01-29T22:08:33.053Z","detailStatus":"Error"} "Events": null " } ] } Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes. Running 2023-01-29 22:05:24+00:00 Creating Container Registry if not exists. 2023-01-29 22:05:24+00:00 Registering the environment. 2023-01-29 22:05:24+00:00 Use the existing image. 2023-01-29 22:05:24+00:00 Generating deployment configuration. 2023-01-29 22:05:25+00:00 Submitting deployment to compute. 2023-01-29 22:05:29+00:00 Checking the status of deployment aci-diabetes-model.. 2023-01-29 22:06:51+00:00 Checking the status of inference endpoint aci-diabetes-model. Failed --------------------------------------------------------------------------- WebserviceException Traceback (most recent call last) Cell In [14], line 60 57 deployment_config = AciWebservice.deploy_configuration(cpu_cores=2, memory_gb=8) 59 service = Model.deploy(ws, aci_service_name, [model], inference_config, deployment_config, overwrite=True) ---> 60 service.wait_for_deployment(True) 62 print(service.state) File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/webservice/webservice.py:918, in Webservice.wait_for_deployment(self, show_output, timeout_sec) 915 if not logs_response: 916 logs_response = 'Current sub-operation type not known, more logs unavailable.' --> 918 raise WebserviceException('Service deployment polling reached non-successful terminal state, current ' 919 'service state: {}\n' 920 'Operation ID: {}\n' 921 '{}\n' 922 'Error:\n' 923 '{}'.format(self.state, self._operation_endpoint.split('/')[-1], 924 logs_response, format_error_response), logger=module_logger) 925 print('{} service creation operation finished, operation "{}"'.format(self._webservice_type, 926 operation_state)) 927 except WebserviceException as e: WebserviceException: WebserviceException: Message: Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 400cd7af-33f4-4513-912f-f8127ca63592 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function. 1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. 2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.", "details": [ { "code": "CrashLoopBackOff", "message": "Your container application crashed. This may be caused by errors in your scoring file's init() function. 1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. 2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information." }, { "code": "AciDeploymentFailed", "message": "Your container application crashed. Please follow the steps to debug: 1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information. 2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information. 3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information. 4. View the diagnostic events to check status of container, it may help you to debug the issue. "RestartCount": 3 "CurrentState": {"state":"Waiting","startTime":null,"exitCode":null,"finishTime":null,"detailStatus":"CrashLoopBackOff: Back-off restarting failed"} "PreviousState": {"state":"Terminated","startTime":"2023-01-29T22:08:22.99Z","exitCode":111,"finishTime":"2023-01-29T22:08:33.053Z","detailStatus":"Error"} "Events": null " } ] } InnerException None ErrorResponse { "error": { "message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 400cd7af-33f4-4513-912f-f8127ca63592\nMore information can be found using '.get_logs()'\nError:\n{\n \"code\": \"AciDeploymentFailed\",\n \"statusCode\": 400,\n \"message\": \"Aci Deployment failed with exception: Your container application crashed. This may be caused by errors in your scoring file's init() function.\n\t1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.\n\t2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.\",\n \"details\": [\n {\n \"code\": \"CrashLoopBackOff\",\n \"message\": \"Your container application crashed. This may be caused by errors in your scoring file's init() function.\n\t1. Please check the logs for your container instance: aci-diabetes-model. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs.\n\t2. You can interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t3. You can also try to run image 361fa07ff378469392f42cc8f56c7bf9.azurecr.io/azureml/azureml_106d726f179c1be5003ca71bceba5197 locally. Please refer to https://aka.ms/debugimage#service-launch-fails for more information.\"\n },\n {\n \"code\": \"AciDeploymentFailed\",\n \"message\": \"Your container application crashed. Please follow the steps to debug:\n\t1. From the AML SDK, you can run print(service.get_logs()) if you have service object to fetch the logs. Please refer to https://aka.ms/debugimage#dockerlog for more information.\n\t2. If your container application crashed. This may be caused by errors in your scoring file's init() function. You can try debugging locally first. Please refer to https://aka.ms/debugimage#debug-locally for more information.\n\t3. You can also interactively debug your scoring file locally. Please refer to https://docs.microsoft.com/azure/machine-learning/how-to-debug-visual-studio-code#debug-and-troubleshoot-deployments for more information.\n\t4. View the diagnostic events to check status of container, it may help you to debug the issue.\n\"RestartCount\": 3\n\"CurrentState\": {\"state\":\"Waiting\",\"startTime\":null,\"exitCode\":null,\"finishTime\":null,\"detailStatus\":\"CrashLoopBackOff: Back-off restarting failed\"}\n\"PreviousState\": {\"state\":\"Terminated\",\"startTime\":\"2023-01-29T22:08:22.99Z\",\"exitCode\":111,\"finishTime\":\"2023-01-29T22:08:33.053Z\",\"detailStatus\":\"Error\"}\n\"Events\": null\n\"\n }\n ]\n}" } } -
romungi-MSFT 42,206 Reputation points Microsoft Employee
2023-02-03T09:09:01.3466667+00:00 Is your score.py file same as the one in your initial post? That is the following:
import json import joblib from azureml.core.model import Model def init(): global model model_path = Model.get_model_path(model_name = 'model_estimator_500.pkl') model = joblib.load(model_path) def run(raw_data): try: data = np.array(json.loads(raw_data)['data']) y_hat = model.predict(data) return y_hat.to_list() except Exception as e: result = str(e)Sample for reference.
I think you could check the following:
- Ensure your scoring script does not contain the rest of the deployment steps as seen in the original post.
- Is the model available that you are trying to load in the script?
- Try a local deployment if the above are correct, This will confirm an issue with ACI deployments. ACI deployments are not recommended because of its limitations. It is highly advised to debug locally before deploying to the web service on ACI.
Sign in to comment