How to fix ERROR: "message": "An unexpected error occurred in scoring script. Check the logs for more info."

Martin Orator 20

I created autogenerated ML predictive model in Azure Machine Learning studio and trained the model on tabular data in CSV format.

Succeeded

Then I directly deployed model to the Endpoint

Succeeded.

But when I try to test the model by sending request via API I am getting error:

I am using Python in VSCode to send the request - that works well.

The request failed with status code: 424 server: azureml-frontdoor date: Wed, 27 Mar 2024 19:42:44 GMT content-type: application/json content-length: 92 x-ms-run-function-failed: True x-ms-server-version: azmlinfsrv/0.8.4.2 x-ms-request-id: 8b508eaa-914f-4c26-8764-1559cbbc5543 x-request-id: 8b508eaa-914f-4c26-8764-1559cbbc5543 ms-azureml-model-error-reason: model_error ms-azureml-model-error-statuscode: 500 azureml-model-deployment: blueflower-1 connection: close {"message": "An unexpected error occurred in scoring script. Check the logs for more info."}

I suspect I am sending request in wrong format. I can not find anywhere what is expected input to trigger successfuly the expected output from the model so I am trying to send same data as was the training dataset, just converted to JSON format. It seems there is no real sample input and output generated with the model so I am a bit blind as I can not find real exact expected format incl all details. Also can not find any detailed logs to understand what exactly is failing in scoring script besides the run function. Thank you.

This is the only thing I find on the endpoint tabs to test it, but when I run the test as is I am getting following error directly in the gui

: An unexpected error occurred in scoring script. Check the logs for more info. ID pro sledování : 546ccea3-e3a0-417e-b8a8-37a8e8912191

User's image

{
  "input_data": {}
}

Martin Orator 20

Thanks a lot @dupammi ,

this answer it is very useful in order to get the logs that I found. I also used the structure of JSON from your sample, just used my own data and columns same as in training dataset.

I am getting this exception:

Encountered Exception: Traceback (most recent call last):
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py", line 132, in invoke_run
    run_output = self._wrapped_user_run(**run_parameters, request_headers=dict(request.headers))
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py", line 156, in <lambda>
    self._wrapped_user_run = lambda request_headers, **kwargs: self._user_run(**kwargs)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/inference_schema/schema_decorators.py", line 64, in decorator_input
    return user_run(*args, **kwargs)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/inference_schema/schema_decorators.py", line 92, in decorator_input
    return user_run(*args, **kwargs)
  File "/var/mlflow_resources/mlflow_score_script.py", line 215, in run
    return _get_jsonable_obj(model.predict(input), pandas_orient="records")
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/mlflow/pyfunc/__init__.py", line 492, in predict
    return _predict()
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/mlflow/pyfunc/__init__.py", line 480, in _predict
    return self._predict_fn(data)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml/training/tabular/models/mlflow/timeseries/__init__.py", line 345, in predict
    y_res, _ = self.forecasting_model.forecast(X_pred=X_pred, y_pred=y_pred, ignore_data_errors=True)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper_base.py", line 576, in forecast
    forecast, forecast_df = self._scenario_forecast(
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper_base.py", line 449, in _scenario_forecast
    forecast, forecast_df = self._scenario_forecast_automl(
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper_base.py", line 536, in _scenario_forecast_automl
    forecast, forecast_df = self._pipeline_forecast_internal(
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper.py", line 386, in _pipeline_forecast_internal
    self._prepare_prediction_data_for_forecast(Xy_pred_in, ignore_data_errors=ignore_data_errors)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper_base.py", line 1799, in _prepare_prediction_data_for_forecast
    forecast_first_irow = tsds_one.time_index.get_loc(last_known_y_date) + 1

TypeError: unsupported operand type(s) for +: 'slice' and 'int'

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml_inference_server_http/server/routes.py", line 222, in handle_score
    timed_result = main_blueprint.user_script.invoke_run(request, timeout_ms=config.scoring_timeout)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py", line 139, in invoke_run
    raise UserScriptException(ex) from ex
azureml_inference_server_http.server.user_script.UserScriptException: Caught an unhandled exception from the user script

and this is my sampled request that I put to the same shape as your advice:

data = {
        "input_data": {
            "columns": [
                "date",
                "price",
                "volume",
                "score1",
                "score2",
                "score3",
                "count",
                "mean",
                "sum",
                "Cdate",
                "Cvol1",
                "Cvol2",
                "Cvol3",
                "Cvol4",
                "Cvol5",
                "Xdate",
                "Xvol1",
                "Xvol2",
                "Xvol3",
                "Xvol4",
                "Xvol5"
            ],
            "index":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20],
            "data":[
                ["2024-03-27 00:00:00",70685.0,1739.0,-0.8378,0.3148,0.263,15,-0.0867,-1.2999,"2024-03-27 00:00:00",64681.45,66399.88,63421.0,63856.74,287.4128,"2024-03-27 00:00:00",0.1277,0.1292,0.1226,0.1232,720866.0]
            ]
            },
        "Global_parameters": {
        "method": "predict_proba"
            }
        }

Please can you see any obvious issue? I am not sure how can I get to the model code to make more troubleshooting. Data structure and formats are the same as in training data.


TypeError: unsupported operand type(s) for +: 'slice' and 'int'

This indicates there might be some issue with some data type or structure of the input data, but without detailed check of the code that is processing I can be just guessing.

After some testing now I am sure the error is tied to the datetime data somehow. Not sure what format the model is really expecting.

dupammi 6,390 Reputation points Microsoft Vendor

2024-03-28T11:18:25.3733333+00:00

Hi @Martin Orator

Thank you for the details of the error message.

The error message "TypeError: unsupported operand type(s) for +: 'slice' and 'int'" suggests that there may be an issue with the date format or data type. The cause of the exception needs to be investigated and resolved in the user script, or in the format of input date and int values were passed.

To debug this issue, I recommend checking the date format in the input data to ensure it matches the format used during training. If the date format is incorrect, the model may not be able to process the data correctly. Additionally, it's important to check the data types of all the numerical values in the dataset to ensure they are compatible.

Additionally, you can check the swagger for datatype checks and artifacts to see whether slicing is being done really or is it considering the "time" part's ":" symbol in the date field as slicing. Try by just passing the "YYYY-MM-DD".

I hope this helps. Thank you.

Martin Orator 20

Thanks a lot for your answers @dupammi

Whatever I try it seems dates are not processed properly, it seems like closest to meaningful response I was in this case when I used format 2024-3-27 without "" and after removing 0 before number of month:

"data":[
                [2024-3-27,70685.0,1739.0,-0.8378,0.3148,0.263,15,-0.0867,-1.2999,2024-3-27,64681.45,66399.88,63421.0,63856.74,287.4128,2024-3-27,0.1277,0.1292,0.1226,0.1232,720866.0]
            ]

When I got this error:

Message: Input prediction data X_pred or input forecast_destination contains dates prior to the latest date in the training data. Please remove prediction rows with datetimes in the training date range or adjust the forecast_destination date.
	InnerException: None
	ErrorResponse 
{
    "error": {
        "code": "UserError",
        "message": "Input prediction data X_pred or input forecast_destination contains dates prior to the latest date in the training data. Please remove prediction rows with datetimes in the training date range or adjust the forecast_destination date.",
        "target": "X_pred",
        "inner_error": {
            "code": "BadData"
        },
        "reference_code": "9aa1211b-115a-474c-b522-c1170ae8c8b3"
    }
}

But for sure this is not before any training data as it ends on 2024-3-23.

dupammi 6,390 Reputation points Microsoft Vendor

2024-03-28T14:17:11.9466667+00:00

Hi @Martin Orator

Thank you for giving more details.

For the purpose of debugging, I tried the endpoint and payload from POSTMAN tool and got the status code 200, which is success. see below. :

As we can observe, it seems that Azure endpoint is not correctly processing the date format sent in the JSON payload.

In the Postman request, the "WeekStarting" field is sent as a string in the format "1996-11-28", which represents November 28, 1996. However, in the Azure endpoint test, the date is represented as a Unix timestamp: 849139200000, which corresponds to the same date but in milliseconds since the Unix epoch.

To resolve this issue, you need to ensure that the Azure endpoint can correctly parse the date format "1996-11-28". You may need to review your scoring script or any preprocessing steps to handle date formats correctly. If necessary, update your scoring script to handle different date formats or convert them to a consistent format before processing. Check for any error messages or logs in the Azure endpoint logs that might provide more information about why the date parsing is failing.

By ensuring consistency in the date format between your JSON payload and your scoring script, you should be able to resolve the issue and successfully process the requests in your Azure endpoint.

I hope this helps us understand.

Kindly let me know if this answers your question and if I can convert the "comment" to "Answer" and if you are okay, please accept the converted answer so that it will be useful to other community members searching for similar issues.

Thank you for your patience throughout the issue.
Martin Orator 20 Reputation points

2024-03-28T14:30:33.0366667+00:00

Thanks a lot for the helpful answers. Let me try to find a scoring script to check how it handles date and make necessary changes to fix the problem in my case. If I will find some interesting detail I'll share it in this thread.
Martin Orator 20 Reputation points

2024-03-28T14:38:33.59+00:00

@dupammi You can convert this to answered! Thanks a lot.

Martin Orator 20

@dupammi I already marked that answered as I think your advices helps a lot. But excuse me - I suspect I can not fix it from my end in any way.

In the meantime I retrained the model and deployed again on the endpoint.

And situation repeats with one small difference, newly I am getting one more error that looks like the data got deeper into processing, but it hits another issue with '<' sign. Still this is related to the date field of the request only.

To me it seems like this will require quick fix from MS side as it seems like a bug in the response data processing for 'date'/'datetime' datatype. It is not possible to change anything in the request by at least I tried many options and found none that would be successfully processed. Date format in training data is correct and consistent %YY-%mm-%dd with what I am using in the request.

I suspect this time it doesn't have solution and it requires fix from Microsoft.

2024-03-29 10:06:03,933 E [74] azmlinfsrv - Encountered Exception: Traceback (most recent call last):
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/azureml_inference_server_http/server/user_script.py", line 132, in invoke_run
    run_output = self._wrapped_user_run(**run_parameters, request_headers=dict(request.headers))
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/azureml_inference_server_http/server/user_script.py", line 156, in <lambda>
    self._wrapped_user_run = lambda request_headers, **kwargs: self._user_run(**kwargs)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/inference_schema/schema_decorators.py", line 64, in decorator_input
    return user_run(*args, **kwargs)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/inference_schema/schema_decorators.py", line 92, in decorator_input
    return user_run(*args, **kwargs)
  File "/var/mlflow_resources/mlflow_score_script.py", line 215, in run
    return _get_jsonable_obj(model.predict(input), pandas_orient="records")
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/mlflow/pyfunc/__init__.py", line 513, in predict
    return self._predict_fn(data)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/azureml/training/tabular/models/mlflow/timeseries/__init__.py", line 345, in predict
    y_res, _ = self.forecasting_model.forecast(X_pred=X_pred, y_pred=y_pred, ignore_data_errors=True)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper_base.py", line 585, in forecast
    forecast, forecast_df = self._scenario_forecast(
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper_base.py", line 454, in _scenario_forecast
    forecast, forecast_df = self._scenario_forecast_automl(
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper_base.py", line 541, in _scenario_forecast_automl
    forecast, forecast_df = self._pipeline_forecast_internal(
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper.py", line 390, in _pipeline_forecast_internal
    self._prepare_prediction_data_for_forecast(Xy_pred_in, ignore_data_errors=ignore_data_errors)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/azureml/training/tabular/models/forecasting_pipeline_wrapper_base.py", line 1815, in _prepare_prediction_data_for_forecast
    tsds_one.data.iloc[forecast_first_irow:, missing_target_icol] = not_imputed_val
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/pandas/core/indexing.py", line 723, in __setitem__
    iloc._setitem_with_indexer(indexer, value, self.name)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/pandas/core/indexing.py", line 1730, in _setitem_with_indexer
    self._setitem_with_indexer_split_path(indexer, value, name)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/pandas/core/indexing.py", line 1758, in _setitem_with_indexer_split_path
    lplane_indexer = length_of_indexer(pi, self.obj.index)
  File "/azureml-envs/azureml-automl-dnn-gpu/lib/python3.9/site-packages/pandas/core/indexers.py", line 336, in length_of_indexer
    elif start < 0:
TypeError: '<' not supported between instances of 'slice' and 'int'

dupammi 6,390 Reputation points Microsoft Vendor

2024-03-29T10:18:16.16+00:00

Hi @Martin Orator

Thank you for your follow up question.

When deploying a real-time or batch inference endpoint in Azure Machine Learning, you can choose to have the environment and scoring script auto-generated. However, note that the auto-generated scoring script is not automatically saved to a specific location. You will need to manually save the scoring script to a location of your choice for future reference.

To locate the scoring script, you can download it from the 'Outputs' tab of the specific AutoML job that you used to generate the endpoint. Once you have the scoring script, you can open it in Jupyter Notebook inside Azure ML Studio and make any necessary updates.

Once you have made the updates, you can save the scoring script and use it to deploy your endpoint.

See this: For more information on how to deploy real-time or batch inference endpoints & how-to-deploy-online-endpoints

I hope this helps. Thank you.
dupammi 6,390 Reputation points Microsoft Vendor

2024-03-29T10:49:02.6266667+00:00

Hi @Martin Orator

Apologies for the inconvenience you experienced.

I request you to raise a support case through Azure portal.

I hope you understand. Thank you!
Martin Orator 20 Reputation points

2024-03-29T12:29:58.53+00:00
Hi @dupammi I really appreciate your help anyway. One last observation. When I trained other model and used regression ( CPU) model instead of predictive (GPU) model and deployed on CPU endpoint. Example of the imput data was autogenerated properly:

{ "input_data": { "columns": [ "date", "price", "volume", "score1", "score2", "score3", "count", "mean", "sum", "Cvolume1", "Cvolume1", "Cvolume3", "Cvolume4", "Cvolume5", "Xvolume1", "Xvolume2", "Xvolume3", "Xvolume4" ], "index": [], "data": [] } }

That proves really it is MS issue for predictive GPU models endpoints that need to be fixed. CPU regression model endpoint seems to behave fine. I would like to make a ticket on Azure support, but I have just basic subscription so I am afraid MS will not find this issue out. Thanks a lot.
dupammi 6,390 Reputation points Microsoft Vendor

2024-03-29T12:55:31.47+00:00

Hi @Martin Orator

It is possible that you are experiencing some technical glitch. To assist you with this, we recommend that you upload a document mentioning your issues with screenshots, using the feedback form available on the Azure Machine Learning. See below.

Our technical team will review the information you provide and work to resolve the issue.

Thank you for your cooperation and patience throughout the repros.

Accepted answer

dupammi 6,390 Reputation points Microsoft Vendor

2024-03-28T00:33:15.7666667+00:00

Hi @Martin Orator

Thank you for using the Microsoft Q&A forum.

To resolve the "HttpResponseError" with message "An unexpected error occurred in scoring script. Check the logs for more info." when invoking an online endpoint in Azure ML, I suggest checking the logs for more information about the error, verifying that the input data format matches the format used during training. Verify that the online endpoint is running and deployed correctly.

If the problem still persists, I suggest you test the online endpoint from the Azure Machine Learning studio portal or POSTMAN tool, to verify if the endpoint is running correctly. Please also check the error logs from the portal, once the endpoint was hit in the portal with the required JSON, parametric values.

I tried to repro an error randomly by removing one required parameter from my online endpoint JSON and received below error. This repro I have done just to show where to check the logs. Please see below:

By passing the required number of parameter values, I was able to successfully get the response from the online endpoint. Please see below

Please refer to Azure Machine Learning Studio documentation, and try providing a sample input format that matches the format used during training.

I hope you understand.

Thank you.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

How to fix ERROR: "message": "An unexpected error occurred in scoring script. Check the logs for more info."

0 additional answers