(UserError) Error when parsing request; unable to deserialize request body

Question

Hi, getting this error when i run an azureml experiment with custom_docker_image (basegpu image of mcr) - can anybody help me understand this? Have tested this in local compute and it works, not sure why this does not work on a training cluster vm?

   azureml._restclient.exceptions.ServiceException: ServiceException:
        Code: 400
        Message: (UserError) Error when parsing request; unable to deserialize request body
        Details:

        Headers: {
            "Date": "Mon, 08 Jun 2020 11:03:52 GMT",
            "Content-Type": "application/json; charset=utf-8",
            "Transfer-Encoding": "chunked",
            "Connection": "keep-alive",
            "Request-Context": "appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b",
            "x-ms-response-type": "error",
            "Strict-Transport-Security": "max-age=15724800; includeSubDomains; preload"
        }
        InnerException: {
        "additional_properties": {},
        "error": {
            "additional_properties": {},
            "code": "UserError",
            "message": "Error when parsing request; unable to deserialize request body",
            "details_uri": null,
            "target": null,
            "details": [],
            "inner_error": null,
            "debug_info": null,
            "message_format": null,
            "message_parameters": null,
            "reference_code": null
        },
        "correlation": {
            "operation": "e96d6285280f5849a4a5e3f172d65d36",
            "request": "1beee8ecb7180147"
        },
        "environment": "westeurope",
        "location": "westeurope",
        "time": {}
    }

Answer 1

My bad, after giving it some days and looking at the code, I noticed i had forgotten to add the parameters for the estimator configuration. Here is the estimator configuration that works for me:

estimator = Estimator(source_directory=experiment_folder,
                      compute_target=compute_target,
                      script_params=script_params,
                      entry_script='rps_efn_b0.py',
                      node_count=1,        
                      conda_packages=['ipykernel'],
                      pip_packages = ['azureml-sdk',
                                      'pyarrow',
                                      'pyspark',
                                      'azureml-mlflow',
                                      'joblib',
                                      'matplotlib',
                                      'Pillow',
                                      'tensorflow==2.2',
                                      'tensorflow-datasets',
                                      'tensorflow-hub',
                                      'azureml-defaults',
                                      'azureml-dataprep[fuse,pandas]'],
                      custom_docker_image='mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.1-cudnn7-ubuntu18.04')

Answer 2

@Sadegh Hosseinpoor Apologies for not responding on the previous thread as we are having platform issues on providing response to some threads.

In this case having access to the backend details of your experiment run could help to suggest what changes are required in your current experiment setup. This can be addressed by our support team if you can raise a support ticket against this resource from the azure portal.

Meanwhile, I have tried to lookup similar error messages faced by other users and based on your scenario are you installing any package in your environment with package name and version which is failing to read the request json? I found one user who corrected this error during environment setup by correcting the request for spark package installation in their environment.

Before:

environment.spark.packages=[SparkPackage(group='com.microsoft.azure', artifact='azure:azure-sqldb-spark', version='1.0.2')]  

After:

environment.spark.packages=[{'group':'com.microsoft.azure', 'artifact': 'azure-sqldb-spark', 'version':'1.0.2'}]  

Is there any similar requests in your case?