py4j.security.Py4JSecurityException

Question

py4j.security.Py4JSecurityException

Ahuja, Rachit 0

Hello I am trying to run spark XGBoostRegression model on Databricks cluster with Databricks runtime: 14.3 LTS.

I am getting the following error:

Py4JError: An error occurred while calling o547.resourceProfileManager. Trace: py4j.security.Py4JSecurityException: Method public org.apache.spark.resource.ResourceProfileManager org.apache.spark.SparkContext.resourceProfileManager() is not whitelisted on class class org.apache.spark.SparkContext at py4j.security.WhitelistingPy4JSecurityManager.checkCall(WhitelistingPy4JSecurityManager.java:473) at py4j.Gateway.invoke(Gateway.java:305) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750)

Here is my cluster configuration JSON: { "cluster_id": "0412-193940-roa64zgt", "creator_user_name": "b90f2c51-c49c-4f97-95a9-d58c6c1e8fc8", "spark_context_id": 5504105953918341000, "driver_healthy": true, "cluster_name": "DsShared", "spark_version": "14.3.x-cpu-ml-scala2.12", "spark_conf": { "spark.databricks.delta.preview.enabled": "true" }, "azure_attributes": { "first_on_demand": 1, "availability": "ON_DEMAND_AZURE", "spot_bid_max_price": -1 }, "node_type_id": "Standard_D4s_v5", "driver_node_type_id": "Standard_D4s_v5", "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "autotermination_minutes": 20, "enable_elastic_disk": true, "disk_spec": {}, "cluster_source": "UI", "enable_local_disk_encryption": false, "instance_source": { "node_type_id": "Standard_D4s_v5" }, "driver_instance_source": { "node_type_id": "Standard_D4s_v5" }, "data_security_mode": "LEGACY_PASSTHROUGH", "runtime_engine": "STANDARD", "effective_spark_version": "14.3.x-cpu-ml-scala2.12", "autoscale": { "min_workers": 1, "max_workers": 3, "target_workers": 1 }, "default_tags": { "Vendor": "Databricks", "Creator": "b90f2c51-c49c-4f97-95a9-d58c6c1e8fc8", "ClusterName": "DsShared", "ClusterId": "0412-193940-roa64zgt", }, "termination_reason": { "code": "INACTIVITY", "type": "SUCCESS", "parameters": { "inactivity_duration_min": "20" } }, "pinned_by_user_name": "4460559388477034", "init_scripts_safe_mode": false, "spec": { "cluster_name": "DsShared", "spark_version": "14.3.x-cpu-ml-scala2.12", "spark_conf": { "spark.databricks.delta.preview.enabled": "true" }, "azure_attributes": { "first_on_demand": 1, "availability": "ON_DEMAND_AZURE", "spot_bid_max_price": -1 }, "node_type_id": "Standard_D4s_v5", "driver_node_type_id": "Standard_D4s_v5", "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "autotermination_minutes": 20, "enable_elastic_disk": true, "cluster_source": "UI", "data_security_mode": "LEGACY_PASSTHROUGH", "runtime_engine": "STANDARD", "autoscale": { "min_workers": 1, "max_workers": 3 } } }

Any help would be highly appreciated.

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2024-05-06T17:59:40.56+00:00

Hello Ahuja, Rachit,

Welcome to the Microsoft Q&A forum.

Can you please let us know the notebook commands that you are using here?
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2024-05-10T23:05:39.4066667+00:00

Hello Ahuja, Rachit,

I am checking to see if you had a chance to look into my above response.
PRADEEPCHEEKATLA 91,866 Reputation points

2024-05-20T08:05:29.2833333+00:00
@Ahuja, Rachit - It seems like the error is related to Py4JSecurityException. This error occurs when a method is not whitelisted on the class. In your case, the method "public org.apache.spark.resource.ResourceProfileManager org.apache.spark.SparkContext.resourceProfileManager()" is not whitelisted on class org.apache.spark.SparkContext.

To resolve this issue, you can try adding the following configuration to your cluster:

JavaOptions": "-Dspark.driver.extraJavaOptions=-Dspark.security.credentials.whitelist=org.apache.spark.resource.ResourceProfileManager", "spark.executor.extraJavaOptions": "-Dspark.executor.extraJavaOptions=-Dspark.security.credentials.whitelist=org.apache.spark.resource.ResourceProfileManager"

This configuration will whitelist the ResourceProfileManager class and should resolve the Py4JSecurityException error.

Let me know if this helps.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2024-05-21T21:04:06.7066667+00:00

Thanks PRADEEPCHEEKATLA-MSFT

Hello Ahuja, Rachit,

Please check the above response and let us know if you have any further questions.