Spark Catalog Whitelisted for Shared Clusters & Table Access Control

Lin, Kevin 20

I have recently enabled Unity Catalog for my cluster that is on 13.0 Runtime Version, and was trying to create a database that required a spark dataframe. However, whenever it reached the spark parameter it would then try to perform a rdd() function on it causing this error to appear:

py4j.security.Py4JSecurityException: Method public org.apache.spark.rdd.RDD org.apache.spark.api.java.JavaRDD.rdd() is not whitelisted on class class org.apache.spark.api.java.JavaRDD

I've looked into this issues, and many discussion posts stated that I could overcome this error by adding this line in the spark config:
spark.databricks.pyspark.enablePy4JSecurity false

I tried that and this is the error I got:

User's image

Therefore, I've been trying to create a whitelist of classes that I want to specify in the spark config but am not sure on how to do so as I couldn't find any documentation on it. Would appreciate the help if anyone has figured this help.

Thank you

ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2023-06-07T08:47:10.8633333+00:00
Hi Lin, Kevin,

Thank you for posting query in Microsoft Q&A Platform.

It seems like you are facing a security issue with Py4J in your Spark application. The error message indicates that the JavaRDD.rdd() method is not whitelisted, which means that it is not allowed to be called from your PySpark code.

To resolve this issue, you can try the following steps:

Instead of disabling Py4J security altogether, you can try to whitelist the specific classes and methods that you need to use in your PySpark code. To do this, you can set the spark.jvm.class.allowlist configuration property in your Spark configuration. For example, you can set it to org.apache.spark.api.java.JavaRDD to allow the JavaRDD class and its methods to be used in your PySpark code.

If you are using Databricks, you can try setting the spark.databricks.pyspark.enablePy4JSecurity configuration property to true and then use the databricks.koalas.config.set_option method to set the spark.jvm.class.allowlist property. For example:

import databricks.koalas.config as kc kc.set_option("spark.jvm.class.allowlist", "org.apache.spark.api.java.JavaRDD")

This will allow you to use the JavaRDD class and its methods in your PySpark code.

If none of the above solutions work, you can try upgrading to a newer version of Spark that has better Py4J security features.

I hope this helps you resolve your issue. Let me know how it goes and any further queries. Thank you.
Lin, Kevin 20 Reputation points

2023-06-07T14:33:48.93+00:00
Thanks for your response , I've tried your both of your approaches but to no avail. I've tried adding it to my spark config and also tried it with databricks.koalas route. This is the error I got using koalas:

OptionError: "No such option: 'spark.jvm.class.allowlist'. Available options are [display.max_rows, compute.max_rows, compute.shortcut_limit, compute.ops_on_diff_frames, compute.default_index_type, compute.ordered_head, plotting.max_rows, plotting.sample_ratio, plotting.backend]"

For clarification, here is my code that was done within my databricks notebook:

!pip install koalas import databricks.koalas.config as kc kc.set_option("spark.jvm.class.allowlist", "org.apache.spark.api.java.JavaRDD")

I've had to change my python version to 3.9 for this to work. Also I've stated previously that I'm not able to use this line in my spark config spark.databricks.pyspark.enablePy4JSecurity due to shared access mode. Is there any other way to whitelist?

Thank you