py4j.security.Py4JSecurityException Databricks

Question

I was following the Databricks Academy "New Capabilities Overview: Feature Store" module. However when I try to run the code in the example notebook I get a security exception as explained below.

 

When I try to run the example notebook "01-Populate a feature store" I get an error while creating a feature table, I get a

py4j.security.Py4JSecurityException: Method public java.lang.String org.apache.spark.api.java.JavaSparkContext.getLocalProperty(java.lang.String) is not whitelisted on class class org.apache.spark.api.java.JavaSparkContext

 

based on google search it is due to High Concurrency cluster. It can be resolved when a standard cluster ML cluster was used. but i do not have option to change the cluster type, can you please let me know how can i fix this

Answer 1

Hello @Sharukh Kundagol - Thanks for the question and using MS Q&A platform.

The error you are seeing is due to the fact that the method getLocalProperty is not whitelisted on the JavaSparkContext class. This is a known issue with High Concurrency clusters in Databricks. One way to resolve this issue is to use a Standard cluster instead of a High Concurrency cluster. However, if you do not have the option to change the cluster type, you can try the following workaround:

1. Open the notebook where you are seeing the error.
2. Click on the "Edit" button to edit the notebook.
3. Add the following line of code at the beginning of the notebook:

spark.conf.set("spark.driver.extraJavaOptions", "-Dio.netty.tryReflectionSetAccessible=true")

1. Save the notebook and try running it again.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.