spark catalog whitelisted for shared clusters with unityCatalog

Chandra Doredla 30

I have recently initiated unity catalog for my workspace with the use of shared clusters, now I want to check if a data base exists using the spark.catalog.databaseExists() but the py4j error says the class is not whitelisted.

The issue is when using the shared or single purpose cluster but not with the old clusters.

Thank you

KranthiPakala-MSFT 46,427 Reputation points Microsoft Employee

2023-05-15T23:03:44.2566667+00:00

Hi @Chandra Doredla ,

Thanks for using Microsoft Q&A forum and posting your query.

To better assist on this query, could you please share complete error message that you are seeing? Also please confirm, what is the Databricks runtime being used for those clusters that are having issue? Also, would you please compare the cluster configuration for the once that is working and the ones that are not working to identify if you can see any additional configuration for whitelisting the catalog class?

Thanks
Chandra Doredla 30 Reputation points

2023-05-16T05:05:51.9133333+00:00

py4j.security.Py4JSecurityException: Method public boolean org.apache.spark.sql.internal.CatalogImpl.databaseExists(java.lang.String) is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

This is the full error message.

Cluster details:

AccessMode: Shared

RuntimeVersion: 13.0.x-scala2.12

This error happens to occur when accessing hive metastore or unity catalog using the catalog class of spark
Chandra Doredla 30 Reputation points

2023-05-16T05:08:34.9766667+00:00

hi @KranthiPakala-MSFT

py4j.security.Py4JSecurityException: Method public boolean org.apache.spark.sql.internal.CatalogImpl.databaseExists(java.lang.String) is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

This is the error which I encounter when using the catalog class to check if a databse exists in any metastore (hivemetastore or using unity catalog) Cluster Details: Accessmode: Shared Runtime: 13.0.x-scala2.12
KranthiPakala-MSFT 46,427 Reputation points Microsoft Employee

2023-05-17T19:49:13.95+00:00

Hi @Chandra Doredla ,

Thanks for the additional details.

This happens because Unity Catalog is supported on clusters where isolation is turned on (shared and single). One way to overcome this error is to disable the security feature, spark.databricks.pyspark.enablePy4JSecurity false , but this option is not recommended. Alternatively, you can create a whitelist of classes and specify it in spark config, that way the security feature is turned on and a specific not safe class is allowed to execute.

Hope this info helps.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Chandra Doredla 30 Reputation points

2023-05-17T23:11:59.8933333+00:00

hi @KranthiPakala-MSFT

That really helps, Is there any present documentation available for whitelisting the required classes for spark.

Thank you

Regards

Chandra
Lin, Kevin 20 Reputation points

2023-06-05T17:45:10.0266667+00:00

Is there any documentation on how to create a whitelist of classes in the spark config? @KranthiPakala-MSFT @Chandra Doredla @Chandra Doredla
Andrew Haigh 5 Reputation points

2023-11-10T14:45:45.47+00:00

I'm running into the same issue - I need to whitelist a class in my shared cluster but can't find ANY info on how to do so

Any hints, suggestions, directions, or out-right lies would be welcome

Cheers

-=A=-

.
Gurvinder 0 Reputation points

2023-12-15T17:54:24.68+00:00

If anyone has documentation on whitelisting a class in spark, please share.

Running into the following issue when using shared cluster (for UnityCatalog) in Databricks.

py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.common.rpc.CommandContext

Thanks in advance for your help.
Syed Akbar 0 Reputation points

2024-02-28T17:09:21.45+00:00

I am also having same whitelisting error in unity catalog enabled workspace using shared cluster. Even I am using now 14.3 Runtime version but still error is coming. Please provide assistance as it is becoming issue in implementing the code in unity catalog workspace.

Share via

spark catalog whitelisted for shared clusters with unityCatalog