Hi @Lakshmi Narayana Sarma Bhamidipati
Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!
The error you are facing is because, when utilizing a High Concurrency cluster and enabling credential pass through, this problem occurs with various library operations. When you visit a method that Azure Databricks hasn't expressly designated as safe for Azure Data Lake Storage credential passthrough clusters, this issue is raised.
For more details, please refer: https://learn.microsoft.com/en-us/azure/databricks/archive/credential-passthrough/adls-passthrough#passthrough-troubleshooting
Below are the workarounds that you may consider for resolving your query.
- Use a different cluster mode as a workaround if it describes your situation.
- Update
spark.databricks.pyspark.enableProcessIsolation
tofalse
for this your cluster needs to be with no isolation shared access mode - Update
spark.databricks.pyspark.enablePy4JSecurity
tofalse
- Use standard clusters in your case as a workaround.
Hope this helps. Do let us know if you have any further queries.
If this answers your query, do click `Accept Answer`
and `Yes`
for was this answer helpful. And, if you have any further query do let us know.