UDF PySpark With Shared Access Mode Cluster

Question

UDF PySpark With Shared Access Mode Cluster

Gianluca Evangelisti 0

Hello,
we use some UDF PySpark in Databricks with a cluster in Shared Access Mode.

We get a SparkOutOfMemoryError error (see the attachment).

Is possible use UDF with a Unity Catalog cluster, or there are any problems?

OutOfMemory.png

Thanks in advance
Gianluca

Venkat Reddy Navari 1,780 Reputation points Microsoft External Staff Moderator

2025-05-06T13:47:20.1133333+00:00
Hi @Gianluca Evangelisti
It sounds like you're running into a SparkOutOfMemoryError while using UDFs with PySpark in Databricks on a cluster in Shared Access Mode. This can be a bit tricky, but here are a few things you can try to resolve the issue:

Increase the Executor Memory: If your UDFs are processing large volumes of data, consider increasing the executor memory for your cluster. You can do this in the Databricks workspace by adjusting the cluster configuration settings.

Optimize UDFs: If the UDF itself is inefficient or is processing more data than necessary, that might lead to memory issues. Review the UDF logic to ensure it's optimized for performance.

Data Partitioning: Ensure your data is properly partitioned. If you're working with large datasets, repartitioning can help distribute the workload more evenly across the executors.

Use Unity Catalog: Regarding your question about using UDFs in a Unity Catalog cluster—yes, you can use UDFs with Unity Catalog. However, check the limitations and requirements for UDFs when using Unity Catalog as there might be specific constraints to follow. For example, as per the documentation, Hive UDFs are not supported, and certain Spark functions might also have limitations.

Databricks Runtime Version: Ensure your cluster is using a Databricks Runtime version that supports the features you need. For example, Python scalar UDFs require Databricks Runtime 13.3 LTS or above.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Venkat Reddy Navari 1,780 Reputation points Microsoft External Staff Moderator

2025-05-07T09:57:02.2066667+00:00

@Gianluca Evangelisti We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Venkat Reddy Navari 1,780 Reputation points Microsoft External Staff Moderator

2025-05-08T10:21:30.2066667+00:00

@Gianluca Evangelisti We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Your answer

Venkat Reddy Navari 1,780 Reputation points Microsoft External Staff Moderator

2025-05-06T13:47:20.1133333+00:00

Hi @Gianluca Evangelisti
It sounds like you're running into a SparkOutOfMemoryError while using UDFs with PySpark in Databricks on a cluster in Shared Access Mode. This can be a bit tricky, but here are a few things you can try to resolve the issue:

Increase the Executor Memory: If your UDFs are processing large volumes of data, consider increasing the executor memory for your cluster. You can do this in the Databricks workspace by adjusting the cluster configuration settings.

Optimize UDFs: If the UDF itself is inefficient or is processing more data than necessary, that might lead to memory issues. Review the UDF logic to ensure it's optimized for performance.

Data Partitioning: Ensure your data is properly partitioned. If you're working with large datasets, repartitioning can help distribute the workload more evenly across the executors.

Use Unity Catalog: Regarding your question about using UDFs in a Unity Catalog cluster—yes, you can use UDFs with Unity Catalog. However, check the limitations and requirements for UDFs when using Unity Catalog as there might be specific constraints to follow. For example, as per the documentation, Hive UDFs are not supported, and certain Spark functions might also have limitations.

Databricks Runtime Version: Ensure your cluster is using a Databricks Runtime version that supports the features you need. For example, Python scalar UDFs require Databricks Runtime 13.3 LTS or above.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Venkat Reddy Navari 1,780 Reputation points Microsoft External Staff Moderator

2025-05-07T09:57:02.2066667+00:00

@Gianluca Evangelisti We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Venkat Reddy Navari 1,780 Reputation points Microsoft External Staff Moderator

2025-05-08T10:21:30.2066667+00:00

@Gianluca Evangelisti We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

UDF PySpark With Shared Access Mode Cluster

Your answer