Scala and Java user-defined functions (UDFs) in Unity Catalog

This page describes how to create Scala and Java user-defined functions (UDFs), register them in Unity Catalog, and share them across compute environments. Unity Catalog UDFs let you reuse existing JVM logic with Unity Catalog governance and access controls.

Unlike session-scoped Scala UDFs that are limited to a single notebook or cluster, registered UDFs in Unity Catalog are:

Governed: Managed with Unity Catalog permissions and access controls.
Reusable: Shared across teams, notebooks, jobs, and SQL warehouses.
Discoverable: Visible in Catalog Explorer and system tables.
Isolated: Run in sandboxes with a one-time cold-start cost per session. Subsequent calls are fast.

Requirements

Your workspace must be enabled for Unity Catalog. The following additional requirements apply.

Compute: All compute types are supported, including serverless notebooks and jobs, SQL warehouses, and Spark Declarative Pipelines on Lakeflow. Classic compute requires Databricks Runtime 18.2 or above. On serverless compute and SQL warehouses, the UDF definition must specify Environment Version 4 or above in the environment_version field. This requirement applies to the UDF definition, not to the calling notebook or job. See Serverless environment versions.

Development:

Scala: 2.13.16. Scala 2.12 is not supported.
JDK: 17.
Packaging: A fat JAR containing all third-party dependencies used by the UDF.

Permissions:

Create a UDF: USAGE and CREATE FUNCTION on the schema, and USAGE on the catalog.
Run a UDF: EXECUTE on the function, and USAGE on the schema and catalog.
Access the JAR file: READ VOLUME on the volume where the JAR is stored.

See Manage privileges in Unity Catalog for more information on Unity Catalog permissions.

Build your UDF JAR

Package your compiled code as a JAR and upload it to a Unity Catalog volume before registering the UDF. Choose a build method:

Build locally
Build in a notebook

Build locally

Follow these steps to build a fat JAR using a local development environment.

Set up your environment

Install the required tools on your local machine. The following commands are for macOS. For other platforms, install JDK 17 and sbt (Scala) or Maven (Java) using your platform's package manager.

Scala

Install JDK 17 and sbt:

brew install openjdk@17
brew install sbt

Verify your installation:

java -version   # Should show Java 17
sbt --version   # Should show sbt version

Java

Install JDK 17 and Maven:

brew install openjdk@17
brew install maven

Verify your installation:

java -version   # Should show Java 17
mvn --version   # Should show Maven version

Create your project

Set up a project in Scala or Java.

Scala

Create a new Scala project using sbt:

sbt new scala/scala-seed.g8

When prompted, enter a project name (for example, my-udf-project).

Configure build.sbt

Replace the contents of your build.sbt file with the following configuration:

scalaVersion := "2.13.16"

ThisBuild / organization := "com.example"

lazy val myUDF = (project in file("."))
  .settings(
    name := "my-udf"
  )

Enable sbt-assembly plugin

Create or edit project/assembly.sbt and add:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.0.0")

This plugin creates a fat JAR containing all your dependencies.

Java

Create a new Maven project using the quickstart archetype:

mvn archetype:generate \
  -DgroupId=com.example \
  -DartifactId=my-udf \
  -DarchetypeArtifactId=maven-archetype-quickstart \
  -DinteractiveMode=false

This command creates the standard Maven project structure with src/main/java and src/test/java directories.

Configure pom.xml

In the generated pom.xml file, within the <project></project> tags, add a <properties> block with the following configuration:

<properties>
  <maven.compiler.source>17</maven.compiler.source>
  <maven.compiler.target>17</maven.compiler.target>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

Also within the <project></project> tags, add a <build> block with the following configuration:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.5.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The maven-shade-plugin creates a fat JAR containing all your dependencies.

Write your UDF

When writing your UDF, refer to Data types for supported data types and Language mappings to see how Scala and Java types map to SQL types.

Your UDF handler must meet the following requirements:

Scala: Define the handler as a method on an object (not a class). The HANDLER value resolves to a method on a Scala object.
Java: Define the handler as a public static method.
Signature: The method's parameter types, order, and return type must correspond to the argument list and RETURNS type in your CREATE FUNCTION statement.
Scalar only: The handler must return a single scalar value. Table return types are not supported.
Self-contained: The handler must operate only on its input arguments. It cannot use Spark APIs or depend on Spark core packages. See Limitations.

Note

For Scala, a handler with a primitive parameter type (such as Int) is skipped and returns NULL when any input argument is SQL NULL. To receive and handle NULL values, wrap the parameter in Option, such as Option[Int].

Scala

Create a Scala object in src/main/scala/com/example/MyUDF.scala and define your UDF function.

Basic example

package com.example

object MyUDF {
  def addOne(x: Int): Int = x + 1
}

Example with external dependency

To use external libraries, add them to your build.sbt file:

scalaVersion := "2.13.16"

ThisBuild / organization := "com.example"

lazy val myUDF = (project in file("."))
  .settings(
    name := "currency-udf",
    libraryDependencies ++= Seq(
      "org.apache.commons" % "commons-lang3" % "3.12.0"
    )
  )

Then use the dependency in your UDF:

package com.example

import org.apache.commons.lang3.StringUtils

object CurrencyUDF {
  private val rates: Map[String, Double] = Map(
    "USD" -> 1.0,
    "EUR" -> 1.1,
    "GBP" -> 1.3,
    "JPY" -> 0.007
  )

  def convertToUSD(price: Double, currency: String): Double = {
    require(currency != null, "Currency must not be null")

    val normalizedCurrency = StringUtils.upperCase(currency)

    rates.get(normalizedCurrency) match {
      case Some(rate) => price * rate
      case None => throw new IllegalArgumentException(s"Unsupported currency: $currency")
    }
  }
}

Test your UDF with unit tests before deploying. See Testing UDFs locally.

Java

Create a Java class in src/main/java/com/example/MyUDF.java and define your UDF as a public static method.

Basic example

package com.example;

public class MyUDF {
    public static int addOne(int x) {
        return x + 1;
    }
}

Example with external dependency

To use external libraries, add them to the <dependencies> section of your pom.xml file:

<dependencies>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.12.0</version>
    </dependency>
</dependencies>

Then use the dependency in your UDF:

package com.example;

import org.apache.commons.lang3.StringUtils;
import java.util.Map;
import java.util.HashMap;

public class CurrencyUDF {
    private static final Map<String, Double> rates = new HashMap<>();

    static {
        rates.put("USD", 1.0);
        rates.put("EUR", 1.1);
        rates.put("GBP", 1.3);
        rates.put("JPY", 0.007);
    }

    public static double convertToUSD(double price, String currency) {
        if (currency == null) {
            throw new IllegalArgumentException("Currency must not be null");
        }

        String normalizedCurrency = StringUtils.upperCase(currency);

        if (!rates.containsKey(normalizedCurrency)) {
            throw new IllegalArgumentException("Unsupported currency: " + currency);
        }

        return price * rates.get(normalizedCurrency);
    }
}

Test your UDF with unit tests before deploying. See Testing UDFs locally.

Note

Your UDF runs in an isolated sandbox with no active Spark session, so it cannot use Spark APIs from within the function body. For example, you cannot create or operate on DataFrames or Datasets, run spark.sql(...), or access SparkSession or SparkContext. The UDF must be self-contained logic over its input arguments. It also cannot depend on Spark core packages.

Build your fat JAR

Build your project to create a fat JAR containing all dependencies.

Scala

From your project root directory, run:

sbt clean assembly

The fat JAR is created in target/scala-2.13/ with a name like my-udf-assembly-0.1.0-SNAPSHOT.jar.

Java

From your project root directory, run:

mvn clean package

The fat JAR is created in target/ with a name like my-udf-1.0-SNAPSHOT.jar.

Upload your JAR to a Unity Catalog volume

If you don't already have a Unity Catalog volume, create one:

CREATE VOLUME IF NOT EXISTS my_catalog.my_schema.udf_jars
COMMENT 'Storage for UDF JAR files';

If other users need to run the UDF, grant them READ VOLUME on the volume:

GRANT READ VOLUME ON VOLUME my_catalog.my_schema.udf_jars TO `user@example.com`;

Upload your JAR file to the volume using Catalog Explorer:

In your Azure Databricks workspace, click Catalog to open Catalog Explorer.
Select the catalog, then select the schema that contains your volume.
Click the volume name.
Click Upload to this volume and select your JAR file.
Click Upload.
After the upload completes, click the name of your JAR file.
Click Copy path to copy the volume path to your clipboard. For example, /Volumes/my_catalog/my_schema/udf_jars/my-udf-assembly-0.1.0-SNAPSHOT.jar (Scala) or /Volumes/my_catalog/my_schema/udf_jars/my-udf-1.0-SNAPSHOT.jar (Java). You need this path when you register the UDF.

Build in a notebook

You can compile a UDF, package it as a JAR, and upload it to a Unity Catalog volume directly from a Azure Databricks notebook. This approach works for small, dependency-free UDFs. For UDFs with third-party libraries, use Build locally.

The following Python cell writes a Java UDF that cleans a string (trims whitespace, collapses repeated spaces, and lowercases), compiles it with JDK 17, packages it as a JAR, and copies it to a Unity Catalog volume. Update the volume_path to point to an existing volume you have WRITE VOLUME permission on.

import os
import subprocess
import shutil

build_dir = "/tmp/udf_build"
package_dir = f"{build_dir}/src/com/databricks/udf"
classes_dir = f"{build_dir}/classes"
os.makedirs(package_dir, exist_ok=True)
os.makedirs(classes_dir, exist_ok=True)

# The UDF handler: a public static method on a plain Java class.
# The doubled backslashes produce a single backslash in the Java source (\\s+).
udf_code = """package com.databricks.udf;
public class StringCleanUDF {
    public static String clean(String input) {
        if (input == null) return null;
        return input.trim().replaceAll("\\\\s+", " ").toLowerCase();
    }
}
"""
with open(f"{package_dir}/StringCleanUDF.java", "w") as f:
    f.write(udf_code)

# Compile with JDK 17 to match Environment Version 4.
subprocess.run(
    ["javac", "--release", "17", "-d", classes_dir, f"{package_dir}/StringCleanUDF.java"],
    check=True,
)

# Package the compiled class into a JAR.
jar_path = f"{build_dir}/string_clean_udf.jar"
subprocess.run(["jar", "cf", jar_path, "-C", classes_dir, "."], check=True)

# Copy the JAR to a Unity Catalog volume.
volume_path = "/Volumes/my_catalog/my_schema/udf_jars/string_clean_udf.jar"
os.makedirs(os.path.dirname(volume_path), exist_ok=True)
shutil.copy2(jar_path, volume_path)

print(f"JAR uploaded to: {volume_path}")

After the JAR is in the volume, register the UDF. Use LANGUAGE JAVA and set the HANDLER to the fully qualified method, such as com.databricks.udf.StringCleanUDF.clean.

Register your UDF in Unity Catalog

After you build and upload your JAR, use the CREATE FUNCTION statement to register your UDF in Unity Catalog.

Scala

CREATE OR REPLACE FUNCTION my_catalog.my_schema.add_one(x INT)
RETURNS INT
LANGUAGE SCALA
DETERMINISTIC
ENVIRONMENT (
  java_dependencies = '["/Volumes/my_catalog/my_schema/udf_jars/my-udf-assembly-0.1.0-SNAPSHOT.jar"]',
  environment_version = '4'
)
HANDLER 'com.example.MyUDF.addOne';

Java

CREATE OR REPLACE FUNCTION my_catalog.my_schema.add_one(x INT)
RETURNS INT
LANGUAGE JAVA
DETERMINISTIC
ENVIRONMENT (
  java_dependencies = '["/Volumes/my_catalog/my_schema/udf_jars/my-udf-1.0-SNAPSHOT.jar"]',
  environment_version = '4'
)
HANDLER 'com.example.MyUDF.addOne';

The CREATE FUNCTION statement uses the following parameters:

LANGUAGE: The language of the UDF.
HANDLER: Fully qualified path to the method, in the format 'package.Object.method' (Scala) or 'package.ClassName.method' (Java).
DETERMINISTIC: Declares that the function always returns the same output for the same input, enabling query optimization.

Note

Remove DETERMINISTIC if your function calls external APIs or has any other non-deterministic behavior.
ENVIRONMENT: Defines the execution environment for the UDF.
- java_dependencies: A JSON array of JAR file paths in your Unity Catalog volumes. This is the file path you copied in the previous step. Use single quotes around the array and double quotes around paths.
- environment_version: Must be '4' or above for Scala and Java UDFs. Environment Version 4 specifies Scala 2.13.16 and JDK 17. See Serverless environment versions.

Call your UDF in SQL and notebooks

After registration, you can call the UDF in SQL queries, notebooks, and views:

-- Simple select
SELECT my_catalog.my_schema.add_one(5) AS result;

-- With table data
SELECT
  id,
  price,
  currency,
  my_catalog.my_schema.convert_to_usd(price, currency) AS price_usd
FROM my_catalog.my_schema.transactions;

-- Filtering
SELECT *
FROM my_catalog.my_schema.products
WHERE my_catalog.my_schema.convert_to_usd(price, currency) > 100;

-- Aggregation
SELECT
  category,
  SUM(my_catalog.my_schema.convert_to_usd(price, currency)) AS total_usd
FROM my_catalog.my_schema.sales
GROUP BY category;

Use Unity Catalog permissions to control who can run your UDF and to make it discoverable across your organization.

Grant permissions

Use Catalog Explorer or SQL to grant the necessary permissions for other users to run your UDFs.

Catalog Explorer

In the sidebar, click Catalog.
Select the catalog, then select the schema that contains your function.
Click the function name.
In the Permissions tab, click Grant.
Select the principals you want to grant access to, and select the EXECUTE permission.
Click Confirm.

SQL

Run the following command in a notebook or the Databricks SQL editor to grant EXECUTE permissions to a user or group.

-- Grant to a specific user
GRANT EXECUTE ON FUNCTION my_catalog.my_schema.add_one TO `user@example.com`;

-- Grant to a group
GRANT EXECUTE ON FUNCTION my_catalog.my_schema.add_one TO `data-engineers`;

Revoke permissions

Use Catalog Explorer or SQL to revoke permissions from other users.

Catalog Explorer

In the sidebar, click Catalog.
Select the catalog, then select the schema that contains your function.
Click the function name.
In the Permissions tab, select the checkbox next to the principal you want to revoke access from. Click Revoke.
In the notification, click Revoke.

SQL

Run the following command in a notebook or the Databricks SQL editor to revoke EXECUTE permissions from a user or group.

-- Revoke from specific user
REVOKE EXECUTE ON FUNCTION my_catalog.my_schema.add_one FROM `user@example.com`;

-- Revoke from a group
REVOKE EXECUTE ON FUNCTION my_catalog.my_schema.add_one FROM `data-engineers`;

Discover UDFs

To find UDFs managed in Unity Catalog, query the information_schema.routines table, replacing the my_catalog and my_schema values:

SELECT
  routine_catalog,
  routine_schema,
  routine_name,
  routine_definition,
  created
FROM system.information_schema.routines
WHERE routine_catalog = 'my_catalog'
  AND routine_schema = 'my_schema';

Update your UDF

To update an existing Unity Catalog UDF with new code:

Make changes to your code locally.
Rebuild the JAR with a new version number.
- Scala: sbt clean assembly (for example, my-udf-assembly-0.2.0-SNAPSHOT.jar)
- Java: mvn clean package (for example, my-udf-2.0-SNAPSHOT.jar)
Upload the new JAR to the Unity Catalog volume.
Use CREATE OR REPLACE FUNCTION with the same function name to update the UDF. Verify that you reference the latest JAR in your java_dependencies.

Azure Databricks uses the new code on the next invocation. You don't need to restart your cluster.

Performance optimization

Cold start latency

The first UDF call in a session initializes the isolated sandbox, which adds latency. Subsequent calls in the same session are faster. Account for this when benchmarking or designing latency-sensitive workloads.

Caching expensive computations

If your UDF performs expensive initialization or computation, cache the result to compute it only once.

Scala

Use a val field in the Scala object to cache the result:

package example

object CachedUDF {
  // Computed once and cached
  val expensiveData: Map[String, Double] = {
    // Load data from somewhere expensive
    Map("key1" -> 1.0, "key2" -> 2.0)
  }

  def lookup(key: String): Double = {
    expensiveData.getOrElse(key, 0.0)
  }
}

Java

Use a static field with a static initializer block to cache the result:

package example;

import java.util.Map;
import java.util.HashMap;

public class CachedUDF {
    // Computed once and cached
    private static Map<String, Double> expensiveData;

    static {
        // Load data from somewhere expensive
        expensiveData = new HashMap<>();
        expensiveData.put("key1", 1.0);
        expensiveData.put("key2", 2.0);
    }

    public static double lookup(String key) {
        return expensiveData.getOrDefault(key, 0.0);
    }
}

Use DETERMINISTIC when appropriate

Mark your UDF as DETERMINISTIC if it always produces the same output for the same input. This allows the query optimizer to cache results and improve performance.

Limitations

Only scalar UDFs are supported. User-defined aggregate functions (UDAFs) and user-defined table functions (UDTFs) are not supported.
UDFs run in an isolated sandbox with no active Spark session. Spark APIs (SparkSession, SparkContext, spark.sql(...), DataFrame and Dataset operations) are not available.
UDFs cannot depend on Spark core packages.
UDFs do not have access to workspace files or Unity Catalog volumes at runtime.

Best practices

Databricks recommends the following practices:

Version your JAR files. For example, my-udf-0.1.0.jar, my-udf-0.2.0.jar.
Validate SQL type mappings before deployment. See Language mappings.
Grant READ VOLUME and EXECUTE permissions only to users who need to run the UDF. Use group ownership for UDFs shared across teams.

Testing UDFs locally

Test your UDF with unit tests before deploying to production.

Scala

To test src/main/scala/example/MyUDF.scala, create a test file in src/test/scala/example/MyUDFTest.scala:

package example

import org.scalatest.funsuite.AnyFunSuite

class MyUDFTest extends AnyFunSuite {
  test("addOne should add 1 to input") {
    assert(MyUDF.addOne(5) == 6)
  }

  test("addOne should handle negative numbers") {
    assert(MyUDF.addOne(-1) == 0)
  }
}

Add the test dependency to build.sbt:

libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.15" % Test

To run the tests:

sbt test

Java

To test src/main/java/com/example/MyUDF.java, create a test file in src/test/java/com/example/MyUDFTest.java:

package com.example;

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

public class MyUDFTest {
    @Test
    public void testAddOne() {
        assertEquals(6, MyUDF.addOne(5));
    }

    @Test
    public void testAddOneWithNegativeNumbers() {
        assertEquals(0, MyUDF.addOne(-1));
    }
}

Add the JUnit dependency to the <dependencies> section of your pom.xml:

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>5.10.0</version>
    <scope>test</scope>
</dependency>

To run the tests:

mvn test

Additional resources

გამოხმაურება

ეს გვერდი სასარგებლო იყო?

Last updated on 2026-07-16

Scala and Java user-defined functions (UDFs) in Unity Catalog

Requirements

Build your UDF JAR

Build locally

Set up your environment

Scala

Java

Create your project

Scala

Java

Write your UDF

Scala

Java

Build your fat JAR

Scala

Java

Upload your JAR to a Unity Catalog volume

Build in a notebook

Register your UDF in Unity Catalog

Scala

Java

Call your UDF in SQL and notebooks

Governance and sharing

Grant permissions

Catalog Explorer

SQL

Revoke permissions

Catalog Explorer

SQL

Discover UDFs

Update your UDF

Performance optimization

Cold start latency

Caching expensive computations

Scala

Java

Use DETERMINISTIC when appropriate

Limitations

Best practices

Testing UDFs locally

Scala

Java

Additional resources

გამოხმაურება

დამატებითი რესურსები