How to use a different version of a Spark Java library dependency (antlr4) in a Databricks notebook?

Question

How to use a different version of a Spark Java library dependency (antlr4) in a Databricks notebook?

Martin Medina 5

Hello. I need to use in a Databricks notebook a custom made Java library which depends on Drools v8.40.1.Final which depends on ANTLR4 v4.10.1. When I try to invoke a method in my Java library I get the following error: "ANTLR Tool version 4.10.1 used for code generation does not match the current runtime version 4.9.3. UnsupportedOperationException: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with version 4 (expected 3)". What I have found out searching the web is that the Databricks runtime Spark Java libraries uses ANTLR4 4.9.3, which is incompatible with the version used by my Java library (https://github.com/graphql-java-kickstart/graphql-java-tools/issues/767#issuecomment-1803407930). Also, that the Databricks Spark Java libraries cannot be overriden and if I add conflicting libraries, the Spark driver libraries take precedence (https://docs.databricks.com/en/workflows/jobs/how-to/use-jars-in-workflows.html#manage-library-dependencies). Do you know if there is a way to make my library works? I know that I could downgrade to a Drools version that uses an older ANTLR4 library. In fact, Drools v8.30.0.Final uses ANTLR4 v4.9.2, which works in Databricks clusters with runtime v14.2. But I have a non functional requirement to use a specific Drools version which has support.

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2024-01-23T17:03:21.3833333+00:00
Hi Martin Medina, Thanks for reaching out here and posting your query.

In your case the safest bet is to downgrade but looks like it doesn't meet your requirements to use specific Drools version. But if you want to try replacing 4.9.3 with 4.10.1 and test it out, you may try to do the following and see if that works out.

Create a cluster scoped init script, that will replace the library like below and see if it works:

#!/bin/bash wget -O /dbfs/FileStore/...4,10.1... https://...maven.../...jar rm -rf /databricks/jars/...4.9.3...jar cp /dbfs/FileStore/...4.10.1 /databricks/jars

Let us know how it goes. Thanks
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2024-01-24T17:35:49.0133333+00:00

Hi Martin Medina, Just checking to see if you have got a chance to try the suggested workaround. If so, please keep us posted if it worked or if you have found a different solution. Thank you
Martin Medina 5 Reputation points

2024-01-25T22:31:13.15+00:00

Hello @KranthiPakala-MSFT I'll ask my coworker who has access to the cluster to try what you suggested. Thanks!

1 answer

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2024-01-23T17:03:21.3833333+00:00

Hi Martin Medina, Thanks for reaching out here and posting your query.

In your case the safest bet is to downgrade but looks like it doesn't meet your requirements to use specific Drools version. But if you want to try replacing 4.9.3 with 4.10.1 and test it out, you may try to do the following and see if that works out.

Create a cluster scoped init script, that will replace the library like below and see if it works:

#!/bin/bash wget -O /dbfs/FileStore/...4,10.1... https://...maven.../...jar rm -rf /databricks/jars/...4.9.3...jar cp /dbfs/FileStore/...4.10.1 /databricks/jars

Let us know how it goes. Thanks
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2024-01-24T17:35:49.0133333+00:00

Hi Martin Medina, Just checking to see if you have got a chance to try the suggested workaround. If so, please keep us posted if it worked or if you have found a different solution. Thank you
Martin Medina 5 Reputation points

2024-01-25T22:31:13.15+00:00

Hello @KranthiPakala-MSFT I'll ask my coworker who has access to the cluster to try what you suggested. Thanks!

Answer 1

Martin Medina 5

We ended up using shading with a maven plugin to change the antlr4 parent package, so it doesn't collide with the library that the Spark runtime uses.

Carlos Irazabal 0 Reputation points

2024-05-24T16:55:41.88+00:00

Hi , I have the same problem trying to run a JAR with drools core that execute a DMN rules:

UnsupportedOperationException: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with version 4 (expected 3). Caused by: InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with version 4 (expected 3).

I was trying to change the dependency of the antlr package without results. Could you share your fix in your pom.xml?

Thanks a lot.

Carlos

Share via

How to use a different version of a Spark Java library dependency (antlr4) in a Databricks notebook?

1 answer

Your answer