Apache slf4j error when setting up spark pool

Question

I am getting the following errors when my apache spark pool job is running. The main issue seems to be the slf4j error. How do I fix this:

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt/tmp
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt/tmp
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/5.2.20240522.3/spark3/jars/spark-streaming-kafka-0-10-assembly_2.12-3.3.1.5.2.20240522.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/5.2.20240522.3/spark3/jars/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Warning: Ignoring non-Spark config property: EntitySizes
Warning: Ignoring non-Spark config property: EntityNames
24/09/02 08:01:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/09/02 08:01:54 WARN AzureBlobFileSystemStore: checkDnsEntry: prdcpf11wukmefstor.dfs.core.windows.net not found in the file /etc/hosts.

Answer 1

@Daniel Pearson
I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: Getting errors when apache spark pool job is running. The main issue seems to be the slf4j error.

Solution: This was caused by an unknown config on our synapse workspace. Solution was to create a new synapse workspace with default config and then setup a new spark pool which we then connected to Azure Synapse Link.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.