Synapse Spark Pool(spark 3.1): unable to find LoginModule class: kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule

Question

I am trying to test a scala notebook in synapse I have to ingest kafka messages using databricks with success.

I know synapse have included kafka packages on new spark runtime 3.1 (https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-3-runtime) but when I run my notebook I got an error.

Basically kafka cannot create the consumer due the failure : org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class: kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule

I have tested using a pyspark version of notebook but the error is the same

I think there are missing dependencies or some issues with the existing kafka packages on that runtime. I guess it could be the Kafka-Client version.

Any helps?

Best regards

Answer 1

Hello @Robson Soares ,

Here is the detailed response provided by the internal team:

I ran the scenario and some things came up.

1. The guide that you are usin`g is starter code is dated. It relies on Spark 2.4, it’s not targeted to Synapse; so some instructions are misleading.
2. It states the you should use kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModuleinstead oforg.apache.kafka.common.security.plain.PlainLoginModule, in the sasl_jaas_configvariable, claiming it’s a Databricks only library. That is incorrect, this class is part of thekarfka-clients-2.4.1.5.0.jar` we ship in Synapse Spark 3.1 runtime. Here is the class reference.
3. Also some options in the readStream command are dated and should be as follows:
   .option("kafka.sasl.mechanism", "PLAIN")
   .option("kafka.security.protocol", "SASL_SSL")
   .option("kafka.sasl.jaas.config", sasl_jaas_config)

Could you please try the above and do let us know the status.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators