Synapse Spark Pool(spark 3.1): unable to find LoginModule class:

Robson Soares 1 Reputation point

I am trying to test a scala notebook in synapse I have to ingest kafka messages using databricks with success.

I know synapse have included kafka packages on new spark runtime 3.1 ( but when I run my notebook I got an error.

Basically kafka cannot create the consumer due the failure : org.apache.kafka.common.KafkaException: unable to find LoginModule class:

I have tested using a pyspark version of notebook but the error is the same

I think there are missing dependencies or some issues with the existing kafka packages on that runtime. I guess it could be the Kafka-Client version.

Any helps?

Best regards

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,826 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 70,581 Reputation points Microsoft Employee

    Hello @Robson Soares ,

    Here is the detailed response provided by the internal team:

    I ran the scenario and some things came up.

    1. The guide that you are usin`g is starter code is dated. It relies on Spark 2.4, it’s not targeted to Synapse; so some instructions are misleading.
    2. It states the you should use, in the sasl_jaas_configvariable, claiming it’s a Databricks only library. That is incorrect, this class is part of thekarfka-clients-` we ship in Synapse Spark 3.1 runtime. Here is the class reference.
    3. Also some options in the readStream command are dated and should be as follows:
      .option("kafka.sasl.mechanism", "PLAIN")
      .option("", "SASL_SSL")
      .option("kafka.sasl.jaas.config", sasl_jaas_config)

    Could you please try the above and do let us know the status.

    Hope this will help. Please let us know if any further queries.


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators