What is the proper manner to use Kryo on Synapse Analytics?

Pablo Chinchilla Valverde 36 Reputation points
2022-11-23T02:00:17.057+00:00

Hi All,
Recently I have tried to implement Kryo into a synapse notebook. But I have faced some issues while doing that. And the information out there is not that clear or at least didn't help me with my specific issue.
These are the SparkConf that I'm using right now for Kryo:
263170-image.png

And this is the way I'm registering classes:
263252-image.png

And as you can see, one of those classes is StructType, but even when I'm clearly registering that class, I'm getting this error message when I do something as simple as creating a Dataframe:

Caused by: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.types.StructType[]
Note: To register this class use: kryo.register(org.apache.spark.sql.types.StructType[].class);

Link to full error message: https://pastebin.com/mYzANVLY

And if I try to follow what the error message said, I got this other error:
263253-image.png

And even if I try something different like this:
263187-image.png

I'm still facing Class is not registered issue.
An example code in which I can get this error message:
263200-image.png

Another example (I previously registered Row as a class and still facing the issue):
263237-image.png

So, If anyone can give me some insight in what the proper manner is to register classes in synapse or if am I missing something, it will be appreciate it!

Thank you!

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,532 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Pablo Chinchilla Valverde 36 Reputation points
    2022-12-07T18:15:39.363+00:00

    Hi @KranthiPakala-MSFT , I was able to fix the error. I disabled spark.kryo.registrationRequired, and register manually those classes that I needed.

    Thank you!

    1 person found this answer helpful.

  2. KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator
    2022-11-24T02:39:40.787+00:00

    Hello @Pablo Chinchilla Valverde ,

    Thanks for the question and using MS Q&A platform.

    Have you got a chance look at below stackoverflow threads where a similar issue has been discussed. If not please refer to them and see if that helps.

    1. Require kryo serialization in Spark (Scala)
    2. Kryo serialization refuses to register class
    3. kryo Documentation

    Do let us know how it goes.

    Thank you


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.