Hi @KranthiPakala-MSFT , I was able to fix the error. I disabled spark.kryo.registrationRequired, and register manually those classes that I needed.
Thank you!
This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Hi All,
Recently I have tried to implement Kryo into a synapse notebook. But I have faced some issues while doing that. And the information out there is not that clear or at least didn't help me with my specific issue.
These are the SparkConf that I'm using right now for Kryo:
And this is the way I'm registering classes:
And as you can see, one of those classes is StructType, but even when I'm clearly registering that class, I'm getting this error message when I do something as simple as creating a Dataframe:
Caused by: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.types.StructType[]
Note: To register this class use: kryo.register(org.apache.spark.sql.types.StructType[].class);
Link to full error message: https://pastebin.com/mYzANVLY
And if I try to follow what the error message said, I got this other error:
And even if I try something different like this:
I'm still facing Class is not registered
issue.
An example code in which I can get this error message:
Another example (I previously registered Row as a class and still facing the issue):
So, If anyone can give me some insight in what the proper manner is to register classes in synapse or if am I missing something, it will be appreciate it!
Thank you!
Hi @KranthiPakala-MSFT , I was able to fix the error. I disabled spark.kryo.registrationRequired, and register manually those classes that I needed.
Thank you!
Hello @Pablo Chinchilla Valverde ,
Thanks for the question and using MS Q&A platform.
Have you got a chance look at below stackoverflow threads where a similar issue has been discussed. If not please refer to them and see if that helps.
Do let us know how it goes.
Thank you