Data profiling on azure synapse using pyspark

Question

Data profiling on azure synapse using pyspark

Shivank.Agarwal 61

I am trying to do the data profiling on synapse database using pyspark. I was able to create a connection and loaded data into DF.

import spark_df_profiling

report = spark_df_profiling.ProfileReport(jdbcDF)

but getting the below error.

PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2021-06-01T08:49:43.123+00:00

Hello @Shivank.Agarwal ,

Welcome to the Microsoft Q&A platform.

Could you please share the complete steps which you have tried? What does jdbcDF contains?

As per the repro I'm able to use spark-df-profiling, by loading in your Spark DataFrame.
Shivank.Agarwal 61 Reputation points

2021-06-01T15:19:51.447+00:00

jdbcDF = spark.read.format("jdbc").option("url", f"jdbc:sqlserver://XXXXXX={database}").option("dbtable", "XXXX").option("user", 'XXXX').option("password", "XXXXX").option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()

Answer accepted by question author

1 additional answer

Your answer

PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2021-06-01T08:49:43.123+00:00

Hello @Shivank.Agarwal ,

Welcome to the Microsoft Q&A platform.

Could you please share the complete steps which you have tried? What does jdbcDF contains?

As per the repro I'm able to use spark-df-profiling, by loading in your Spark DataFrame.
Shivank.Agarwal 61 Reputation points

2021-06-01T15:19:51.447+00:00

jdbcDF = spark.read.format("jdbc").option("url", f"jdbc:sqlserver://XXXXXX={database}").option("dbtable", "XXXX").option("user", 'XXXX').option("password", "XXXXX").option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()

Answer 1

PRADEEPCHEEKATLA 91,496 Moderator

Hello @Shivank.Agarwal ,

Welcome to the Microsoft Q&A platform.

It looks like issue with the package spark-df-profiling 1.1.13 .

The package spark-df-profiling 1.1.13 is quite old and could you please try with the latest package spark-df-profiling-new 1.1.14.

As per the repro, I had tried with both the versions.

Spark2 cluster is installed with spark-df-profiling 1.1.13, when I execute the query I get similar kind of error message as shown.

Spark1 cluster is installed with spark-df-profiling-new 1.1.14, when I execute the query I’m able to successfully get the output.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Shivank.Agarwal 61 Reputation points

2021-06-03T05:45:58.367+00:00

@PRADEEPCHEEKATLA Thanks!! it worked.
PRADEEPCHEEKATLA 91,496 Reputation points Moderator

2021-06-03T08:46:55.603+00:00

Hello @Shivank.Agarwal ,

Glad to know that your issue has resolved.

Answer 2

Gladys Cardozo 1

Hi to all! I already tryied what you explain and it works! But my problem is I don't know how to read the object I obtained: <spark_df_profiling.ProfileReport object at 0x7fa1008dfb38>. I tryied to save to the azure blob but I don't what I'm doing wrong. I try this because I wanted to explore the html generated but seems like azure don't recognize it.

How do you manage to open the result after executing profiling library?

King Regards,

Gladys.

Share via

Data profiling on azure synapse using pyspark

1 additional answer

Your answer