Share via

Not able to display a dataframe in Pyspark, I needed the interactive chart that synapse renders. The display function shows an inconsistent behaviour.

Albert Varghese Rajan 0 Reputation points
Jul 10, 2023, 1:01 PM

Not able to display a parquet data frame in Pyspark, but the show function works. I needed the interactive chart that Synapse renders. I believe it is to do the lazy evaluation, but what can be done so that the display function works?

The display function is returning

DataFrame[a: string, b: array, b: array, d: array, e: array, f: array, g: array]

In the same notebook, it is the same case with other data frames, I tried running in other sparkpools again the same results

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,094 questions
{count} votes

1 answer

Sort by: Most helpful
  1. KranthiPakala-MSFT 46,602 Reputation points Microsoft Employee
    Jul 11, 2023, 10:41 PM

    @Albert Varghese Rajan

    Welcome to Microsoft Q&A forum and thanks for reaching out here.

    I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

    Error Message:

    Issue:

    • Unable to display the dataframe in Pyspark and the display function shows inconsistent behavior

    Solution:

    • The issue could be due to size of the dataframe. Hence to overcome the issue, you have scaled up the node configuration of your apache spark pool and the issue is resolved. Here is how to scale up the node configuration for an Apache Spark pool:

    User's image

    User's image

    If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

    I hope this helps!

    If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.