Missing columns in Spark

Ryan Abbey 1,181 Reputation points
2022-04-20T23:15:58.42+00:00

I was trying to do some analysis on duplicate rows and noticed something a little unusual... my intent was to compare the "ModDate" for changes by running below

194874-image.png

Result is not relevant here but what I noticed is that the "max(ModDate)" column was not displaying... I'm able to apply the filter against it so that would suggest it is "aware" of the field (possibly the literal calculation of min() and max() as opposed to the naming of the field at this point?)

So I tried just selecting it

194884-image.png

to no avail... it appears that while it does the aggregation of these columns, when it comes to displaying them, the input column name for an aggregation will only output one column... a flaw in the naming process?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,172 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,476 Reputation points Microsoft Employee
    2022-04-21T23:06:45.003+00:00

    Hello @Ryan Abbey ,
    Thanks for the question and using MS Q&A platform.
    As we understand the ask here is to how we can show more then one aggregate value , please do let us know if its not accurate.

    If you use only one aggregate function its should work fine . In your case you are using both Max & Min . I tried with one and it worked .
    195332-image.png

    For more then one aggregate the below code worked for me .

    from pyspark.sql import functions as F

    195329-image.png

    Please do let me if you have any queries.
    Thanks
    Himanshu


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.