pyspark notebook

Question

Hello,
I have a dataframe which I have managed to get to work as follows:
...
actual_df = actual_df.withColumn(col_name, date_format("LoadDate", "dd MMM yyyy"))
...

Question:
How is it possible to getthe below dataframe to be able to use the expr function?

expression = date_format("LoadDate", "dd MMM yyyy")
actual_df = actual_df.withColumn(col_name, expr(expression))

The code above gives an error as follows:
SQL
date_format("LoadDate",

Thank you

Answer 1

Hi @arkiboys ,
Thanks for posting question on Microsoft Q&A platform.

expr function converts string into an expression. In your case, you already have "expression" variable which is holding a column expression of a dataframe.
While trying to use withColumn function on top of your dataframe, it expects the input parameter as (colName, col) where col is nothing but a column expression.

expr function is not needed in this case. Kindly check the below screenshot for your reference.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators