date folders

arkiboys 9,686 Reputation points

I am populating the output folder with .parquet files as follows:
folder_path = "container/foldername/output"

What I would like to do is for each load to load data into separate day folder as follows:

What do I have to add to the end of the folder_path to get to this please?

I tried the below but it does not seem right
from pyspark.sql.functions import year
from pyspark.sql.functions import month
from pyspark.sql.functions import to_timestamp,date_format
from pyspark.sql.functions import current_timestamp


yearNo = date_format(current_timestamp(), 'Y')
monthNo = date_format(current_timestamp(), 'M')
dayNo = date_format(current_timestamp(), 'D')

print (yearNo)
print (MonthNo)
print (DayNo)

Thank you

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,070 questions
0 comments No comments
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 38,326 Reputation points Microsoft Employee

    Hi @arkiboys ,

    Thank you for posting query in Microsoft Q&A Platform.

    As per my understanding you are trying to compile path by getting year, month and day parts from current data time in python. Please correct me if I am wrong.

    In Python we should consider importing datetime module to work with dates.

    Please check below screenshot to get better understanding of logic to write.

    Code used in above image:

    import datetime  
    path = 'container/foldername/output/'  
    currentDateTime = #gives you current date time  
    year = currentDateTime.year  
    month = currentDateTime.month  
    fullpath = path + 'year=' + str(year) + '/month=' + str(month) + '/day=' + str(day)  

    Hope this helps. Please let us know if any further queries.


    Please consider hitting Accept Answer button. Accepted answers help community as well.

0 additional answers

Sort by: Most helpful