date folders

arkiboys 9,686 Reputation points
2022-04-21T15:12:05.747+00:00

I am populating the output folder with .parquet files as follows:
folder_path = "container/foldername/output"

What I would like to do is for each load to load data into separate day folder as follows:
...
container/foldername/output/year=2022/month=04/day=19
container/foldername/output/year=2022/month=04/day=20
container/foldername/output/year=2022/month=04/day=21
...

What do I have to add to the end of the folder_path to get to this please?

I tried the below but it does not seem right
from pyspark.sql.functions import year
from pyspark.sql.functions import month
from pyspark.sql.functions import to_timestamp,date_format
from pyspark.sql.functions import current_timestamp

current_timestamp()

yearNo = date_format(current_timestamp(), 'Y')
monthNo = date_format(current_timestamp(), 'M')
dayNo = date_format(current_timestamp(), 'D')

print (yearNo)
print (MonthNo)
print (DayNo)

Thank you

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,070 questions
0 comments No comments
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 38,326 Reputation points Microsoft Employee
    2022-04-22T06:55:32.86+00:00

    Hi @arkiboys ,

    Thank you for posting query in Microsoft Q&A Platform.

    As per my understanding you are trying to compile path by getting year, month and day parts from current data time in python. Please correct me if I am wrong.

    In Python we should consider importing datetime module to work with dates.

    Please check below screenshot to get better understanding of logic to write.
    195378-image.png

    Code used in above image:

    import datetime  
      
    path = 'container/foldername/output/'  
      
    currentDateTime = datetime.datetime.now() #gives you current date time  
      
    year = currentDateTime.year  
    month = currentDateTime.month  
    day =currentDateTime.day  
      
    fullpath = path + 'year=' + str(year) + '/month=' + str(month) + '/day=' + str(day)  
      
    print(fullpath)  
    

    Hope this helps. Please let us know if any further queries.

    -----------------

    Please consider hitting Accept Answer button. Accepted answers help community as well.


0 additional answers

Sort by: Most helpful