date folders

Question

I am populating the output folder with .parquet files as follows:
folder_path = "container/foldername/output"

What I would like to do is for each load to load data into separate day folder as follows:
...
container/foldername/output/year=2022/month=04/day=19
container/foldername/output/year=2022/month=04/day=20
container/foldername/output/year=2022/month=04/day=21
...

What do I have to add to the end of the folder_path to get to this please?

I tried the below but it does not seem right
from pyspark.sql.functions import year
from pyspark.sql.functions import month
from pyspark.sql.functions import to_timestamp,date_format
from pyspark.sql.functions import current_timestamp

current_timestamp()

yearNo = date_format(current_timestamp(), 'Y')
monthNo = date_format(current_timestamp(), 'M')
dayNo = date_format(current_timestamp(), 'D')

print (yearNo)
print (MonthNo)
print (DayNo)

Thank you

Answer 1

Hi @arkiboys ,

Thank you for posting query in Microsoft Q&A Platform.

As per my understanding you are trying to compile path by getting year, month and day parts from current data time in python. Please correct me if I am wrong.

In Python we should consider importing datetime module to work with dates.

Please check below screenshot to get better understanding of logic to write.

Code used in above image:

import datetime  
  
path = 'container/foldername/output/'  
  
currentDateTime = datetime.datetime.now() #gives you current date time  
  
year = currentDateTime.year  
month = currentDateTime.month  
day =currentDateTime.day  
  
fullpath = path + 'year=' + str(year) + '/month=' + str(month) + '/day=' + str(day)  
  
print(fullpath)  

Hope this helps. Please let us know if any further queries.

-----------------

Please consider hitting Accept Answer button. Accepted answers help community as well.