Extract the date hierarchy from previous data in Data Lake Partition

RahulK 101 Reputation points

Hi Team,

Would be of great help to get some clear assistance on the below query.

Data source is Dynamic365 & SAP Hana.

My requirement is to create a data lake partition(ADLS Gen2) with date hierarchy(SubjectArea-->RawData-->Year-->Month-->Day). Data available is from 2018-2020 and new data will be flowing-in daily.

Can you pls guide me to create this structure using ADF to copy multiple tables & extract the date format from previous data and store the current data as well.

Thanks in advance.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
992 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,139 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. ChiragMishra-MSFT 946 Reputation points

    Hi Pramod,

    If you want to copy data daily and dynamically generate the folder hierarchy, you can set up a dynamic path in your sink dataset (ADLS Gen2) as shown below :


    The expression I have used will create a structure like "base path"/<current year>/<current month>/<current date>/

    @concat('base path' , formatDateTime(utcnow(), 'yyyy'), '/', formatDateTime(utcnow(), 'MM'), '/', formatDateTime(utcnow(), 'dd'), '/')  

    Hope this helps.

    1 person found this answer helpful.

  2. ChiragMishra-MSFT 946 Reputation points

    If you want to use existing folders in your sink to determine the next folder's structure, you can do the following :


    • Proceed as above, use a dynamic expression in your dataset and use your variable(s) and string functions if needed to create the hierarchy you want.

    Hope this helps.