Generate date rows in ADF data flow

Question

I'm just starting to learn Azure Data Factory as we have a need to migrate our Alteryx workflows.

I've created a data flow within data factory which summarizes data to return a max and min date column per recruiter, as per below.

What I now need to do is to create a row for each recruiter for each date between, and including, the min and max dates. So the result would be-

Is this possible?

Answer 1

Hello @Richard Hawkins (AXIS) glad you found the solution. Since you cannot accept your answer I am summarizing it here so you can accept it as an answer.

1. Derived Column Transformation:
   • You created a Derived Column that uses the mapLoop function. This function iterates over a specified number of times, allowing you to dynamically create dates within the range of Min Date and Max Date.
   • The expression you used is:

          
          mapLoop((Max_Date - Min_Date) + 1, toString(addDays(Min_Date - 1, #index)))
          
     

   • Explanation of the Expression:
     • Max_Date - Min_Date + 1: Calculates the total number of days between Min Date and Max Date, inclusive.
     • mapLoop(...): Repeats the logic for each day in the range, generating a list of dates.
     • addDays(Min_Date - 1, #index): This function calculates each specific date by adding the loop index (#index) to Min Date - 1, generating consecutive dates starting from Min Date.
     • toString(...): Converts each date in the list to a string format.
   • This results in a concatenated string of dates for each recruiter in the format [Date1, Date2, Date3,...].
2. Flatten Transformation:
   • After creating the concatenated list of dates, you used the Flatten transformation to split this list into individual rows.
   • The Flatten transformation extracts each date in the concatenated string and generates a row for each date, providing the desired output of one row per date per recruiter.

Answer 2

Hi @Richard Hawkins (AXIS)

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Ask: I'm just starting to learn Azure Data Factory as we have a need to migrate our Alteryx workflows.

I've created a data flow within data factory which summarizes data to return a max and min date column per recruiter, as per below.

What I now need to do is to create a row for each recruiter for each date between, and including, the min and max dates. So the result would be-

Is this possible?

Solution: I managed to find a different solution in the end. I created a derived column using-

mapLoop((Max_Date-Min_Date)+1, toString(addDays(Min_Date-1,#index)))

This created a concatenated string of dates for each recruiter. I then simply used the flatten tool to transform the concatenated dates into rows.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members