Please tell me how to retrieve data.

NishimuraChinatsu-9854 756 Reputation points
2023-02-27T12:09:41.4366667+00:00

I would like to get the data for the previous 2 weeks from the date specified in the parameter, and I would like to know how to get the data for 2 weeks in one pipeline.

The data is stored in GEN2 as AAA/year/mm/day by day.

Do I use variables or for each or until activities?

Sorry for my lack of study, but the image is like this.

User's image

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,355 questions
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2023-02-28T20:47:22.94+00:00

    Hi @ ponponpon ,

    Welcome to Microsoft Q&A forum and thanks for reaching out here.

    As per my understanding, you have an input parameter, which carries a date value and based on that you will have to copy the last 2 weeks data from the provided date to your desired sink data store and your source is ADLS Gen2 with folder structure as

    User's image

    Please correct if my understanding is not accurate.
    To achieve your requirement, rather getting each date and looping through each folder, the ideal solution would be to utilize the wildcard file path feature in combination with Filter by last modified feature in copy activity to copy only the file data for past 2 weeks from the given date using a single copy activity.

    Please follow below settings in your Copy activity.
    Since you are having parameter for date as reference point. You can use adddays() function to find the past 14 days date as @adddays(pipeline().parameters.param_inputDate, -14) which you will pass as Start time for filter by last modified property and for End time you can pass your input date parameter as is (@pipeline().parameters.param_inputDate) and then select Recursively property to true so that all files under Kranthi/AAA which have datetime in the past 2 weeks will get copied to your sink.
    User's image

    Please note that above solution best suits when your dated folder contains files with created/modified datetime as the folder name date. Incase if your file folder name date and file created/modified datetime varies, please let me know.

    Hope this info helps. Let us know if you have further questions.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.


0 additional answers

Sort by: Most helpful