question

pankajchaturvedi-6530 avatar image
0 Votes"
pankajchaturvedi-6530 asked pankajchaturvedi-6530 commented

Schedule ADF pipeline schedule based on spec day

Hi Team,

Hope you are doing well.!

I do want to trigger my pipeline based on spec day using Schedule trigger or If it there is any other way().

For example: In the month of March I want to trigger the pipeline on dates 24,25th March and the Month of April I want to trigger on 8th,9th April and the month of May 12,13th May and so on for the rest of the months.

Could you please suggest me how can I implement it if someone has done it or any idea about it.

Many thanks,
Pankaj

azure-data-factory
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @pankajchaturvedi-6530 ,

Just checking in to see if the below answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hi @pankajchaturvedi-6530 , Following up to see if below answer helps you? If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let us know if any further queries.

0 Votes 0 ·

1 Answer

NandanHegde-7720 avatar image
0 Votes"
NandanHegde-7720 answered pankajchaturvedi-6530 commented

Hey,
Below are the 2 ways that come to my mind at the top:
1) You would have to create 12 different triggers pertaining to each month and select specific days in each trigger and associate it with the pipeline
2) Create a config table that would contain the dates and within your pipeline use a lookup activity at start to check whether utcnow() is equal to the value in that config table, if yes proceed else skip the steps.
In this scenario, you would have to trigger the pipeline daily.

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Nandan,

Thanks for your reply.

Could you please share the config table schema and screenshots for the above implementation.?

Thanks!!

0 Votes 0 ·
NandanHegde-7720 avatar image NandanHegde-7720 pankajchaturvedi-6530 ·

Hey,
PFB the sample table format :
create table #ExtractDt
(
PipelineNm varchar(255),
ExtractDt Datetime,
Status varchar(50)
)

 insert into #ExtractDt
 select 'Test','2021-12-30','Processed'
 union
 Select 'Test','2022-04-01','Not Processed'



ADF logic:
1) Use lookup activity to get the extractdate that is next in line to be processed
2) IF activity to compare whether the lookup activity output equals utcnow()
if yes, then proceed with other activities else skip
3) If proceeded with activites , then update the status of that date in config to Processed


186382-image.png


0 Votes 0 ·
image.png (75.9 KiB)

Hi Nandan,

Thanks for your response. Thanks!!

0 Votes 0 ·
Show more comments