Stop a copy activity while it is in progress

Question

Stop a copy activity while it is in progress

Prith 1

I have the following requirements in Azure Data Factory. Could somebody tell me if it is possible in ADF -

Stop/Fail a copy activity while it is in progress (multiple files in source). And also Stop/Fail the whole ADF pipeline.
Restart the pipeline. The copy activity should resume from the point of failure (It should only copy the files which are pending to be copied.)

I actually have a dashboard which will have Start and Stop buttons. Start Button will start a pipeline. Stop will stop the pipeline. Client wants control over large copy tasks from HDFS folders. A hdfs file copy task from the dash board should have the ability to Pause and Resume.

Can somebody help me in this?

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-07-22T09:08:19.733+00:00

Hi @Prith ,

Just checking in to see if the below answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.

1 answer

Your answer

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-07-22T09:08:19.733+00:00

Hi @Prith ,

Just checking in to see if the below answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.

Answer 1

Nandan Hegde 36,146 MVP Volunteer Moderator

Hey,
Based on my understanding below are the inputs:

It is not possible to juts cancel the Copy activity but we can cancel the Pipeline run via REST API:
https://learn.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/cancel

2) For the pipeline to resume from the point of failure, best way would be to archive the file after successful copy activity so only the files that needs to be copied remains in the source location.

@ShaikMaheer-MSFT @KranthiPakala-MSFT : please provide your thoughts on the same

Prith 1 Reputation point

2022-07-11T13:42:33.977+00:00
Thanks For the prompt reply !!

So, when we cancel the pipeline run, it will cancel all activities in the pipeline and stop altogether .. right. It should not wait for any activity to complete.

I cant archive source files. Any other solution to resume copy from where it stopped in the last run ?
Subashri Vasudevan 11,226 Reputation points

2022-07-11T13:56:19.99+00:00

Hi,

If you are unable to archive files, I could think of logging the details of successful file loads and do a check before loading it. If the file is loaded, mark it as completed in the control table.

And next time you pick up a file, check in the control table if it's already processed or completed.

Thanks

Share via

Stop a copy activity while it is in progress

1 answer

Your answer