Can we restart the ADF pipeline from the point of failure of record or files
Hi We have a copy activity for which we put retry option I have 10 files in container folder and in middle copy failed after copy 4files and at 5th file Will retry option copies from file1 and re copy from beginning or resume from failure like copying from 5th file?? Note: Dataset type Binary files
When you use the retry option in a copy activity, it will resume copying from the point of failure. If the copy activity failed after copying 10 files, it will resume copying from the 11th file.
Regarding the comment you mentioned, it is true that the copy activity can resume from an arbitrary number of copied files for some connectors such as Amazon S3, Azure Blob, Azure Data Lake Storage Gen2, and Google Cloud Storage. However, for the rest of the file-based connectors as source, the copy activity supports resuming from a limited number of files, usually at the range of tens of thousands and varies depending on the length of the file paths.
please go through: https://winscp.net/eng/docs/resume
please try to test this by running the copy activity with a small number of files and checking the logs to see which files were copied.
Do let us know if you have any further questions.
Sign in to comment