This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 87%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBF%3C/text%3E%3C/svg%3E)
I've been noticing an issue over the past week where pipeline activities are being executed twice, with no apparent reason based on the pipeline design. For example, I have a pipeline that simply calls two sub-pipelines sequentially:
For some reason, the IDL_WZN_Ingest_Hourly_Files pipeline gets executed twice in parallel. This problem seems to occur transiently. I've also noticed it this morning with a different pipeline, and in that case there are GetMetadata, Filter, ForEach and ExecutePipeline activities that are all getting executed twice in the same pipeline run, staggered by different amounts.
Is this a bug in ADF? If so, it seems like a major bug, and I don't see how I could design around it.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
Thanks @Ben Forbes for patiently waiting for the fix to be deployed. Our Product team confirmed that the bug fix is deployed, please suggest if you are still experiencing the issue and we will be glad to assist further.
Thanks @HarithaMaddi-MSFT , I haven't seen the issue for almost a week, so it looks like the fix is good. I'll keep an eye out just in case. Appreciate your assistance!
Hi @Ben Forbes ,
Welcome to Microsoft Q&A Platform. Thanks for posting the issue.
Please suggest if the "Wait on completion" flag in "Execute Pipeline" activity is selected for each pipeline. If it is not selected, pipeline will be triggered and execute pipeline task will be immediately completed. Hence, before actual pipeline execution is completed, next pipeline task is triggered and it will cause both the pipelines to run in parallel. Please find below GIF showing the same. Also, if this option is enabled and still facing the issue, please confirm if this is transient issue. Kindly share the pipeline run id, region of data factory for us to investigate further if you are able to experience it multiple times.
Hope this helps! Please let us know if our understanding is incorrect or for further queries and we will be glad to assist.
Hi @HarithaMaddi-MSFT ,
Thanks for your response. I can confirm that the "Wait on completion" flag is selected for both activities.
This is indeed a transient issue; the same pipelines under similar conditions typically complete successfully.
The data factory is in the East US region, and the pipeline run IDs where I have observed the problem are the following:
c5f5fd14-16dc-441e-98d8-44b8d41d138b
eee4668d-4336-4eca-a973-689f138b4a59
6bb6aba5-6872-418d-bb2e-2a7db8f5b529
The same problem but in a different pipeline (this time GetMetadata, Filter, ForEach and ExecutePipeline activities):
20b476dc-9968-433f-90ff-989fe4381e58
03584ab5-8824-46b3-9dd1-547c1f6c224e
Many thanks for your assistance
Ben
Apologies @Ben Forbes for the inconvenience caused. I saw that this issue was reported a while back to Product team and marked as resolved. I am working with internal teams to understand the details and I will get back to you once I hear back from them.
Thanks for your patience!
Thanks @HarithaMaddi-MSFT . Is there any support ticket or documentation you can point me to so I can get more context on the problem?
I've seen it happen again this morning in a different pipeline:
fbbedb0a-f9e7-439e-8f30-84e9ae05a605
EDIT:
Another one this morning:
5d2e57d6-eb21-47f0-9a34-ad806195a953
This is becoming really problematic. I'm concerned it may cause data integrity issues.
Hi @HarithaMaddi-MSFT , as another interesting piece of information, where pipeline run ID 20b476dc-9968-433f-90ff-989fe4381e58 had spurious activity executions, a copy activity within one of those executions seems to have behaved strangely.
There is a column within a source file that normally contains a datetime, but sometimes NULL. This pipeline normally handles the NULLs fine, but on this particular run yesterday morning, it spat the error "Conversion failed when converting date and/or time from character string."
I assume this is somehow caused by the problem with activities executing twice, but I'm not exactly sure how.
Thanks @Ben Forbes for sharing the details. I got confirmation from Product team that it is a bug where internal server errors can cause the same pipeline to be executed twice with the same run id. There is a potential fix that is ready and is getting deployed this week (Expected to be deployed in EastUs by end of this week). We are very sorry for the trouble. I will share update in early next week to confirm the fix.
Thanks for your patience!
Thanks @HarithaMaddi-MSFT , I appreciate this being addressed. Looking forward to the fix!
Regards,
Ben