Azure Data factory V2 for each loop run twice in pipeline with couple of seconds delay
Hi,
We had one incident in our production environment of Azure data factory v2 instance, where for-each iterator loop run twice in one single run of pipeline with some couple of seconds delay.
In foreach loop iterator, we are doing some data load steps based on some criteria. Due to loop running twice, Data copy activities has duplicated data into target tables.
Please let me know, any details you need to investigate further.
Azure Data Factory
-
KranthiPakala-MSFT 46,422 Reputation points • Microsoft Employee
2020-09-29T18:43:21.477+00:00 Hi @Mansing ,
Welcome to Microsoft Q&A platform and thanks for using this forum.
Could you please confirm if you have noticed this behavior for the first time(only once) or it happened before as well? Also please confirm if
isSequential
is set tofalse
ortrue
?Please do share your below details so that we can try to investigate on logs with internal team.
- Pipeline runID?
- Both ForEach activities runID's?
- Data factory region?
Looking forward to your response.
Thank you -
Mansing 1 Reputation point
2020-09-30T09:20:13.757+00:00 Hi Kranthi
Yes, We have observed this behavior only once.
IsSequential setting is set to false with Batch count of 10.Pipeline Run Id : 2232bc2d-a780-4a03-a0e8-d73afe62bfe6
ForEach Activity Run ID's :
98db839f-4def-41c8-b2b2-c21341a97c1a
27ff8ca7-7a21-45e2-b4aa-37252798e585Data Factory Region : North Europe
Mansing
-
KranthiPakala-MSFT 46,422 Reputation points • Microsoft Employee
2020-10-01T21:32:39.747+00:00 Hi Mansing,
Thanks for your response. I checked with internal team based on the runId's provided but we weren't able to find any info for those runID's. Could you please double check if the runID's provided are correct and related to the original issue of this post? Also have you got a chance to check what is the input array values passed to ForEach activity for iteration and evaluate the flow inside forEach activity?
To further investigate on this issue, please do share the exact runID's related to the issue.
Looking forward to your response
Thank you -
Mansing 1 Reputation point
2020-10-02T10:51:24.55+00:00 Hi Kranti
The details I provided are correct. Pls see image below.
Issue happned on 9/2/2020 7:00:51 PM(UTC time) -
KranthiPakala-MSFT 46,422 Reputation points • Microsoft Employee
2020-10-02T19:43:52.257+00:00 Hi @Mansing ,
Thanks for your response and confirmation. I tried to find the logs using provided details but still no luck.
For deeper investigation and immediate assistance, If you have a support plan you may file a support ticket, else could you please send an email to AzCommunity[at]Microsoft[dot]com with the below details.Email subject: <Attn - Kranthi : Microsoft Q&A Thread title>
Thread URL: <Microsoft Q&A Thread URL>
Subscription ID: <your subscription id>Looking forward to your response.
Thank you
-
Mansing 1 Reputation point
2020-10-02T20:01:41.81+00:00 Hi Kranti,
I have sent email as you have requested.
Thanks
Mansing -
KranthiPakala-MSFT 46,422 Reputation points • Microsoft Employee
2020-10-02T20:17:34.367+00:00 Hi @Mansing ,
Could you also please confirm if you are having 2 lookup activities (
CleanTemporaryWaterMarkTable
&LookupOldWaterMark
) inside your Foreach activity (IterateSQLTables
) ? Also is this a child pipeline being executed by a different parent pipeline (using Execute pipeline activity)?Looking forward to your response.
Thanks -
Mansing 1 Reputation point
2020-10-03T08:21:25.753+00:00 Hi Kranti,
BElow image is complete steps of activities inside ForEach-Loop.
'Also, yes the pipeline is getting executed as child pipeline from parent pipeline.
Parent Pipeline Run ID : 20771d58-068c-42df-9449-8db8063ddb7c
Child Pipeline Run ID Having Issues : 2232bc2d-a780-4a03-a0e8-d73afe62bfe6 -
KranthiPakala-MSFT 46,422 Reputation points • Microsoft Employee
2020-10-05T18:59:08.9+00:00 Thank you for the additional details @mansin.
Please follow the instructions provided in email to file a support request for deeper investigation on this issue. And please do share the SR# with me so that I can the track the case internally.
Looking forward to your response.
Thank you -
KranthiPakala-MSFT 46,422 Reputation points • Microsoft Employee
2020-10-09T21:37:04.297+00:00 Hi @mansin.
Following up to check if you have got a chance to file a SR regarding this issue. If so could you please share the SR# with us for tracking.
Also once the SR resolves the issue, please do consider to post the root cause and resolution details here, so that it would be helpful for other community members who experience similar issue.Thank you
-
Mansing 1 Reputation point
2020-10-11T16:50:51.777+00:00 Hi Kranti,
Sorry for late response. Service ticket is raised and we are in touch with support guy to get feedback on issue. I have provided all support log files as requested.
Mansing
Sign in to comment