Synapse Pipeline - Inner activity in Succeeded state, outer activity in progress

Question

Synapse Pipeline - Inner activity in Succeeded state, outer activity in progress

Artem Shupenko 5

I have a problem with Synapse Pipelines where the inner activity is in the Succeeded state while the outer one stuck 'In progress'. This problem has been occurring frequently in various environments, locations, IR and activities in the past week, and I haven't been able to identify any consistent pattern. The attached screenshots just as one example. Any assistance would be greatly appreciated. User's image

User's image

Vinodh247 41,291 Reputation points MVP Volunteer Moderator

2023-05-10T03:59:47.3466667+00:00

I think the outer activity you are saying here if the notification activity, right? If yes, the issue is it's taking time to complete the notification though the main activity if getting succeeded earlier. Can you check?
Artem Shupenko 5 Reputation points

2023-05-10T09:05:52.32+00:00

Hi! No, the outer activity is update__file__watermark (Execute pipeline) it stuck in Progress, inside we have SQL stored procedure (completed successfully) and notif activity will work only if SQL pool sp is failed (not our case).
KranthiPakala-MSFT 46,822 Reputation points Microsoft Employee Moderator

2023-05-11T22:56:43.0933333+00:00

@Artem Shupenko Thanks for using Microsoft Q&A forum and posting your query.

The behavior is very strange. Ideally, when the inner update_watermark SP Activity succeeded, then the status of that child pipeline to be succeeded and likely the outer activity in the parent pipeline should be succeeded.

Could you please confirm if the child pipeline which is having update_watermark SP Activity was succeeded or still in progress? The outer activity in the parent pipeline will move forward only if the child pipeline is completed. Kindly share few additional screenshots showing the pipeline details as well.

In case if the child pipeline's activities are succeeded but the child pipeline status is still in-progress then there is some issue with that pipeline and needs deeper investigation.

Let us know what you noticed.

Thanks
Artem Shupenko 5 Reputation points

2023-05-12T11:00:14.7166667+00:00
Hi! We cannot check the result of the activity in a database now, I suppose it was actually completed, please see below what we have in the update_watermark 's output. The problem is that it shows activity state as Completed in the UI. After we canceled the pipeline manually , outer pipeline changed its state to Cancelled, but inner pipeline remained Succeeded state.

{ "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (Central US)", "executionDuration": 0, "durationInQueue": { "integrationRuntimeQueue": 1 }, "billingReference": { "activityType": "ExternalActivity", "billableDuration": [ { "meterType": "ManagedVNetIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }
KranthiPakala-MSFT 46,822 Reputation points Microsoft Employee Moderator

2023-05-12T23:02:24.5233333+00:00

@Artem Shupenko Thanks for your response.

I meant to ask if the status of child pipeline that is having the update_watermark is succeeded or in-progress? This ask is about the pipeline status not the stored proceedure activity status. If the pipeline status is still in progress, then the outer activity from Parent pipeline will be in progress. First the child pipeline status should be changed to succeeded. Irrespective, if any of the activities or pipeline status are in progress for a longer period of time, that means there is something wrong happening and this kind of issues require deeper investigation. If you have a support plan, please file a support ticket for deeper investigation. In case if you don't have a support plan, please let me know here.

Thank you
KranthiPakala-MSFT 46,822 Reputation points Microsoft Employee Moderator

2023-05-17T22:17:17.3266667+00:00

@Artem Shupenko Checking to see if you have got a chance to file a support ticket for deeper investigation. If so, could you please help share the SR# here with us so that we can track it internally. In case if you are still blocked and no support plan, please let us know.

Thanks

Your answer

Vinodh247 41,291 Reputation points MVP Volunteer Moderator

2023-05-10T03:59:47.3466667+00:00

I think the outer activity you are saying here if the notification activity, right? If yes, the issue is it's taking time to complete the notification though the main activity if getting succeeded earlier. Can you check?
Artem Shupenko 5 Reputation points

2023-05-10T09:05:52.32+00:00

Hi! No, the outer activity is update__file__watermark (Execute pipeline) it stuck in Progress, inside we have SQL stored procedure (completed successfully) and notif activity will work only if SQL pool sp is failed (not our case).
KranthiPakala-MSFT 46,822 Reputation points Microsoft Employee Moderator

2023-05-11T22:56:43.0933333+00:00

@Artem Shupenko Thanks for using Microsoft Q&A forum and posting your query.

The behavior is very strange. Ideally, when the inner update_watermark SP Activity succeeded, then the status of that child pipeline to be succeeded and likely the outer activity in the parent pipeline should be succeeded.

Could you please confirm if the child pipeline which is having update_watermark SP Activity was succeeded or still in progress? The outer activity in the parent pipeline will move forward only if the child pipeline is completed. Kindly share few additional screenshots showing the pipeline details as well.

In case if the child pipeline's activities are succeeded but the child pipeline status is still in-progress then there is some issue with that pipeline and needs deeper investigation.

Let us know what you noticed.

Thanks
Artem Shupenko 5 Reputation points

2023-05-12T11:00:14.7166667+00:00

Hi! We cannot check the result of the activity in a database now, I suppose it was actually completed, please see below what we have in the update_watermark 's output. The problem is that it shows activity state as Completed in the UI. After we canceled the pipeline manually , outer pipeline changed its state to Cancelled, but inner pipeline remained Succeeded state.

{ "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (Central US)", "executionDuration": 0, "durationInQueue": { "integrationRuntimeQueue": 1 }, "billingReference": { "activityType": "ExternalActivity", "billableDuration": [ { "meterType": "ManagedVNetIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }
KranthiPakala-MSFT 46,822 Reputation points Microsoft Employee Moderator

2023-05-12T23:02:24.5233333+00:00

@Artem Shupenko Thanks for your response.

I meant to ask if the status of child pipeline that is having the update_watermark is succeeded or in-progress? This ask is about the pipeline status not the stored proceedure activity status. If the pipeline status is still in progress, then the outer activity from Parent pipeline will be in progress. First the child pipeline status should be changed to succeeded. Irrespective, if any of the activities or pipeline status are in progress for a longer period of time, that means there is something wrong happening and this kind of issues require deeper investigation. If you have a support plan, please file a support ticket for deeper investigation. In case if you don't have a support plan, please let me know here.

Thank you
KranthiPakala-MSFT 46,822 Reputation points Microsoft Employee Moderator

2023-05-17T22:17:17.3266667+00:00

@Artem Shupenko Checking to see if you have got a chance to file a support ticket for deeper investigation. If so, could you please help share the SR# here with us so that we can track it internally. In case if you are still blocked and no support plan, please let us know.

Thanks

Share via

Synapse Pipeline - Inner activity in Succeeded state, outer activity in progress

Your answer