Azure Data Factory Pipeline Activities Stuck In Queued status

Frank GC 1

We have been experiencing a number of failed jobs over the last week because of pipeline activities becoming stuck in a queued state. These seemed to have been working until we upgraded to the latest IR version. An example pipelinerunID is c560992a-d974-4806-9c39-1b6b71e5c7f4. We have looked at the IR server and nothing seems to be wrong but we can't figure out why activities keep becoming stuck and won't start.

PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2021-03-18T11:01:47.823+00:00

Hello @Frank GC ,

Thanks for the ask and also for using the Microsoft Q&A forum.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.

1 answer

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-19T14:11:36.267+00:00

Hi @Frank GC ,

Sorry for your experience with this issue and thanks for reporting this to us.

I have verified with internal product team and confirmed that there is a potential defect identified in the newer SHIR version (5.2 and above), which could cause the SHIR's capacity degradation under big loads. This might be the reason you are seeing this issue.

For mitigation, a QFE SHIR is available in download center now. Product team have already sent a notification to the potential impacted customers with the mitigation steps and QFE download link.

Here is the link: https://download.microsoft.com/download/E/4/7/E4771905-1079-445B-8BF9-8A1A075D8A10/IntegrationRuntime_5.2.7740.4.msi. Please download and do the manually update, in the meanwhile a new round of auto update is also in progress by the product team.

Apologizes for the inconvenience due to this issue.

Let us know how it goes.
Please sign in to rate this answer.

1 person found this answer helpful.
Frank GC 1 Reputation point

2021-03-19T15:48:11.693+00:00

Thank you for the info. I will share this with our internal teams.

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-19T18:01:20.397+00:00

Thanks for your response @Frank GC . Do let us know if you notice any issues after installing the suggested SHIR version.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Frank GC 1 Reputation point

2021-03-19T19:37:05.897+00:00

We were seeing a lot of odd network activity during this period. Is there a way to get more details about the bug so we can understand if the network activity we are seeing was impacted?

Tp 1 Reputation point

2021-03-23T12:13:23.73+00:00

When I try to install the QFE it complains that "The same or newer version already installed". We're running 5.3.77287.2. All the jobs that should be run on that IR are stuck in a queue.

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-23T23:53:49.993+00:00

@Tp - Thanks for reaching out. Could you please try uninstalling the the version you are having and try to manually download the QFE version and then install it.

Let us know how it goes.

Thanks

Tp 1 Reputation point

2021-03-24T07:08:27.223+00:00

Restart of the IR service solved the problem for now. I'll look into uninstall option if needed.

Can you give any estimate as for when a proper fix will be issued, we might be able to wait for that?

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-25T14:36:25.333+00:00

Hi @Tp , Glad to know that the issue was resolved. Product team confirmed that SHIR 5.2.7740.4 is a stable version and if you have auto update option enabled in SHIR configuration settings, then your SHIR will auto update to latest stable version which has a fix. I think you should be good now. Please feel free to reach out if you notice any strange behavior.

Thank you

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-30T01:16:09.823+00:00

Hi @Tp ,

Following up to see if the info info answers your query. Do let us know if you have further query.

Thanks
Sign in to comment