Too long "Queue" time using SAP ECC connector within Data Factory

Question

Too long "Queue" time using SAP ECC connector within Data Factory

Rubén Souto 6

Greetings:

It happens occasionally, "copy data" activity gets stucked in "Queue" state for a long time. I send an example.

What can I do to troubleshoot it? Is there a solution?

Thank you.

Regards.

HarithaMaddi-MSFT 10,156 Reputation points

2020-12-16T11:57:26.053+00:00
Hi @Rubén Souto ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query. I looked at similar issues from customers and observed that one of the below reasons can be a root cause for the issue.

Upgrading IR to latest version

Increasing the "Limit Concurrent jobs" in IR configuration

If above changes did not help in resolving the issue, please share report id along with pipeline run id with us. In the integration runtime configuration manager, please click on diagnostics icon, reproduce the error and then click on ‘Send logs’. Please share report id with us to enable viewing logs.

Ref: issue

Looking forward for your response!
Rubén Souto 6 Reputation points

2020-12-16T16:46:45.33+00:00

Thank you very much, i'll try the solutions and let you know.
HarithaMaddi-MSFT 10,156 Reputation points

2020-12-22T12:18:51.58+00:00

Hi @Rubén Souto ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, let us know above details and we will continue to engage with you on the issue.
Rubén Souto 6 Reputation points

2020-12-22T14:17:24.117+00:00

Hi @HarithaMaddi-MSFT

Increasing the value of "Limit Concurrent Jobs" from 6 to 10 initially reduced a lot the number of cases, but last 2 days it's getting worse. Should I increase it more? I'm not sure about the performance issues this action could trigger.

FYI we are running around 180 "copy data" activities with SAP ECC connector every night. We use a "foreach" activity, allowing up to 3 parallel runs.

So I believe you are right pointing to a concurrency issue, but... apparently the settings are right.

Now I'm gonna update IR, activate Logging function and let you know.

Thank you very much.

1 answer

Your answer

HarithaMaddi-MSFT 10,156 Reputation points

2020-12-16T11:57:26.053+00:00

Hi @Rubén Souto ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query. I looked at similar issues from customers and observed that one of the below reasons can be a root cause for the issue.

Upgrading IR to latest version

Increasing the "Limit Concurrent jobs" in IR configuration

If above changes did not help in resolving the issue, please share report id along with pipeline run id with us. In the integration runtime configuration manager, please click on diagnostics icon, reproduce the error and then click on ‘Send logs’. Please share report id with us to enable viewing logs.

Ref: issue

Looking forward for your response!
Rubén Souto 6 Reputation points

2020-12-16T16:46:45.33+00:00

Thank you very much, i'll try the solutions and let you know.
HarithaMaddi-MSFT 10,156 Reputation points

2020-12-22T12:18:51.58+00:00

Hi @Rubén Souto ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, let us know above details and we will continue to engage with you on the issue.
Rubén Souto 6 Reputation points

2020-12-22T14:17:24.117+00:00

Hi @HarithaMaddi-MSFT

Increasing the value of "Limit Concurrent Jobs" from 6 to 10 initially reduced a lot the number of cases, but last 2 days it's getting worse. Should I increase it more? I'm not sure about the performance issues this action could trigger.

FYI we are running around 180 "copy data" activities with SAP ECC connector every night. We use a "foreach" activity, allowing up to 3 parallel runs.

So I believe you are right pointing to a concurrency issue, but... apparently the settings are right.

Now I'm gonna update IR, activate Logging function and let you know.

Thank you very much.

Answer 1

HarithaMaddi-MSFT 10,156

Thanks @Rubén Souto for sharing the update. As per the document, when the processor and available RAM aren't well utilized, but the execution of concurrent jobs reaches a node's limits, scale up by increasing the number of concurrent jobs that a node can run. If the node capacity also exceeds i.e., when issue persists after increasing the limit, then scale-out approach as suggested i.e., to increase nodes is preferred.

Looking forward for your update on the issue!

HarithaMaddi-MSFT 10,156 Reputation points

2020-12-28T12:37:46.06+00:00

Hi @Rubén Souto ,

We have not received a response from you. Please suggest if above shared details are helpful. Otherwise, let us know and we will continue to engage with you on the query.
HarithaMaddi-MSFT 10,156 Reputation points

2020-12-29T11:17:43.433+00:00

Hi @Rubén Souto ,

We still have not heard back from you. Following up to check if above shared details are helpful. Otherwise, let us know and we will continue to engage with you on the query.

Share via

Too long "Queue" time using SAP ECC connector within Data Factory

1 answer

Your answer