Integration runtime is failing

Question

Hi Team,

we are unable to run the pipeline using Microsoft Integration Runtime and see following errors.

|DEBUG: System.OutOfMemoryException: Exception of type
'System.OutOfMemoryException' was thrown. at
Microsoft.DataTransfer.ClientLibrary.ExceptionHelper.GenerateException(Exception
rawException, BoundedDataArtifact sourceDA) at
Microsoft.DataTransfer.TransferTask.CopyTaskBase.Execute() at
Microsoft.DataTransfer.TaskHosting.ThreadTaskWorker.<>c__DisplayClass32_0.<RunTask>b__0()| | -------- | |DEBUG: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at Microsoft.DataTransfer.ClientLibrary.ExceptionHelper.GenerateException(Exception rawException, BoundedDataArtifact sourceDA) at Microsoft.DataTransfer.TransferTask.CopyTaskBase.Execute() at Microsoft.DataTransfer.TaskHosting.ThreadTaskWorker.<>c__DisplayClass32_0.<RunTask>b__0()|

|Failed to connect to worker pool manager: . Url: Activity ID:
b82c8ce0-ba9c-412b-9f48-429c072e071d| | -------- | |Failed to connect to worker pool manager: . Url: Activity ID: b82c8ce0-ba9c-412b-9f48-429c072e071d|

|Exception of type 'System.OutOfMemoryException' was thrown. Exception of type
'System.OutOfMemoryException' was thrown.| | -------- | |Exception of type 'System.OutOfMemoryException' was thrown. Exception of type 'System.OutOfMemoryException' was thrown.|

Answer 1

Hi Riyaz Ahmed Khaja Hussain,

To address the OutOfMemoryException error with your Microsoft Integration Runtime (SHIR), scaling up and scaling out can indeed be performed on your existing SHIR VM. Here are some steps and considerations:

1. Scale Up:
   • Increase VM Size: Upgrade your VM to a larger size with more CPU and RAM. This can help handle larger workloads and prevent memory issues.
2. Scale Out:
   • Add More Nodes: You can add additional nodes to your SHIR setup. This distributes the load across multiple machines, improving performance and reliability.
   • Kubernetes Support: If you're using a Kubernetes cluster, you can leverage it to scale out your SHIR. This involves setting up SHIR on multiple nodes within the cluster3.
   Configuration Adjustments:
   • Reduce Concurrent Runs: Lower the number of concurrent runs or the load on each node to manage memory usage better.
   • Adjust ForEach Activity: If your pipeline includes a ForEach activity, reduce the batch size to a lower value to avoid overloading the system.

I hope these tips help resolve the issue. If you need further assistance, I'm here to help.

Best regards, Jonathan

-------------------

Your feedback is very important to us! If this response resolved your query, please click 'YES'. This helps us continuously improve the quality and relevance of our solutions.