Microsoft Integration Runtime generating many event log

Question

I noticed the Microsoft Integration Runtime is consuming a lot of CPUs and generating many trace message in the event log non stop in my server.

IR version : 5.58.9391.1 (Data Factory V2)

Sample Event Log :

Please advise what are the issues and whether there is an option to turn off the trace to reduce the CPUs usage.

Answer 1

The high CPU and non-stop trace messages usually point to one thing: the IR is stuck in a very active loop, often trying to process a very large number of files or handle a complex data flow that's taking much longer than expected.

While turning off the traces might seem like a solution, it's like turning off a fire alarm while the kitchen is full of smoke. The logs are our best clue to find the root cause. Instead, let's try to identify and fix the workload that's causing the strain.

Here’s a simple plan to start troubleshooting:

Check the Data Factory Pipeline: Log into your Azure Data Factory portal. Can you identify which pipeline or data flow is currently running on this specific IR? Sometimes, stopping and reconfiguring a problematic pipeline can immediately resolve the loop.

Analyze the Source Data: Is the Integration Runtime reading from a folder with an extremely large number of small files (e.g., millions of tiny CSV or JSON files)? This is a very common cause for this behavior, as the overhead of processing each file individually can overwhelm the IR.

Scale or Restart: As a immediate step, you can try restarting the IR service on your server. This will cancel any stuck tasks and give you a clean slate. If the high CPU returns quickly, it confirms an active pipeline is the trigger. If your workload is consistently high, you might need to consider scaling up to a more powerful IR node.

Let's start by checking what's running in Data Factory. Finding the specific pipeline will point us directly to the solution.

Hope this helps you get started :)