Self-Hosted IR (SHIR) takes ~10 minutes to pick up Data Flow task

Question

I am running an Azure Data Factory Data Flow that utilizes a Self-Hosted Integration Runtime (SHIR) to copy/stage data from SAP CDC (ODP). I am experiencing a consistent delay of roughly 10 minutes between the time the ADF pipeline requests the job and the time the SHIR actually starts executing it.

Detailed Timeline Analysis (from SHIR Event Logs):

10:16:59 ADF Pipeline starts the Data Flow activity. (Approx. 2 mins for Azure Cluster provisioning).

10:19:17 First message came to SHIR

Start to initialize EC context.

10:19:18 Plugin loading begins:

[PrepareConnectionProperties] Plugin is 'Microsoft.DataTransfer.Extensibility.ConnectorWrapper'

... load load plugins ...

10:19:23 AdlsGen2 we are using, great.

Loaded ADLS plugin AdlsGen2CosmosStructuredStream

10:19:23 ... load load plugins ...

[Connector SxS] Plugin Cassandra already be static loaded in plugins list, we have replaced it with dynamic loaded connector

...and many many other plugins...

10:20:12 SAP odp, that's what we are using, great.

[PrepareConnectionProperties] Plugin is 'Microsoft.DataTransfer.Runtime.SapOdp.SapOdpConnector'

10:20:13 SHIR successfully contacts SAP:

Start GetOdpDetailsFunction RODPS_REPL_ODP_GET_DETAIL with context ABAP_CDS, subscriber type RSODP_ODATA, odp name ZODP_VBAP$E

10:20:16 Strange Behavior: SHIR appears to restart loading plugins again...

Loaded ADLS plugin AdlsGen2CosmosStructuredStream from the assembly

10:20:16 During this delay, I see logs indicating the SHIR is loading every available connector assembly, including failures for connectors I don't use... Cassandra for example...

[Connector SxS] Plugin Cassandra already be static loaded in plugins list, we have replaced it with dynamic loaded connector

10:20:17 LOG GAP STARTS HERE:

KeepAlive:False, minWorkerThreads:4, minIOThreads:4 loggerName: TaskHosting traceComponent: ThreadWorkerManager @logId: Information @level: Information @eventId: 30009 recordTime: 2/5/2026 9:20:17 AM region: serviceVersion: serviceName: roleInstanceName: roleInstanceGroup: {"AksSubscriptionId":"","ClusterName":"","NodeName":""} aksScaleUnit: @detail: <Detail><message>KeepAlive:False. minWorkerThreads:4. minIOThreads:4</message><loggerName>TaskHosting</loggerName><traceComponent>ThreadWorkerManager</traceComponent><recordTime>2/5/2026 9:20:17 AM</recordTime><region></region><serviceVersion></serviceVersion><serviceName></serviceName><roleInstanceName></roleInstanceName><roleInstanceGroup>{"AksSubscriptionId":""."ClusterName":""."NodeName":""}</roleInstanceGroup><aksScaleUnit></aksScaleUnit><activityId>00000000-0000-0000-0000-000000000000</activityId></Detail>

• The service appears to go idle or hang here.

10:26:32 6 MIN GAP!!! And this gap doesn't happen everytime on sapnco load. It can be also PulledOffNewTask

Load 3.0 assembly sapnco, version=3.0.0.42, culture=neutral, publickeytoken=50436dca5c7f7d23failed, try to load 3.1 instead

10:28:19 Job finally starts:

TaskMaxRetryCount:10 Job ID: 29631033-b990-c143-9fcd-b0bbaa8d6e6b Task ID: 29631033-b990-c143-9fcd-b0bbaa8d6e6b Queue ID: 885b707c-f1f1-44bc-b1ad-28cca8552551 Log ID: PulledOffNewTask

10:28:25 Started coping stream

Allocated memory limit 492830720 for stage Serialization Job ID: 29631033-b990-c143-9fcd-b0bbaa8d6e6b Log ID: Information

10:28:28 Logs to SAP:

Start GetOdpDetailsFunction RODPS_REPL_ODP_GET_DETAIL with context ABAP_CDS, subscriber type RSODP_ODATA, odp name ZODP_VBAP$E

10:28:35 AM Task completed.

Environment:

• OS: Windows Server
• Source: SAP ODP
• Security: Trend Micro Deep Security / Vision One
• SHIR Version: 5.0

Questions:

1. Is it normal for the SHIR to unload all connector plugins when idle (KeepAlive:False)?
2. Can antivirus be an issue?
3. Is there a configuration setting to force the SHIR to keep these assemblies in memory so we don't pay this "Cold Start" penalty on every run?
4. What is this 6-min time waiting?

Any insights on reducing SHIR startup latency would be appreciated.

Answer 1

Hi Hanna Melashkina

The delay occurs during the cold-start initialization of the Self-Hosted Integration Runtime (SHIR). When the SHIR worker process becomes idle, it is unloaded. On the next pipeline run, SHIR must fully reinitialize the worker process, which includes loading all connector assemblies again, including the SAP ODP connector and SAP .NET Connector (SAP NCo) libraries.

Your logs confirm this behavior. The entry KeepAlive: False, followed by a long pause and then retries to load SAP NCo assemblies, shows that the worker process was recycled and had to reload its plugins. This lifecycle behavior is expected for SHIR and is documented by Microsoft.

The long “silent” gap you see is not caused by Azure Data Factory or SAP. It is caused by endpoint security software (Trend Micro Deep Security / Vision One) scanning DLLs during SHIR startup. When SHIR dynamically loads .NET and native assemblies, antivirus software may intercept and scan these files, which can delay startup by several minutes. This explains why the delay is intermittent and why it appears during SAP NCo loading.

To resolve the issue, antivirus exclusions must be added for the SHIR installation directories and SAP NCo libraries. Once these exclusions are in place and the SHIR service is restarted, the cold-start delay is typically eliminated.

Recommended antivirus exclusions:

C:\Program Files\Microsoft Integration Runtime\
C:\Program Files\Microsoft Integration Runtime\5.0\
C:\Program Files\Microsoft Integration Runtime\5.0\Connectors\

Recommended file exclusions:

sapnco.dll
sapnco_utils.dll
Microsoft.DataTransfer.*.dll

After applying these exclusions, restart the antivirus service and the SHIR service (or reboot the VM).

There is no supported configuration to force SHIR to keep connector assemblies permanently loaded in memory. The correct and supported solution is to prevent antivirus interference during initialization.

The delay happens during SHIR cold-start, not during SAP processing.
SHIR unloading and reloading connectors when idle is expected behavior.
Antivirus scanning is causing the multi-minute delay.
Adding exclusions for SHIR and SAP NCo resolves the issue.

Note: If this didn't resolve please upgrade the SHIR to latest version.

References:
https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-troubleshoot-guide?tabs=data-factory
https://helpcenter.trendmicro.com/en-us/article/tmka-18608

Hope this helps, Please let us know if you have any questions and concerns.