Data Factory: SHIR file system network issue

Riccardo Ambrosini 0 Reputation points
2025-01-21T11:09:42.8633333+00:00

Hello,
I am facing an intermitting error when using a file system dataset, configured to connect to a Self-Hosted Integration Runtime (SHIR). When I connect to the virtual machine, I usually get no error if I run the pipeline using the dataset, but if I run the pipeline after some time I have not logged into the VM, I get the following error in the pipeline:

Error code: 2011 Failure type: User configuration issue Details: Error occurred when trying to access the file in Folder '\path\to\local\filesystem', File filter: '*'.

Some details on the virtual machine for the SHIR:

  • It runs Windows Server 2022 Datacenter Azure Edition.
  • It is located in a virtual network, with no public IP, but it has access to internet.
  • I ran these commands to access the file system:
      .\dmgcmd.exe -EnableLocalMachineAccess
      .\dmgcmd.exe -DisableLocalFolderPathValidation
    

The ADF pipeline first lists the files in a directory and then it loops on the list to copy them to data lake. I get the failure while listing the files (Get Metadata activity).

In the Event Viewer on the virtual machine, I get these kind of events:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Integration Runtime (Self-hosted) Connectors" /> 
  <EventID Qualifiers="0">30000</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2025-01-21T10:13:00.3342564Z" /> 
  <EventRecordID>1460082</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Connectors – Integration Runtime</Channel> 
  <Computer>vm.com.local</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>TraceComponentId: ExecutionTask TraceMessageId: ExecutionTaskCommon @logId: Error FunctionName: ExecutionTask jobId: 5dff4f23-ff29-4925-8aff-bab79345cec8 activityId: 5dff4f23-ff29-4925-8aff-bab79345cec8 eventId: ExecutionTaskCommon message: Execution failed. Error message: 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when trying to access the file in Folder ''\\path\to\local\filesystem', File filter: '*'.,Source=Microsoft.DataTransfer.ClientLibrary,StackTrace= at Microsoft.DataTransfer.ClientLibrary.FileSourceHelper.EnumerateDirectory(DirectoryInfo dirInfo, String fileSearchPattern, Boolean recursive, Boolean fileOnly) at Microsoft.DataTransfer.ClientLibrary.AuthorizedOnPremiseFileReader.EnumerateDirectoryWithUserContext(DirectoryInfo directoryInfo, String searchPattern, Boolean recursive, Boolean fileOnly) at Microsoft.DataTransfer.ClientLibrary.AuthorizedOnPremiseFileReader.EnumerateDirectoryWithErrorHandling(DirectoryInfo directoryInfo, String searchPattern, Boolean recursive, Boolean fileOnly) at Microsoft.DataTransfer.ClientLibrary.AuthorizedOnPremiseFileReader.<>c__DisplayClass33_0.<EnumerateDirectory>b__0() at System.Threading.Tasks.Task.Execute(),''Type=System.IO.IOException,Message=The network path was not found. ,Source=mscorlib,StackTrace= at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileSystemEnumerableIterator`1.CommonInit() at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost) at System.IO.DirectoryInfo.EnumerateFileSystemInfos(String searchPattern, SearchOption searchOption) at Microsoft.DataTransfer.ClientLibrary.FileSourceHelper.EnumerateDirectory(DirectoryInfo dirInfo, String fileSearchPattern, Boolean recursive, Boolean fileOnly),'</Data> 
  </EventData>
  </Event>

I am investigating this issue with the IT department to understand if there is any issue with the firewall or other network-related problems.

Has anyone experienced similar issues? Could this be an issue with the integration runtime and the Windows VM configuration, or is it a Data Factory issue?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,216 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Amira Bedhiafi 28,381 Reputation points
    2025-01-21T13:53:40.1233333+00:00

    Your issue is either related to the network or configuration issues on the VM.

    Check if the VM firewall rules is allowing SHIR access, and verify that the service account running SHIR has necessary file system permissions.

    The idle timeouts or connection resets can be caused by network policies, so you need to verify that the dmgcmd settings persist after inactivity or reboot.

    You need to review SHIR logs (C:\Program Files\Microsoft Integration Runtime\4.0\Logs) and Event Viewer for detailed errors. If the problem persists, consider using an Azure-hosted runtime or alternative file storage solutions like Azure File Share.


  2. Pinaki Ghatak 5,570 Reputation points Microsoft Employee
    2025-01-27T10:15:13.86+00:00

    Hello @Riccardo Ambrosini

    The error message indicates that the network path was not found. This could be due to a network-related problem, such as a firewall issue or a problem with the SHIR configuration.

    To troubleshoot this issue, you can try the following steps:

    1. Check the firewall settings on the SHIR virtual machine to ensure that it allows traffic from the Data Factory service.
    2. Check the network connectivity between the SHIR virtual machine and the Data Factory service. You can try pinging the Data Factory service from the SHIR virtual machine to see if there is any network connectivity issue.
    3. Check the SHIR configuration to ensure that it is correctly configured to access the file system. You can try running the pipeline again after logging into the SHIR virtual machine to see if the issue persists.
    4. Check the event viewer on the SHIR virtual machine for any additional error messages that might provide more information about the issue.

    I hope this helps.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.