Sounds like some kind of memory leak or orphan processes that are not terminating.
On the file server, run computer management and check for sessions and open files from the server that hosts SQL,
On the SQL server, run process explorer.
https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer
In the View menu set the lower pane view to show handles and the update speed to 10 seconds. Find the SS Agent process. Processes will be indented for parent/child. Does the SS Agent have multiple sub processes running? Search the lower pane and look for file handles referencing the file server.