SQL2019: Python: The process cannot access the file because it is being used by another process

Question

Hi,

I have a SQL Server 2017 Developer (14.0.3356.20 (X64)) and a SQL Server 2019 Developer (15.0.4083.2 (X64)) with Machine Learning (R+Python) installed.
Both are installed on different machines
On both sql-servers is Python 3.7.1 installed
On both servers is the same database
On both servers are the same python packaged installed, i.e. same version

When I run my python script via sp_execute_external_script it succeeds without erros on the SQL Server 2017 machine.
When I run the same script on the SQL2019 I get the following error.

any help is welcome.
If necessary, I can also provide the packages and the python/sql code

Msg 39004, Level 16, State 20, Line 4
A 'Python' script error occurred during execution of 'sp_execute_external_script' with HRESULT 0x80004004.
Msg 39019, Level 16, State 2, Line 4
An external script error occurred: 

Error in execution.  Check the output for more information.
Traceback (most recent call last):
  File "<string>", line 5, in <module>
  File "C:\ProgramData\SQL2019\Temp-PY\Appcontainer1\08E40134-42AA-4185-A965-A1BC1650525E\sqlindb_0.py", line 140, in transform
    clustered_dupes = deduper.partition(data_d, 0.5)
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\site-packages\dedupe\api.py", line 175, in partition
    clusters = list(clusters)
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\site-packages\dedupe\api.py", line 190, in _add_singletons
    for record_ids, score in clusters:
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\site-packages\dedupe\api.py", line 322, in cluster
    yield from clustering.cluster(scores, threshold)

Msg 39019, Level 16, State 2, Line 4
An external script error occurred: 
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\site-packages\dedupe\clustering.py", line 219, in cluster
    preserve_input=True)
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\site-packages\fastcluster.py", line 247, in linkage
    linkage_wrap(N, X, Z, mthidx[method])
TypeError: argument 2 must be numpy.ndarray, not numpy.ndarray

SqlSatelliteCall error: Error in execution.  Check the output for more information.Traceback (most recent call last):
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\weakref.py", line 624, in _exitfunc
    f()
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\weakref.py", line 548, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\tempfile.py", line 795, in _cleanup

Msg 39019, Level 16, State 2, Line 4
An external script error occurred: 
    _shutil.rmtree(name)
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\shutil.py", line 507, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\shutil.py", line 391, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\shutil.py", line 389, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\ProgramData\\SQL2019\\Temp-PY\\Appcontainer1\\08E40134-42AA-4185-A965-A1BC1650525E\\tmp_ebbw_2p\\unlabeled_edgelist'

STDOUT message(s) from external script: 
True
reading from c:\temp\dedupe\sql_example_settings.txt
clustering...
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 605, in rx_sql_satellite_call
    rx_native_call("SqlSatelliteCall", params)
  File "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 375, in rx_native_call
    ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.

Answer 1

The process cannot access the file because it is being used by another process.

Some other process has a lock on the file and it conflicts with the lock being asked for (Windows not SQL) by whatever SQL wants to do. You'll need to see who has a lock on it, there are various tools which can be used such as procmon, procexp, handles, etc, from sysinternals.

Answer 2

Hi together,

thanks for your help.
I startet procmon while running the script and filtered it on the filename "unlabeled_edgelist" here is the result of procmon

we have already stopped ntrtscan.exe but that doesn't change anything

Answer 3

At the time of the screenshot, the antivirus was started again. But we have also done some tests without runnning antivirus
I'll ask the developers of the python code if they have some ideas.
From my side you can close this request.

Thanks for your help

Answer 4

sorry for the delay. Other things crossed my way.
We disabled ntrtscan completely, but the error remains.
I will try to find a different solution.
You can close this request.

Thanks for your help
Marco