Share via

SQL Server 2022 - Launchpad - how to eliminate errors in the logs "Failed to delete directory..."

RandomGuy123 40 Reputation points
2024-07-25T08:45:40.51+00:00

Hi,

I'm unable to find a solution to this anywhere and I'm desperate. Perhaps someone faced a similar issue and managed to find a solution. I'm running out of hairs on my head to pull them out.

I got installed SQL Server 2022 with latest CU on Windows Server 2022.

And my dev team is using SQL Server Machine Learning / SQL Server Launchpad for running R scripts / python code. The same issue happens on two different servers.

I've checked every piece of documentation, requirement, permissions, suggestions, even tried desperate attempts like temporarily add NT Service\MSSQLLaunchpad to Administrators to verify if that's not the permission issue, but same error persist.

In latest SQL Server versions, security design has changed and from what I saw Machine Learning is now using Appcontainer isolation. Many of the old problems/solutions do not apply to the recent versions.

In my case, so far most of the things work fine, except the fact that SQL Server Launchpad bloats error logs with thousands of error messages like the errors at the bottom of below log (Extlaunchererrorlog):

2024-07-24 15:04:32.570	Page exclusion bitmap is enabled.
2024-07-24 15:04:32.611	StartServiceDispatcherTask Completed. ErrorCode: 0x00000000
2024-07-24 15:04:32.611	Privilege SeAssignPrimaryTokenPrivilege is present in launchpad process token.
2024-07-24 15:04:32.611	Privilege SeChangeNotifyPrivilege is present in launchpad process token.
2024-07-24 15:04:32.611	Privilege SeIncreaseQuotaPrivilege is present in launchpad process token.
2024-07-24 15:04:32.612	Launchpad process is a part of Service Group. Launchpad does have SeServiceLogonRight.
2024-07-24 15:04:32.612	StartCheckpointingThread Completed. ErrorCode: 0x00000000.
2024-07-24 15:04:32.633	Negotiated Runtime Version for launcher RLauncher.dll is 4
2024-07-24 15:04:32.633	Got API for launcher RLauncher.dll with version 4
2024-07-24 15:04:32.633	Successfully loaded launcher RLauncher.dll and negotiated version
2024-07-24 15:04:32.642	Successfully initialized launcher RLauncher.dll
2024-07-24 15:04:32.642	Successfully found the launcher interfaces for launcher RLauncher.dll
2024-07-24 15:04:32.642	Successfully registered the launchcontext for launcher RLauncher.dll
2024-07-24 15:04:32.642	Successfully registered the launcher RLauncher.dll for the scriptType 1
2024-07-24 15:04:32.660	Negotiated Runtime Version for launcher Pythonlauncher.dll is 4
2024-07-24 15:04:32.660	Got API for launcher Pythonlauncher.dll with version 4
2024-07-24 15:04:32.660	Successfully loaded launcher Pythonlauncher.dll and negotiated version
2024-07-24 15:04:32.669	Successfully initialized launcher Pythonlauncher.dll
2024-07-24 15:04:32.669	Successfully found the launcher interfaces for launcher Pythonlauncher.dll
2024-07-24 15:04:32.669	Successfully registered the launchcontext for launcher Pythonlauncher.dll
2024-07-24 15:04:32.669	Successfully registered the launcher Pythonlauncher.dll for the scriptType 2
2024-07-24 15:04:32.671	Negotiated Runtime Version for launcher commonlauncher.dll is 4
2024-07-24 15:04:32.671	Successfully set runtime version to 4
2024-07-24 15:04:32.671	Got API for launcher commonlauncher.dll with version 4
2024-07-24 15:04:32.671	Successfully loaded launcher commonlauncher.dll and negotiated version
2024-07-24 15:04:32.679	Successfully initialized launcher commonlauncher.dll
2024-07-24 15:04:32.679	Successfully found the launcher interfaces for launcher commonlauncher.dll
2024-07-24 15:04:32.679	Successfully registered the launchcontext for launcher commonlauncher.dll
2024-07-24 15:04:32.679	Successfully registered the launcher commonlauncher.dll for the scriptType 0
2024-07-24 15:04:32.679	Successfully registered the launcher commonlauncher.dll for the scriptType 3
2024-07-24 15:04:32.679	Successfully registered the launcher commonlauncher.dll for the scriptType 101
2024-07-24 15:04:33.078	RemoveWorkingDirectories invoked.
2024-07-24 15:05:51.440	Failed to delete directory C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\FFFEC268-6354-4ABD-A452-F80F1B78011C\Documents with HR 0x80070005.
2024-07-24 15:05:51.440	Failed to delete directory C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\FFFD3B79-D84C-47C1-BAEA-EA37E29D300D\Documents with HR 0x80070005.
2024-07-24 15:05:51.440	Failed to delete directory C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\FFFCEE20-BA34-49C9-830D-E7793ADC026C\Documents with HR 0x80070005.
2024-07-24 15:05:51.440	Failed to delete directory C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\FFF994FE-456B-4122-91F3-9925D091147B\Documents with HR 0x80070005.
(...) 10k lines
2024-07-24 15:05:53.909	Failed to delete directory C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\FFFEC268-6354-4ABD-A452-F80F1B78011C with HR 0x80070091.
2024-07-24 15:05:53.909	Failed to delete directory C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\FFFD3B79-D84C-47C1-BAEA-EA37E29D300D with HR 0x80070091.
2024-07-24 15:05:53.909	Failed to delete directory C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\FFFCEE20-BA34-49C9-830D-E7793ADC026C with HR 0x80070091.
2024-07-24 15:05:53.909	Failed to delete directory C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\FFF994FE-456B-4122-91F3-9925D091147B with HR 0x80070091.

Pytnonlauncher log:

[2024-07-25 07:10:59.066][0000211C][0000220C][Always] ScaleRJobCleaner.Start() successfully started thread 8436 [000020F4].
[2024-07-25 07:10:59.066][0000211C][0000220C][Always] ProcessPoolTrimmer.Start() successfully started thread 10236 [000027FC].
[2024-07-25 07:10:59.066][0000211C][0000220C][Always] StaleDirectoryCleaner.Start() successfully started thread 11016 [00002B08].
[2024-07-25 07:10:59.066][0000211C][0000220C][Always] ComputerName=***********
[2024-07-25 07:10:59.066][0000211C][0000220C][Always] SystemRoot=C:\Windows
[2024-07-25 07:10:59.066][0000211C][0000220C][Always] User=NT Service\MSSQLLaunchpad
[2024-07-25 07:10:59.066][0000211C][0000220C][Always] UserProfile=C:\Windows\ServiceProfiles\MSSQLLaunchpad
[2024-07-25 07:10:59.066][0000211C][0000220C][Always] Users=0
[2024-07-25 07:10:59.066][0000211C][000020F4][Always] ScaleRStaleJobCleaner::Worker(000001B31A06A9C0) started successfully.
[2024-07-25 07:10:59.066][0000211C][000027FC][Always] ProcessPoolTrimmer::Worker(000001B31A06B770) started successfully.

And indeed, folders are not removed. But they seem to be empty, so contents are probably removed, but folder remains and error entries are added. These error logs can grow huge. Multiple GB.

As for the permissions, I'm sure that both NT Service\MSSQLLaunchpad as well as SQL Server service account have full rights on that structure. Moreover, as part of testing, I even added these accounts to local administrators. That also did not solve the issue.

I would really appreciate any hints or suggestions on finding the root cause.

Thanks in advance.

SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
13,788 questions
Windows Server
Windows Server
A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.
13,056 questions
Accepted answer
  1. LucyChenMSFT-4874 4,750 Reputation points
    2024-07-26T03:43:46.9133333+00:00

    Hi @RandomGuy123 ,

    Thanks for your kindly feedback!

    As Erland says:

    'As for why Launchpad not dealing with the attribute(s) - I say it is an oversight. Or in another word: a bug.'

    You can report this issue to this community! We will very concern about your issue, and we will inform you if there is new progress! Hope this issue can be resolved as soon as possible!

    In addition, I don't want to nag you, but maybe our answers are helpful. May I humbly suggest that please accept Answer if they helped.

    Your action would be helpful to other users who encounter the same issue and read this thread.

    The links and methods we provided will also help them resolve their issue or follow up this issue with us together!

    Thanks for your understanding. Your encouragement and recognition are the driving force for our progress!

    Best regards,

    Lucy Chen

    If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

    Note: Please follow the steps in our Documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    https://docs.microsoft.com/en-us/answers/support/email-notifications

    1 person found this answer helpful.

2 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. RandomGuy123 40 Reputation points
    2024-07-25T15:45:12.04+00:00

    To add more details as I made some progress in my checks (Thanks to one of the checks that @Erland Sommarskog did on his machine finding out he is experiencing similar issue).

    All works when deleted from Windows Explorer.

    But it looks like it's an old problem in Windows that rmdir command (and probably used by Launchpad somewhere in the process) is causing problems ( I found some post from other people complaining on that behaviour).

    Even if executed as Admin on elevated CMD prompt:

    set zm="C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\ExtensibilityData\Appcontainer1\22C73C41-4113-48C0-BA07-8646D49ECB0B\Documents"
rmdir %zm%  
Access is denied.

    It's same error that is visible in Launchpad error logs.

    I did a test and If I modify the directory attributes and remove the read only and system, I'm able to remove the folder. Now it makes me wonder why these attributes are there in the first place and why Launchpad is failing on delete this. 

    attrib -h -r -s %zm% /s /d
rmdir %zm%

    It works, folder is removed.

    However this does not solve my problem as there are hundreds of folders automatically managed by SQL Launchpad

    I also discovered that I have the same issue on other environment, where SQL Server 2019 and Windows 2019 is installed. Just fewer entries are generated.

    1 person found this answer helpful.
  2. LucyChenMSFT-4874 4,750 Reputation points
    2024-07-25T09:31:41.66+00:00

    Hi @RandomGuy123 ,

    Thank you for your reaching out and welcome to Microsoft Q&A!

    Could you please tell us what action you have done in detail?

    Here is an official document about Installing SQL Server 2022 Machine Learning Services (Python and R) on Windows, we can get the Pre-installation checklist and Run setup in it, please check whether you had any improper operation! Hope it can help you well.

    I've checked every piece of documentation, requirement, permissions, suggestions, even tried desperate attempts like temporarily add NT Service\MSSQLLaunchpad to Administrators to verify if that's not the permission issue, but same error persist.

    I checked the Isolation changes for Machine Learning Services, your operation can change the permissions indeed:

    By default, external Python and R scripts only have read access permission to their working directories.

    If your Python or R scripts need access to any other directory, you need give either Read & execute and/or Write permissions to the NT Service\MSSQLLaunchpad service user account and ALL APPLICATION PACKAGES on this directory.

    And my dev team is using SQL Server Machine Learning / SQL Server Launchpad for running R scripts / python code. The same issue happens on two different servers.

    After you meet the prerequisites, try to run R scripts / Python codes again. Here is an official document, you can follow the steps in it.

    https://www.r-bloggers.com/2022/12/using-r-and-python-in-microsoft-sql-server-2022/

    In my case, so far most of the things work fine, except the fact that SQL Server Launchpad bloats error logs with thousands of error messages like the errors at the bottom of below log (Extlaunchererrorlog):

    Your error message is not complete, could you please provide the error logs here in detail to help us narrow down this issue? It would be best if we can get the SQL Server error log at the same time!

    In addition, could you please follow the content of this article to troubleshoot before receiving out reply!

    Thanks for your understanding. Your time and cooperation are much valued by us. We are looking forward to hearing from you to assist further.

    Best regards,

    Lucy Chen

    If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

    Note: Please follow the steps in our Documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    https://docs.microsoft.com/en-us/answers/support/email-notifications

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.