Azure Arc - MDE extension installation failed due to required proxy

Question

Hi,

currently I have the problem, that the MDE extension can't be installed on an Arc enabled server due to a "no internet connection" problem:

Extension Message: Failed to configure Microsoft Defender for Endpoint: Microsoft Defender for Endpoint installation failed with error code 52, error message No internet connectivity [error code 4]. Microsoft Defender for Endpoint installation failed  ResourceId: /subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX, executionlog: 2023-07-20 02:01:05.213079, 30 - Failed get latest installation script -> using default script. Exception: <urlopen error [Errno 97] Address family not supported by protocol>
2023-07-20 02:01:21.112092, 40 - MDE process stdout: b'--- mde_installer.sh v0.6.2 ---\n[>] detected: ubuntu 22.04 jammy (debian)\n[>] scaled: 22.04\n[v] set package manager: apt\n/usr/bin/wget\n[final] connected=\n[S] MDE not installed.\n[x] internet connectivity needed for package installation\n[*] exiting (4)\n'
2023-07-20 02:01:21.112343, 40 - Microsoft Defender for Endpoint installation failed with error code 52, error message No internet connectivity [error code 4]. Microsoft Defender for Endpoint installation failed 
2023-07-20 02:01:21.112432, 40 - Failed to configure Microsoft Defender for Endpoint: Microsoft Defender for Endpoint installation failed with error code 52, error message No internet connectivity [error code 4]. Microsoft Defender for Endpoint installation failed  ResourceId: /subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX
2023-07-20 02:01:21.112931, 40 - Stack trace: Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.9/src/MdeInstallerWrapper.py", line 136, in main
    logutils.throw_and_write_log(error_message)
  File "/var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.9/src/LogUtils.py", line 25, in throw_and_write_log
    raise Exception(message)
Exception: Microsoft Defender for Endpoint installation failed with error code 52, error message No internet connectivity [error code 4]. Microsoft Defender for Endpoint installation failed 

Extension Error: Python 3.10.6
2023-07-20 01:52:19,024, INFO - Start executing handler action: enable
2023-07-20 01:52:19,033, INFO - Set handler status file /var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.9/status/0.status, Status= transitioning, Code= 1, Message= Configuration In Progress
2023-07-20 01:52:19,033, INFO - Successfully retreived AzureResourceID from extension public settings: /subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX
2023-07-20 01:52:19,036, INFO - Successfully retreived AzureResourceID from IMDS: /subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX for machine type: ARC
2023-07-20 01:52:19,036, INFO - Using AzureResourceID from IMDS
2023-07-20 01:52:19,036, INFO - Successfully retreived AutoUpdate from extension public settings: True
2023-07-20 01:52:19,036, INFO - Running command in separate process: ./PythonRunner.sh src/MdeInstallerWrapper.py --workspaceId SUB-XXX --azureResourceId /subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX --logFolder /var/lib/GuestConfig/extension_logs/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.9 --statusFolder /var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.9/status --configFolder /var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.9/config --autoUpdate TRUE
2023-07-20 01:52:19,037, INFO - End executing handler action: enable
2023-07-20 01:52:18,532, INFO - Start executing handler action: install
2023-07-20 01:52:18,532, INFO - MDE installation will occur in 'enable'
2023-07-20 01:52:18,533, INFO - End executing handler action: install
Python 3.10.6

It's a Ubuntu machine in a separate network were internet access is only possible via proxy server.
The proxy server is configured as a system wide proxy in /etc/environment and also explicitly set in the Azure Arc agent. I also changed the sudoers file to "remember" the proxy environment variables, but this didn't change a thing.
In the MDE scripts which were successfully downloaded by Azure Arc I saw that there is something regarding a rpm proxy and I thought it's all about the relevant environment variables, but this wasn't the case.

Hope somebody has an idea.

Thanks,
Markus

Answer 1

Just a quick update, because nobody had an idea so far.
I tried to execute the PythonRunner.sh command with parameters from the logfile manually after typing sudo su and now the extension was installed without any problem.
No error message in Azure anymore, just succeeded.

root@XXX:/var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.10# ./PythonRunner.sh src/MdeInstallerWrapper.py --workspaceId SUB-XXX --azureResourceId /subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX --logFolder /var/lib/GuestConfig/extension_logs/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.10 --statusFolder /var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.10/status --configFolder /var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.10/config --autoUpdate TRUE --avMode noAction
2023-08-02 06:38:11,048, INFO - Start executing installer wrapper
2023-08-02 06:38:11,049, INFO - Get latest installation script from ##URL isn't allowed to be posted here - Microsoft GitHub repository -> mdatp-xplat##/mde_installer.sh
2023-08-02 06:38:11,163, INFO - Start to run the install command: src/mde_installer.latest.sh --debug --install --channel prod --onboard onboardingScript.tmp.py --passive-mode --tag SecurityWorkspaceId SUB-XXX --tag AzureResourceId /subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX -y --http-proxy http://XXX:XXX --https-proxy http://XXX:XXX --log-path /var/lib/GuestConfig/extension_logs/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.10/MdeInstallerLog.log
2023-08-02 06:38:49,542, INFO - MDE process stdout: b'--- mde_installer.sh v0.6.3 ---\n[>] detected: ubuntu 22.04 jammy (debian)\n[>] scaled: 22.04\n[v] set package manager: apt\n[v] no conflicting applications found\n[v] required pkgs are installed\n[>] installing MDE\n[v] installed\n[>] MDE/EPP already in passive mode\n[v] passive mode set\n[>] onboarding script: onboardingScript.tmp.py\n[v] onboarded\n[v] tags set.\n[S] MDE installed.\n[S] Version: "101.23062.0010"\n[S] Onboarded: true\n[S] Passive mode: true\n[S] Device tags: [{"key":"AzureResourceId","value":"/subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX"},{"key":"SecurityWorkspaceId","value":"SUB-XXX"}]\n[S] Subsystem: "fanotify"\n[S] Conflicting applications: []\n[v] --- mde_installer.sh ended. ---\n[*] exiting (0)\n'
2023-08-02 06:38:49,542, INFO - Wait for MDE service to be available
2023-08-02 06:38:49,608, INFO - MDE is onboarded
2023-08-02 06:38:49,640, INFO - MDC tags in MDE are valid
2023-08-02 06:38:49,704, WARNING - Could not determine OS details. Exception: module 'platform' has no attribute 'linux_distribution'
2023-08-02 06:38:49,714, INFO - Set handler status file /var/lib/waagent/Microsoft.Azure.AzureDefenderForServers.MDE.Linux-1.0.3.10/status/0.status, Status= success, Code= 0, Message= {"azureResourceId": "/subscriptions/SUB-XXX/resourceGroups/RG-XXX/providers/Microsoft.HybridCompute/machines/XXX", "securityWorkspaceId": "SUB-XXX", "osDetails": null, "machineId": "e04fe32d0793eeb617ceb02769f5d148bffb963b", "onboardingPackageOperationResultCode": "Success"}
2023-08-02 06:38:49,715, WARNING - Failed to remove install.status. Exception: [Errno 2] No such file or directory: 'install.status'
2023-08-02 06:38:49,715, INFO - End executing installer wrapper

Maybe Azure Arc has problems detecting the proxy via Python, because the relevant part of the script worked quite fine with my user and also with the root user.
I couldn't find out which user executes the commands via Arc (I think it's a user called himds and at least the environment variable could be resolved there as well). Maybe the /etc/environment file isn't considered in this case and that's why the proxy wasn't working...
(I also adjusted the file /lib/systemd/system/himdsd.service to use an explicit proxy configuration, but this didn't work as well.)

After the - in my opinion successful - installation, I set the proxy like described here:
https://learn.microsoft.com/en-us/microsoft-365/security/defender-endpoint/linux-static-proxy-configuration?view=o365-worldwide#post-installation-configuration

And the connection test was successful as well.

mdatp connectivity test
Using proxy 'XXX:XXX'
Testing connection with [https://nf.smartscreen.microsoft.com/api/network/mac]() ... [OK]
Using proxy 'XXX:XXX'
Testing connection with [https://europe.x.cp.wd.microsoft.com/api/report]() ... [OK]
Using proxy 'XXX:XXX'
Testing connection with [https://usseu1northprod.blob.core.windows.net/]() ... [OK]
Using proxy 'XXX:XXX'
Testing connection with [https://usseu1westprod.blob.core.windows.net/]() ... [OK]
Using proxy 'XXX:XXX'
Testing connection with [https://europe.smartscreen.microsoft.com//api/network/mac]() ... [OK]
Using proxy 'XXX:XXX'
Testing connection with [https://europe.smartscreen-prod.microsoft.com//api/network/mac]() ... [OK]
Using proxy 'XXX:XXX'
Testing connection with [https://go.microsoft.com/fwlink/?linkid=2144709]() ... [OK]
Testing connection with [https://winatp-gw-weu.microsoft.com/test]() ... [OK]
Testing connection with [https://winatp-gw-neu.microsoft.com/test]() ... [OK]
Testing connection with [https://eu-v20.events.data.microsoft.com/ping]() ... [OK]
Testing connection with [https://automatedirstrprdneu.blob.core.windows.net]() ... [OK]
Testing connection with [https://automatedirstrprdweu.blob.core.windows.net]() ... [OK]

Sidenode: Also the recently released Microsoft article
https://techcommunity.microsoft.com/t5/core-infrastructure-and-security/deploying-microsoft-defender-for-servers-in-network-restricted/ba-p/3886437
didn't help.