systemd stops working until VM is rebooted

Sellis Markos 1

The symptom is that systemctl will timeout - until the VM is rebooted. Everything else works normally in the VM.

In Boot Diagnostics I cannot see Serial Log (it timeouts), however the screenshot option shows the same error message multiple times :

"systemd - journald: Failed to send WATCHDOG-1 notification message: Transport endpoint is not connected"

Note that the VM (ALMA Linux 9) runs perfectly well, all services are ok, except systemctl.

Has anyone seen this before?

1 answer

vipullag-MSFT 26,021 Reputation points

2023-11-09T08:11:09.5266667+00:00

Hello Sellis Markos

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Based on the error message you provided, it seems like there might be an issue with the systemd-journald service. One possible solution is to try restarting the service by running the following command:

sudo systemctl restart systemd-journald

If that doesn't work, you can try resetting the journal files by running the following commands:

sudo systemctl stop systemd-journald

sudo rm /var/log/journal/*

sudo systemctl start systemd-journald

If the issue persists even after trying these solutions, Support team will be able to check and help on this. I would recommend you to open a azure support case.

Hope this helps.
Please sign in to rate this answer.
Sellis Markos 1 Reputation point

2023-11-09T08:47:30.6666667+00:00

Hello vipullag - thanks for the reply... However the problem is that systemctl does not work (it times out), so it cannot be used as part of the solution...

vipullag-MSFT 26,021 Reputation points

2023-11-10T07:09:49.03+00:00

Hello Sellis Markos

In that case, you can try restarting the systemd service itself by running the following command:

sudo systemctl daemon-reexec

This will cause systemd to re-execute itself, which might help resolve the issue. If that doesn't work, you can try rebooting the VM by running the following command:

sudo reboot

This will restart the VM and might help resolve the issue with systemd. If the issue persists even after trying this solutions, I would recommend you to open a azure support case.

vipullag-MSFT 26,021 Reputation points

2023-11-13T05:13:32.2733333+00:00

Hello Sellis Markos

Any update on the issue?

Just checking in to see if you got a chance to see previous response.

Sellis Markos 1 Reputation point

2023-11-13T07:45:00.3633333+00:00

Hello vipullag,

thanks for the replies. systemctl does not work so no sudo systemctl daemon-reexec works.
The problem goes away when i reboot the VM from Azure

I am curious how would it be possible to understand why this happens but could not find anything. For example Boot Diagnostics times out when selecting Serial Log (in Azure).

Thanks again - if anyone has this problem and finds a way to learn more about why it happens please add a comment

vipullag-MSFT 26,021 Reputation points

2023-11-15T05:31:59.9666667+00:00

Hello Sellis Markos
Thanks for responding back and confirming that "The problem goes away when i reboot the VM from Azure".

For your query "I am curious how would it be possible to understand why this happens but could not find anything."
This issue needs deeper investigation. Support team will be able to check and help on this. I would recommend you to open a azure support case.

If you don't have a support plan enabled on your subscription, I request you to send an email to AzCommunity@Microsoft.com with Subject as "Attn:Vikas" referencing this thread along with your subscription ID. I can then enable your subscription with a one-time free technical support.

Once the issue is resolved, request you to post the resolution here for the benefit of community.

Sellis Markos 1 Reputation point

2023-11-15T15:16:02.65+00:00

Thanks a lot for the support - if the issue reappears I will do so and update this thread

vipullag-MSFT 26,021 Reputation points

2023-11-17T07:06:38.37+00:00

Hello Sellis Markos

Sure thanks.
Sign in to comment

Share via

systemd stops working until VM is rebooted

1 answer