I tried your scenario in my lab environment and it is working as expected without any issues. I was able to stop the VM without any issues.
Below are the steps that you need to do to disable password auth in Linux VM.
Now, you have both ssh and password auth methods enabled on you VM.
In the below screenshot you see the 2 types of logins.
Once you are logged into the VM with ssh keys, use below command and modify the file sshd_config
to make changes for two parameters PasswordAuthentication and PubkeyAuthentication to no and yes respectively as shown in below screenshot.
sudo vi /etc/ssh/sshd_config
Restart the ssh service with below command
sudo systemctl reload ssh
With this you will now be able to login to VM only with SSH keys.
Hope this helps.
Please 'Accept as answer' if the provided information is helpful, so that it can help others in the community looking for help on similar topics.