Problem logging into Windows 2022 OpenSSH server using public and private key authentication

Question

Hello.

We have set up an OpenSSH server on our Windows 2022 Server with Desktop Experience configuration. Testing has confirmed that username/password authentication is working without issue. We would like to have public/private key authentication, but we are running into difficulties.

We have gone into the C:\ProgramData\ssh folder of the server, and have made the following modifications to the sshd_config file:

1. PubkeyAuthentication yes
2. PasswordAuthentication no

The OpenSSH SSH server service has been restarted, but access has been denied.

When connecting to the server from a client system, the MobaXterm utility was used to create the public and private keys. The names of the keys are as follows:

1. id_rsa <- public key
2. id_rsa.ppk <- private key

The connection is done using the -v parameter to get the connection readout. The output file has been attached to this posting. The following text was spotted towards the end of the readout:

Load key "/home/mobaxterm/.ssh/id_rsa": invalid format

Can someone provide assistance on this?

ssh connection readout.txt

Answer 1

I have some progress. I was able to get the correct format of the public and private keys, and I was able to upload the public key to the SSH server. The 'invalid format' error is not present. However, I am still being prompted for the password. The excerpt shown below lists what is occurring:

debug1: Trying private key: /home/mobaxterm/.ssh/id_rsa debug3: sign_and_send_pubkey: RSA SHA256:FEde5f/e/t3sF5FGxmWHUH8b7mFXi6i8IaFR2skbYhY debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password,keyboard-interactive debug1: Trying private key: /home/mobaxterm/.ssh/id_dsa debug3: no such identity: /home/mobaxterm/.ssh/id_dsa: No such file or directory debug1: Trying private key: /home/mobaxterm/.ssh/id_ecdsa debug3: no such identity: /home/mobaxterm/.ssh/id_ecdsa: No such file or directory debug1: Trying private key: /home/mobaxterm/.ssh/id_ed25519 debug3: no such identity: /home/mobaxterm/.ssh/id_ed25519: No such file or directory debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: keyboard-interactive debug3: authmethod_is_enabled password debug1: Next authentication method: password <user>@<server hostname>'s password:

What are my next steps?

Answer 2

Hello Domic Vo

Thank-you for your reply. I went through the steps listed in your posting, and unfortunately, I am still getting the invalid format message. When I created the public key, I noticed the text of the file did not appear to be in one single line when opened by a text editor. Also, the file had a header and footer that indicated it was the private key.

Answer 3

Hello Kaplan, Andrew H,

Thank you for sharing the details of your setup that helps a lot. The issue you’re seeing is a common one when moving from password authentication to key‑based authentication on Windows with OpenSSH.

The error message “invalid format” indicates that the server is rejecting the key because of its format. OpenSSH expects keys in OpenSSH PEM format, but MobaXterm generates them in PuTTY (.ppk) format by default.

Here’s how you can resolve this:

1. Convert your key to OpenSSH format
   • In MobaXterm, export the key in OpenSSH format.
   • Or use PuTTYgen: load your .ppk file, go to Conversions → Export OpenSSH key, then save it as id_rsa (private) and id_rsa.pub (public).
2. Place the public key on the server
   • Copy the contents of id_rsa.pub into: Code

          C:\Users\<your-username>\.ssh\authorized_keys
     

   • Ensure the file is named exactly authorized_keys and that permissions are correct (the .ssh folder should be owned by the user).
3. Check your sshd_config Confirm these settings are present: Code

      PubkeyAuthentication yes
   

PasswordAuthentication no AuthorizedKeysFile .ssh/authorized_keys

   
   Restart the OpenSSH service afterwards:
   
   Code
   
   ```ruby
   net stop sshd
net start sshd

1. Connect with the correct private key Use the converted OpenSSH private key when connecting: Code

      ssh -i ~/.ssh/id_rsa user@server
   

   In MobaXterm, make sure you load the OpenSSH‑formatted key, not the .ppk.

You can check to Ensure the public and private keys are a matching pair. Also kindly Watch out for formatting issues (extra spaces or line breaks) and Use ssh -vvv for verbose output to confirm the key is being offered.

I hope this helps,

If this guidance proves helpful, please kindly click “Accept Answer” so we know we’re heading in the right direction 😊. And of course, I’m here if you need further clarification or support.

Domic Vo.