How to forward agent with "az ssh arc"

Question

How to forward agent with "az ssh arc"

Gabor152 20

Hi,

I have two Ubuntu 22.04 machines connected to Azure using azcmagent: host1 and host2.

I can connect to them separately:

az ssh arc --resource-group ${RG} --name host1
az ssh arc --resource-group ${RG} --name host2

But what I need to do is: connect to host1 with -A and from there connect to host2 - and it doesn't work.

Here's what I tried:

generate keys and config file:

az ssh config  --resource-group ${RG} --name host1 --file ssh_config
az ssh config  --resource-group ${RG} --name host2 --file ssh_config

add the IdentityFile (path in ssh_config) to ssh-agent:

eval `ssh-agent` && ssh-add ...id_rsa

az ssh arc --resource-group ${RG} --name host1 -- -A
# on host1
ssh-add -l
# key is there
ssh host2
... Permission denied (publickey).

On host2 I see this error in the log: The public key is of type ssh-rsa, not a certificate.

I did some research and I tried a couple of things:

created two Ubuntu 24.04 machines: same result
created two Ubuntu 20.04 machines: same result
added -o "PubkeyAcceptedKeyTypes=+******@openssh.com" : same result

I'm sure I'm doing something wrong but I don't know what. Unfortunately the documentation lacks the details: https://learn.microsoft.com/en-us/cli/azure/ssh?view=azure-cli-latest#az-ssh-arc.

Any idea how to do it right?

SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2024-11-27T17:05:19.26+00:00
@Gabor152 Thank you for reaching out to us!

From the details in your post, it looks like you're trying to use agent forwarding (-A) to connect from host1 to host2 while using SSH keys managed by ssh-agent. The error message Permission denied (publickey) and the log message on host2 that "The public key is of type ssh-rsa, not a certificate" suggests a mismatch or misconfiguration around SSH key types and how they're being forwarded.

Suggest you follow the steps below, might help resolve the issue:

First, ensure that SSH key forwarding is enabled. When you connect to host1 with -A, make sure that SSH agent forwarding is enabled. You can verify this by running ssh-add -l on host1 to see if your keys are listed.

Next, modify your ssh_config file to explicitly allow the ssh-rsa key type. Add the following lines to your ssh_config file:

Host host1 HostName <host1_ip> User <your_username> IdentityFile <path_to_your_id_rsa> ForwardAgent yes Host host2 HostName <host2_ip> User <your_username> IdentityFile <path_to_your_id_rsa> PubkeyAcceptedKeyTypes +ssh-rsa

Ensure that your SSH keys are added to the SSH agent. Run the following commands:

eval $(ssh-agent) ssh-add <path_to_your_id_rsa>

Then, connect to host1 with agent forwarding using the -A option:

az ssh arc --resource-group ${RG} --name host1 -- -A

Once connected to host1, try connecting to host2:

ssh host2

If you still encounter the "Permission denied (publickey)" error, it might be due to the SSH key type not being accepted by host2. In that case, you can try generating a new key pair using a different key type, such as ed25519, and update your SSH configuration accordingly.

Hope this helps. Feel free to reply if you need further assistance.

If the response helped, do "Accept Answer" and up-vote it
Gabor152 20 Reputation points

2024-11-28T09:33:18.77+00:00
@SadiqhAhmed-MSFT , thanks for the hint but it doesn't solve the problem yet.

Remember, I can ssh from my machine to host1 and host2 directly. The problem is with the agent forwarding only.

Now, I got much closer to understanding the issue:

After az ssh config... , I have two file paths in the config file:

IdentityFile

CertificateFile

But I have three files:

id_rsa (this corresponds to IdentityFile)

id_rsa.pub-aadcert.pub (this corresponds to CertificateFile)

id_rsa.pub (this is not mentioned in the config file)

I ssh-add the private key: eval `ssh-agent` && ssh-add id_rsa

I check which public key was added: ssh-add -L - it shows the content of id_rsa.pub. I believe it should show be the id_rsa.pub-aadcert.pub content.

I tried ssh-add -s pkcs11 -C id_rsa but it asks for a password I don't have.

Any ideas?

Accepted answer

1 additional answer

Your answer

SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2024-11-27T17:05:19.26+00:00

@Gabor152 Thank you for reaching out to us!

From the details in your post, it looks like you're trying to use agent forwarding (-A) to connect from host1 to host2 while using SSH keys managed by ssh-agent. The error message Permission denied (publickey) and the log message on host2 that "The public key is of type ssh-rsa, not a certificate" suggests a mismatch or misconfiguration around SSH key types and how they're being forwarded.

Suggest you follow the steps below, might help resolve the issue:

First, ensure that SSH key forwarding is enabled. When you connect to host1 with -A, make sure that SSH agent forwarding is enabled. You can verify this by running ssh-add -l on host1 to see if your keys are listed.

Next, modify your ssh_config file to explicitly allow the ssh-rsa key type. Add the following lines to your ssh_config file:

Host host1 HostName <host1_ip> User <your_username> IdentityFile <path_to_your_id_rsa> ForwardAgent yes Host host2 HostName <host2_ip> User <your_username> IdentityFile <path_to_your_id_rsa> PubkeyAcceptedKeyTypes +ssh-rsa

Ensure that your SSH keys are added to the SSH agent. Run the following commands:

eval $(ssh-agent) ssh-add <path_to_your_id_rsa>

Then, connect to host1 with agent forwarding using the -A option:

az ssh arc --resource-group ${RG} --name host1 -- -A

Once connected to host1, try connecting to host2:

ssh host2

If you still encounter the "Permission denied (publickey)" error, it might be due to the SSH key type not being accepted by host2. In that case, you can try generating a new key pair using a different key type, such as ed25519, and update your SSH configuration accordingly.

Hope this helps. Feel free to reply if you need further assistance.

If the response helped, do "Accept Answer" and up-vote it
Gabor152 20 Reputation points

2024-11-28T09:33:18.77+00:00

@SadiqhAhmed-MSFT , thanks for the hint but it doesn't solve the problem yet.

Remember, I can ssh from my machine to host1 and host2 directly. The problem is with the agent forwarding only.

Now, I got much closer to understanding the issue:

After az ssh config... , I have two file paths in the config file:

IdentityFile

CertificateFile

But I have three files:

id_rsa (this corresponds to IdentityFile)

id_rsa.pub-aadcert.pub (this corresponds to CertificateFile)

id_rsa.pub (this is not mentioned in the config file)

I ssh-add the private key: eval `ssh-agent` && ssh-add id_rsa

I check which public key was added: ssh-add -L - it shows the content of id_rsa.pub. I believe it should show be the id_rsa.pub-aadcert.pub content.

I tried ssh-add -s pkcs11 -C id_rsa but it asks for a password I don't have.

Any ideas?

Answer 1

Summarizing from the discussion (above). @Gabor152, Thanks for sharing the info/solution with the community.

Issue: How to forward agent with "az ssh arc"

Solution: Steps to make it work:

az ssh config --resource-group ${RG} --name host1 --file ssh_config
Rename id_rsa.pub-aadcert.pub to id_rsa-cert.pub (check paths in ssh_config file)
ssh-add id_rsa - it says "Identity added:...", "Certificate added:..."
ssh-add -L shows the certificate
az ssh arc --resource-group ${RG} --name host1 -- -A
from host1: ssh host2 - works, without any further configuration.

Background: If the identity file is say XYZ, then the SSH client will look for the certificate in XYZ-cert.pub.

Since the Microsoft Q&A community has a policy that the question author cannot accept their own answer. They can only accept answers by others, I' have summarized and posted the solution/answer to benefit the community users to find the answers quickly.

If the response helped, do "Accept Answer" and up-vote it

Answer 2

Gabor152 20

I found it!

Steps to make it work:

az ssh config --resource-group ${RG} --name host1 --file ssh_config
Rename id_rsa.pub-aadcert.pub to id_rsa-cert.pub (check paths in ssh_config file)
ssh-add id_rsa - it says "Identity added:...", "Certificate added:..."
ssh-add -L shows the certificate
az ssh arc --resource-group ${RG} --name host1 -- -A
from host1: ssh host2 - works, without any further configuration.

Background: If the identity file is say XYZ, then the SSH client will look for the certificate in XYZ-cert.pub.

SadiqhAhmed-MSFT 49,326 Reputation points Microsoft Employee Moderator

2024-11-28T10:32:18.5633333+00:00

@Gabor152 Thanks for the update. Glad to know you were able to resolve the issue.

Share via

How to forward agent with "az ssh arc"

1 additional answer

Your answer