OpenSSH Series (Extra 3) - On Using and Debugging Forward Agent

Actually this blog should have been introduced in earlier chapters, but I had never used this feature. A few days ago a friend on Windows kept failing and asked me to help debug — only then did I really use this feature. After understanding it deeply, I personally feel I wouldn’t consider using it; here I just introduce the usage and some basic applications, and finally sort out the security issues.

Basic Principle

In an earlier blog (OpenSSH Series (5) - Jumpserver and HTTP Proxy Usage), I introduced a scheme using a jumpserver to access internal machines. The scheme introduced below differs slightly from the earlier one — see the diagram:

The basic principle of Forward Agent is similar to temporarily copying the private keys from your local .ssh directory to the Jump Server, so you can then access your servers from the Jump Server. Note “similar” — the private key isn’t actually fully copied over, but the permissions you hold are no different from having copied it.

All content below assumes the Jump Server is a Linux machine.

Concrete Usage

Continuing with the diagram above: on my computer, ssh to the Jump Server — the difference is adding the -A parameter:

> ssh -A JumpServer
(JumpServer) > ssh Server  # the private key is found automatically here

Debugging

When problems occur, I suggest first printing logs yourself with ssh -vvv to examine the problem; when consulting others, best to paste this log too.

Confirming Whether Forward Agent Succeeded

(Jump Server):~# echo "$SSH_AUTH_SOCK"
/tmp/ssh-t4qX51rQWm/agent.8014

After ssh-connecting to the jumpserver, confirm whether this variable exists. If not, first check my computer — whether ssh-agent has started. A fairly simple confirmation method is whether ssh-add -l returns normally.

Auto-start ssh agent when launching git bash on Windows

Starting forward agent on Linux

Confirming the Keys Were Added

(Jump Server):~# ssh-add -l
4096 SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx corvo@ThinkPad (RSA)
4096 SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx corvo@X1C (RSA)
2048 SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx /home/corvo/.ssh/id_rsa_raspberry (RSA)
2048 SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +GpuM20QI /home/corvo/.ssh/id_rsa_github (RSA)

If the key you want isn’t here, exit, start ssh-agent on the local computer, use ssh-add to add the corresponding private key, then ssh-connect to the jump server.

Persisting ssh-agent Configuration

Persisting ssh-agent configuration on Windows

On Linux I generally use oh-my-zsh; it has an ssh-agent plugin that just works. For other solutions, search on your own.

Compared with ProxyCommand

From the ssh-add output above, keys added to ssh-agent locally are all usable on the Jump Server, and the $SSH_AUTH_SOCK variable can be set directly — that is to say, if your Jump Server allows multiple people to access it, users with root permission can find the AUTH_SOCK files under the /tmp directory and thereby use your private key. This is extremely unsafe behavior.

By contrast, ProxyCommand itself copies no data; it uses the Jump Server as a layer of tcp connection forwarding — even the Jump Server cannot know the transmitted content. From a security standpoint, ProxyCommand is a relatively safe solution.

Besides the security improvement, ProxyCommand also has some features Forward Agent lacks, such as forward proxy port forwarding, custom config files, and more interestingly, it supports multi-layer Jump Servers. Functionally, ProxyCommand is also relatively rich.