Public key cryptography means that the traffic is secured and that if you verify keys correctly, it isn't vulnerable to man-in-the-middle attacks. Exploits are occasionally found, but they are quickly fixed. However, your system could still be vulnerable to brute-force attacks. If you have only a small number of user accounts, if your usernames are unusual, and if your passwords are carefully crafted, this might not be an issue. But if you have multiple users on a system, it becomes harder to ensure that all passwords really are secure. One solution is to run a password-guessing system such as John the Ripper [1], which can discover a poor password before anyone with nefarious intentions has time to exploit it.

Also, you can create firewall rules that establish a maximum number of connection attempts from the same IP address, either indefinitely or for the next few minutes. (The latter option is preferable to avoid problems for real users, who do occasionally mistype their password several times.)

Another alternative is port knocking. In a traditional port-knocking configuration, all ports on the server are closed by default. To the outsider, the network seems inaccessible. A remote user who wants access attempts to initiate a series of connections to a specific sequence of (closed) ports. These connections are not successful, but they are logged by the server. After the appropriate sequence of connection attempts ("knocks"), a daemon running on the server edits the firewall rules to allow a connection from the IP address that has originated the "knock" sequence. The user can then log in normally.

[...]