Fighting dictionary attacks with Sshutout and Fail2ban
Closing the Book
Services that require a username and password for login are potential targets for dictionary attacks. Sshutout and Fail2ban introduce time penalties for invalid attempts.
Sshutout  is a daemon written in C that checks a logfile for invalid SSH logins at frequent intervals. If Sshutout discovers a pattern of failed login attempts by a client, it blocks the client according to iptables rules. After a configurable delay, the penalty is automatically revoked.
Activating the Ban
All it takes to install the 32KB tarball is the usual make; make install.
The daemon is located in /usr/local/sbin after the install, and its configuration file is /etc/sshutout.conf.
The configuration file lets you configure a number of parameters. It is extremely important to specify the right name for the logfile that you want Sshutout to monitor. The name defaults to /var/log/messages, but many distributions write login information to other logfiles. On Ubuntu, for example, you would need this setting:
sshd_log_file = /var/log/auth.log
The threshold option specifies the LOS value (LOS = level of stupidity). In other words, it defines how many invalid login attempts a client can make before it is temporarily blocked. If you have a large number of passwords or a poor memory, you might want to increase this threshold (default = 4).
The lock-out time is related to the threshold value. On the one hand, you need to prevent brute-force attacks as effectively as possible; on the other, it is not a good idea to lock out your users for hours on end just because they have difficulty remembering their passwords after a night on the town. The default value, five minutes, is a useful compromise: delay_penalty = 300.
This will give users enough time to drink a cup of coffee and refresh their memories. Alternatively, you can exclude known password forgetters from the outset. The configuration file has a whitelist line on which to enter names or IP addresses of hosts you don't want to lock out.
Sshutout also does some autonomous whitelisting. When launched, it outputs an overview of the current settings, including the following lines:
Whitelist: 18.104.22.168 10.0.0.254 10.0.0.214
This means that Sshutout's whitelist automatically excludes its own IP address, the default gateway address, and the name server. This shouldn't be too dangerous, but more cautious admins might want to change this behavior by specifying auto_whitelist = no.
Sshutout also logs its own activities in /var/log/sshutout.log by default. If a client is blocked, an entry for the client is written to the file:
10.0.0.42 blocked on Sat Feb 02 15:32:32 2008
iptables -L lets you view the underlying firewall rule (Figure 1). Finally, an exception handling function deals with cases in which a client initiates an SSH connection but does not make an entry, which commonly occurs in cases of denial-of-service attacks. Sshutout ignores these kinds of connections by default. The illegal_user = yes entry tells Sshutout to treat these like any other failed login attempt.
Fail2ban: All-Around Protection
Fail2ban  basically takes the same approach as Sshutout; however, it is not restricted to SSH. In fact, it can protect more or less any service that requires users to log in. Fail2ban makes its decisions on the basis of logfile entries, although it uses a different technical approach.
The tool is split into two components: a server and a client. The server monitors the logfiles and iptables rules. The administrator can use the client to issue commands to the server to change, for example, the logging level.
The jail.conf configuration file defines a number of services that Fail2ban can protect, with ssh heading the list:
[ssh] enabled = true port = ssh,sftp filter = sshd logpath = /var/log/auth.log maxretry = 6
The sections that follow contain parameters for other services. Each of these sections is a "jail" in Fail2ban-speak. The filter = sshd entry is equivalent to a file in the /etc/fail2ban/filter.d directory. The file contains a regular expression that the Fail2ban server searches for in the logfile. At this point you can't configure the duration of a lock-out because this value is set to 600 seconds in a global section, [DEFAULT]. If this is too long for your liking, you can add a bantime = 300 entry in the [ssh] section.