TABLE OF CONTENT
Introduction
Prerequisites
apt install iptables iptables-persistent
Fail2Ban Setup
Fail2Ban Installation
apt install fail2ban python3-systemd
ERROR Failed during configuration: Have not found any log file for sshd jail
Fail2Ban Configuration
Main Configuration Files
The two "main" Fail2Ban configuration files are /etc/fail2ban/fail2ban.conf and /etc/fail2ban/jail.conf. The first defines some general options for the Fail2Ban service, and the second is used to configure protection for various services. The documentation states that we should not modify those files and use .local files instead, and I will show you how to do that.
The general options file fail2ban.conf, in most cases, requires no modifications. Optionally, you can add allowipv6 = no
if you don't use an IPv6 network, to omit the following message:
WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
[DEFAULT]
## MISC
bantime = 86400
findtime = 3600
maxretry = 5
backend = systemd
## ACTIONS
mta = sendmail
destemail = root
sender = F2B-Alerts-<fq-hostname>
action = %(action_xarf)s
Let's observe those options. In the example above, if the violating host (IP) is discovered five times (maxretry) in one hour (findtime = 3600), it will be banned from accessing the server during the next 24 hours (bantime = 86400).
The default backend is the "location" where Fail2Ban will look for violating hosts. In the older versions of Debian Linux, Fail2Ban checked logs in the/var/log directory when Syslog handled logs. In the latest version of Debian, logs are handled by systemd-journald service; thus, backend = systemd.
Optionally, once the violating host is discovered and banned, you can configure Fail2Ban to email the root user (destmail = root). You can check those emails (if any) by listing the contents of the /var/mail/root file. If you want to receive those emails to one of your "real" email addresses, you must configure an email relay. To see how to do that, you can check our guide on the link button below this text:
SSH Configuration
Most brute-force attacks against SSH are the ones where the "attacker" is trying to "guess" a login password. Usually, a bot is installed on the attacking side and is supplied with a list of the most common usernames/passwords, which are then used to attempt to log in to the target server. A very efficient method to protect against such attacks is to deny a password login and configure a private/public key pair, which is then used to log in to the server (if you read our prerequisites articles, you should have that in place).
Unfortunately, this will not stop the attacker (bot) from attempting to log in by using a password (although it can't). This is where Fail2Ban comes into play. When you install Fail2Ban on Debian, SSH protection is enabled by default. A configuration or "jail" can be found at /etc/fail2ban/jail.d/defaults-debian.conf. It uses a filter with all the actions defined (how the violators will be blocked) whose configuration can be found in /etc/fail2ban/filter.d/sshd.conf file. Usually, you don't have to do anything about the configuration; everything will work as-is. Optionally (and something I highly suggest), you can set the jail/filter to "aggressive" mode, so it protects you from any possible scenario. To do that, open /etc/fail2ban/jail.d/defaults-debian.conf and add the following option to the bottom of the file:
mode = agressive
systemctl restart fail2ban.service
SSH Configuration - Repeating Offenders
Once the violating IP is discovered, it is banned according to the rules set in the main configuration file (/etc/fail2ban/jail.conf). This can be modified either in the main configuration "override" file (/etc/fail2ban/jail.local - see Main Configuration Files section above) or in a jail-specific configuration file (set in /etc/fail2ban/jail.d directory). For my SSH configuration above, ban rules will be applied according to the options I've set in /etc/fail2ban/jail.local file; if the violating IP is discovered five times in one hour, it will be banned for the next 24 hours.
After 24 hours, the ban will expire, and the attack on your server will resume until the attacking host IP gets banned again. Usually, brute-force bots are configured to attempt to "break in" the server until they succeed. So, blocking the same attacking IP will be performed by Fail2Ban literally in an endless loop.
Fail2Ban has mechanisms to configure jails and banning a bit "smarter" (i.e., check bantime.increment, bantime.rndtime, bantime.maxtime, bantime.formula and other options in /etc/fail2ban/jail.conf file), but what I like to do regarding such "repeating" offenders is to create a new jail/filter that will handle those. It looks like this: if an IP is banned by a default SSH filter five times in one year, that IP is banned permanently. Done!
To do that, a "helper" file that will store such repeating IPs must be created first. You can do it by executing the following in the Terminal:
touch /etc/fail2ban/ip.blocklist.repeatoffender
chmod 770 /etc/fail2ban/ip.blocklist.repeatoffender
# Fail2Ban configuration file - repeat offenders
[INCLUDES]
before = iptables-common.conf
[Definition]
# Option: actionstart
# Notes.: command executed once at the start of Fail2Ban.
# Values: CMD
#
actionstart = iptables -N f2b-<name>
iptables -A f2b-<name> -j RETURN
iptables -I <chain> -p <protocol> -j f2b-<name>
# Establish chain and blocks for saved IPs
iptables -N f2b-ip-blocklist
iptables -A f2b-ip-blocklist -j RETURN
iptables -I <chain> -p <protocol> -j f2b-ip-blocklist
cat /etc/fail2ban/ip.blocklist.<name> |grep -v ^\s*#|awk '{print $1}' | while read IP; do iptables -I f2b-ip-blocklist 1 -s $IP -j REJECT --reject-with icmp-port-unreachable; done
# Option: actionstop
# Notes.: command executed once at the end of Fail2Ban
# Values: CMD
#
actionstop = iptables -D <chain> -p <protocol> -j f2b-<name>
iptables -F f2b-<name>
iptables -X f2b-<name>
# Remove chain and blocks for saved IPs to prevent duplicates on service restart
iptables -D <chain> -p <protocol> -j f2b-ip-blocklist
iptables -F f2b-ip-blocklist
iptables -X f2b-ip-blocklist
# Option: actioncheck
# Notes.: command executed once before each actionban command
# Values: CMD
#
actioncheck = iptables -n -L <chain> | grep -q 'f2b-<name>[ \t]'
# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionban = VERIFY="<ip>*"
ADD="<ip> # f2b/$( date '+%%Y-%%m-%%d %%T' ): Perma-Banned"
FILE=/etc/fail2ban/ip.blocklist.<name>
grep -q "$VERIFY" "$FILE" || iptables -I f2b-<name> 1 -s <ip> -j DROP
grep -q "$VERIFY" "$FILE" || echo "$ADD" >> "$FILE"
# Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionunban = # Do nothing becasuse their IP is in the blocklist file
# To manually unban from the ip blocklist file run this command:
# Be warned that if the ip is in log rotated files it must be whitelisted
#
# sed -i '/^<ip>/d' /etc/fail2ban/ip.blocklist.repeatoffender
#
[Init]
# Default name of the chain
#
name = default
# Option: protocol
# Notes.: internally used by config reader for interpolations.
# Values: [ tcp | udp | icmp | all ] Default: tcp
#
protocol = all
# Option: chain
# Notes specifies the iptables chain to which the fail2ban rules should be
# added
# Values: STRING Default: INPUT
chain = INPUT
# Fail2Ban configuration file - repeat offenders
# This filter monitors the fail2ban log file, and permanently
# bans the ip addresses of persistent attackers.
[Definition]
_daemon = repeatoffender
failregex = fail2ban.actions\s+.+:\s+NOTICE\s+\[(?:.*)\]\s+Ban\s+<HOST>
ignoreregex = fail2ban\.actions\s+\[.+\]:\s+NOTICE\s+\[%(_daemon)s\]\s+Ban\s+<HOST>
[repeatoffender]
enabled = true
backend = auto
mode = agressive
filter = repeatoffender
action = repeatoffender[name=repeatoffender]
sendmail-whois[name=Repeat-Offender, dest=root, sender=F2B-Alerts-<fq-hostname>]
logpath = /var/log/fail2ban.log*
maxretry = 5
findtime = 31536000
bantime = -1
/var/log/fail2ban.log {
# weekly
monthly
# rotate 4
rotate 12
# compress
# delaycompress
missingok
notifempty
postrotate
fail2ban-client flushlogs 1>/dev/null
endscript
# If fail2ban runs as non-root it still needs to have write access
# to logfiles.
# create 640 fail2ban adm
create 640 root adm
}
HTTP/HTTPS Configuration
When we are talking about HTTP/HTTPS and attacks on web servers in general, the most common ones are DDOS attacks. The other typical case "can't" be quite considered as an attack. However, it can still disrupt the normal operations of a web server: poorly configured (mostly on purpose) web crawlers whose crawl rate is way above something that can be considered normal.
The most common web server "services" on Linux are Apache and Nginx, and I will show you how to configure a Fail2Ban filter for the latter. Under the presumption that you are running a self-hosted web server with Nginx running on it, I highly recommend installing Nginx Ultimate Bad Bot & Referrer Blocker by Mitchell Krog. Combined with the Fail2Ban, you can protect your Nginx web server against (almost) all possible attacks.
To start with this, you must know where Nginx is storing logs. Usually, I have a general Nginx log writing in /var/log/nginx/access.log file, and I have separate logs for each domain (website) I'm hosting on the server. Their logs are written in /home/websites/$DOMAIN/logs/access.log, where $DOMAIN stands for the domain of the website.
Once you determine where Nginx stores logs, you can start with the Fail2Ban configuration. First, you need to create a "helper" file which will store violating IP addresses. To do so, execute the following commands in the Terminal:
touch /etc/fail2ban/ip.blocklist.nginx.repeatoffender
chmod 770 /etc/fail2ban/ip.blocklist.nginx.repeatoffender
[INCLUDES]
before = iptables-common.conf
[Definition]
# Option: actionstart
# Notes.: command executed once at the start of Fail2Ban.
# Values: CMD
#
actionstart = iptables -N f2b-<name>
iptables -A f2b-<name> -j RETURN
iptables -I <chain> -p <protocol> -j f2b-<name>
# Sort and Check for Duplicate IPs in our text file and Remove Them
sort -u /etc/fail2ban/ip.blocklist.nginx.repeatoffender -o /etc/fail2ban/ip.blocklist.nginx.repeatoffender
# Persistent banning of IPs reading from our ip.blocklist.nginx.repeatoffender text file
# and adding them to IPTables on our jail startup command
cat /etc/fail2ban/ip.blocklist.nginx.repeatoffender | while read IP; do iptables -I f2b-<name> 1 -s $IP -p tcp -m multiport --dports 80,443 -j DROP; done
# Option: actionstop
# Notes.: command executed once at the end of Fail2Ban
# Values: CMD
#
actionstop = iptables -D <chain> -p <protocol> -j f2b-<name>
iptables -F f2b-<name>
iptables -X f2b-<name>
# Option: actioncheck
# Notes.: command executed once before each actionban command
# Values: CMD
#
actioncheck = iptables -n -L <chain> | grep -q 'f2b-<name>[ \t]'
# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionban = <iptables> -I f2b-<name> 1 -s <ip> -p tcp -m multiport --dports 80,443 -j DROP
# Add the new IP ban to our nginx.repeatoffender file
echo '<ip>' >> /etc/fail2ban/ip.blocklist.nginx.repeatoffender
# Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: See jail.conf(5) man page
# Values: CMD
#
actionunban = <iptables> -D f2b-<name> -s <ip> -p tcp -m multiport --dports 80,443 -j DROP
# Remove IP from our nginx.repeatoffender file
sed -i -e '/<ip>/d' /etc/fail2ban/ip.blocklist.nginx.repeatoffender
[Init]
[Definition]
_daemon = fail2ban\.actions\s*
# The name of the jail that this filter is used for. In jail.conf, name the
# jail using this filter 'nginx-repeatoffender', or change this line!
_jailname = nginx-repeatoffender
failregex = ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) .* \S+\" (?:400|401|403|444) .+$
ignoreregex =
[nginx-repeatoffender]
enabled = true
backend = auto
mode = agressive
logpath = /var/log/nginx/*access.log
/home/websites/*/logs/access.log
filter = nginx-repeatoffender
action = nginx-repeatoffender[name=nginx-repeatoffender]
sendmail-whois[name=Nginx Repeat-Offender, dest=root, sender=F2B-Alerts-<fq-hostname>]
bantime = 86400 ; 1 day
findtime = 604800 ; 1 week
maxretry = 15
Optional - Abuse Reports
In the Main Configuration Files section above in this guide, I mentioned an option to report the violating host/IP address to the organization "in charge" of that host or IP address. Fail2Ban can be instructed to use a special action (action = %(action_xarf)s) to do that. Before I show you how to do it, let's examine what information Fail2Ban needs to send such a report.
First of all, a violating IP address must be banned by Fail2Ban. Once an IP address gets banned, Fail2Ban will use whois to find relevant information to generate an abuse report. Let's examine how that works by looking at an imaginary IP address that gets banned by Fail2Ban:
$ whois XX.XXX.XXX.XXX
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See https://apps.db.ripe.net/docs/HTML-Terms-And-Conditions
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to 'XX.XXX.XXX.0 - XX.XXX.XXX.255'
% Abuse contact for 'XX.XXX.XXX.0 - XX.XXX.XXX.255' is '[email protected]'
inetnum: XX.XXX.XXX.0 - XX.XXX.XXX.255
netname: AUTH-ORG
country: XX
org: ORG-PLXXX-RIPE
admin-c: PLXXXXX-RIPE
tech-c: PLXXXXX-RIPE
status: ASSIGNED PA
mnt-by: IP-RIPE
created: XXXX-XX-XXT20:14:04Z
last-modified: XXXX-XX-XXT20:14:07Z
source: RIPE
organisation: ORG-PLXXX-RIPE
org-name: Auth Org LLC
org-type: OTHER
address: xx-xx Xxxxxxx, x. XXXX, xx. 000
address: 000000 Utopia Square
address: Utopia
abuse-c: PLXXXXX-RIPE
mnt-ref: IP-RIPE
mnt-by: IP-RIPE
created: XXXX-XX-XXT20:10:41Z
last-modified: XXXX-XX-XXT20:10:41Z
source: RIPE # Filtered
role: Auth Org LLC
nic-hdl: PLXXXXX-RIPE
address: xx-xx Xxxxxxx, x. XXXX, XX. 000
address: 000000 Utopia Square
address: Utopia
abuse-mailbox: [email protected]
phone: +0 000 1111111
mnt-by: IP-RIPE
created: XXXX-XX-XXT20:09:34Z
last-modified: XXXX-XX-XXT20:10:30Z
source: RIPE # Filtered
% Information related to 'XX.XXX.XXX.0/24AS000000'
route: XX.XXX.XXX.0/24
origin: AS000000
mnt-by: IP-RIPE
created: XXXX-XX-XXT19:15:16Z
last-modified: XXXX-XX-XXT19:15:16Z
source: RIPE
% This query was served by the RIPE Database Query Service version 1.108 (ABERDEEN)
[Definition]
# bypass ban/unban for restored tickets
norestored = 1
#actionban = oifs=${IFS}; IFS=.;SEP_IP=( <ip> ); set -- ${SEP_IP}; ADDRESSES=$(dig +short -t txt -q $4.$3.$2.$1.abuse-contacts.abusix.org); IFS=${oifs}
actionban = oifs=${IFS};
RESOLVER_ADDR="%(addr_resolver)s"
if [ "<debug>" -gt 0 ]; then echo "try to resolve $RESOLVER_ADDR"; fi
ADDRESSES=$(dig +short +tcp -t txt -q $RESOLVER_ADDR | tr -d '"')
IFS=,; ADDRESSES=$(echo $ADDRESSES)
IFS=${oifs}
IP=<ip>
FROM=<sender>
SERVICE=<service>
FAILURES=<failures>
REPORTID=<time>@<fq-hostname>
TLP=<tlp>
PORT=<port>
DATE=`LC_ALL=C date --date=@<time> +"%%a, %%d %%h %%Y %%T %%z"`
if [ ! -z "$ADDRESSES" ]; then
oifs=${IFS}; IFS=,; ADDRESSES=$(echo $ADDRESSES)
IFS=${oifs}
(printf -- %%b "<header>\n<message>\n<report>\n\n";
date '+Note: Local timezone is %%z (%%Z)';
#printf -- %%b "\n<ipmatches>\n\n<footer>") | <mailcmd> <mailargs> ${ADDRESSES//,/\" \"}
printf -- %%b "\n<ipmatches>\n\n<footer>") | <mailcmd> <mailargs> $ADDRESSES
fi
# Server as resolver used in dig command
#
addr_resolver = <ip-rev>abuse-contacts.abusix.org
# Option: boundary
# Notes: This can be overwritten to be safe for possible predictions
boundary = bfbb0f920793ac03cb8634bde14d8a1e
_boundary = Abuse<time>-<boundary>
# Option: header
# Notes: This is really a fixed value
#header = Subject: abuse report about $IP - $DATE\nAuto-Submitted: auto-generated\nX-XARF: PLAIN\nContent-Transfer-Encoding: 7bit\nContent-Type: multipart/mixed; charset=utf8;\n boundary=Abuse-bfbb0f920793ac03cb8634bde14d8a1e;\n\n--Abuse-bfbb0f920793ac03cb8634bde14d8a1e\nMIME-Version: 1.0\nContent-Transfer-Encoding: 7bit\nContent-Type: text/plain; charset=utf-8;\n
header = Subject: abuse report about $IP - $DATE\nAuto-Submitted: auto-generated\nX-XARF: PLAIN\nContent-Transfer-Encoding: 7bit\nContent-Type: multipart/mixed; charset=utf8;\n boundary=%(_boundary)s;\n\n--%(_boundary)s\nMIME-Version: 1.0\nContent-Transfer-Encoding: 7bit\nContent-Type: text/plain; charset=utf-8;\n
# Option: footer
# Notes: This is really a fixed value and needs to match the report and header
# mime delimiters
#footer = \n\n--Abuse-bfbb0f920793ac03cb8634bde14d8a1e--
footer = \n\n--%(_boundary)s--
# Option: report
# Notes: Intended to be fixed
#report = --Abuse-bfbb0f920793ac03cb8634bde14d8a1e\nMIME-Version: 1.0\nContent-Transfer-Encoding: 7bit\nContent-Type: text/plain; charset=utf-8; name=\"report.txt\";\n\n---\nReported-From: $FROM\nCategory: abuse\nReport-ID: $REPORTID\nReport-Type: login-attack\nService: $SERVICE\nVersion: 0.2\nUser-Agent: Fail2ban v0.9\nDate: $DATE\nSource-Type: ip-address\nSource: $IP\nPort: $PORT\nSchema-URL: http://www.x-arf.org/schema/abuse_login-attack_0.1.2.json\nAttachment: text/plain\nOccurances: $FAILURES\nTLP: $TLP\n\n\n--Abuse-bfbb0f920793ac03cb8634bde14d8a1e\nMIME-Version: 1.0\nContent-Transfer-Encoding: 7bit\nContent-Type: text/plain; charset=utf8; name=\"logfile.log\";
report = --%(_boundary)s\nMIME-Version: 1.0\nContent-Transfer-Encoding: 7bit\nContent-Type: text/plain; charset=utf-8; name=\"report.txt\";\n\n---\nReported-From: $FROM\nCategory: abuse\nReport-ID: $REPORTID\nReport-Type: login-attack\nService: $SERVICE\nVersion: 0.2\nUser-Agent: Fail2ban v0.9\nDate: $DATE\nSource-Type: ip-address\nSource: $IP\nPort: $PORT\nSchema-URL: http://www.x-arf.org/schema/abuse_login-attack_0.1.2.json\nAttachment: text/plain\nOccurances: $FAILURES\nTLP: $TLP\n\n\n--%(_boundary)s\nMIME-Version: 1.0\nContent-Transfer-Encoding: 7bit\nContent-Type: text/plain; charset=utf8; name=\"logfile.log\";
# Option: Message
# Notes: This can be modified by the users
message = Dear Sir/Madam,\n\nWe have detected abuse from the IP address $IP, which according to abusix.com is on your network. We would appreciate if you would investigate and take action as appropriate.\n\nLog lines are given below, but please ask if you require any further information.\n\n(If you are not the correct person to contact about this please accept our apologies - your e-mail address was extracted from the whois record by an automated process.)\n\n This mail was generated by Fail2Ban in a X-ARF format! You can find more information about x-arf at http://www.x-arf.org/specification.html.\n\nThe recipient address of this report was provided by the Abuse Contact DB by abusix.com. abusix.com does not maintain the content of the database. All information which we pass out, derives from the RIR databases and is processed for ease of use. If you want to change or report non working abuse contacts please contact the appropriate RIR. If you have any further question, contact abusix.com directly via email ([email protected]). Information about the Abuse Contact Database can be found here: https://abusix.com/global-reporting/abuse-contact-db\nabusix.com is neither responsible nor liable for the content or accuracy of this message.\n
# Option: loglines
# Notes.: The number of log lines to search for the IP for the report
loglines = 9000
# Option: mailcmd
# Notes.: Your system mail command. It is passed the recipient
# Values: CMD
#
mailcmd = /usr/sbin/sendmail
# Option: mailargs
# Notes.: Additional arguments to mail command. e.g. for standard Unix mail:
# CC reports to another address:
# -c [email protected]
# Appear to come from a different address - the '--' indicates
# arguments to be passed to Sendmail:
# -- -f [email protected]
# Values: [ STRING ]
#
mailargs = -f <sender>
# Option: tlp
# Notes.: Traffic light protocol defining the sharing of this information.
# http://www.trusted-introducer.org/ISTLPv11.pdf
# green is share to those involved in network security but it is not
# to be released to the public.
tlp = green
# ALL of the following parameters should be set so the report contains
# meaningful information
# Option: service
# Notes.: This is the service type that was attacked. e.g. ssh, pop3
service = unspecified
# Option: logpath
# Notes: Path to the log files which contain relevant lines for the abuser IP
# Values: Filename(s) space separated and can contain wildcards (these are
# greped for the IP so make sure these aren't too long
logpath = /dev/null
# Option: sender
# Notes.: This is the sender that is included in the XARF report
sender = [email protected]
# Option: port
# Notes.: This is the port number that received the login-attack
port = 0
Remember, this will only work if an option action = %(action_xarf)s set in /etc/fail2ban/jail.local file. If it is, restart the Fail2Ban service, and abuse reporting should start sending reports automatically.
To conclude, I cannot guarantee everything above will work as described in this guide. Some modifications may be required, but making the necessary adjustments should not be hard. Thank you for reading, and please share the article if you liked it.
Comments
Thanks a lot, very nice guide to hardening fail2ban,