Floodmon : SYN flood attacks monitoring (for Linux)


Floodmon is a small Perl daemon used to monitor a server in order to detect SYN flood attacks, to alert the administrator and to attempt to mitigate the attack up to a certain level.

I - Overview :

Floodmon works mainly by tweaking the TCP/IP stack : activation/deactivation, queues size, timeout and retransmissions etc. All modifications are done in real time in the '/proc' pseudo file system throughout the SYN flood attack. It can also null-route entire subnets (/8, /16 or /24) at a high rate. It has 4 protection levels, each one having its own configuration and can be quite aggressive if needed. It can alert the administrator of a possible attack by email and/or SMS, email detailed reports and even include a capture of the SYN segments for analysis. It can also display some useful network stats.

II - Setup :

The daemon, floodmon, must be installed in '/usr/sbin/' directory, chmoded to 0700 and only accessible to 'root' (chown root:root). Its configuration file 'floodmon.conf' has to be edited and moved to '/etc/'. Two other scripts are also included, one for munin-node and one to start the daemon at boot time (see 'README' for more informations).
Floodmon creates a log file '/var/log/floodmon.log' and backup all user system configuration in '/var/cache/floodmon/' before doing any modification.

III - Programs & Perl modules needed :



Floodmon does not need any other command or third party program. It doesn't use iptables or the netstat command either (it gets all connection informations from '/proc/net' and '/proc/sys/net', just like netstat does).

IV - Parameters :


  Usage :
    --help              : display this menu.
    --debug             : run in debug mode (no daemon).
    --daemon            : run in daemon mode (default).
    --reload            : reload configuration file (/etc/floodmon.conf).
    --stop              : stop and restore user configuration (default).
    --stop-save         : stop but doesn't restore user configuration
                          (TCP/IP stack, routing table).
    --stats             : display network statistics.
    --munin-node        : daemon statistics to use with munin-node.
    --sms-test          : test SMS alerts.
    --capture <num>     : capture <num> SYN packets.

--help : display help menu and version number.

--debug : run the script in debug mode and display all operations in the terminal. However, it does not run in daemon mode and to stop it you must use the [CTRL-C] sequence. That option is interesting if your server is under attack and you want to know what is going on.

--daemon : run in daemon mode. That is the default option.

--reload : if you have to make some change to the configuration file, use this parameter so that the daemon will reload it.

--stop : stop the daemon. Any modification it has done will be deleted and your original configuration will be fully restored. That is the default behavior.

--stop-save : stop the daemon. All modifications done will be kept when quitting (TPC/IP stack, null-routed blocks). To restore your original configuration you will need to use the backup files located in /var/cache/floodmon.

--stats* : display network statistics (IP, TCP, UDP, ICMP, routing table, SYN etc).

--munin-node* : floodmon stats (alert levels) to use with munin-node. You can use the floodmon-munin file and copy it to your /etc/munin/plugins/ directory. Munin-node 'config' parameter is also handled by floodmon.

--sms-test* : allows to test SMS alerts.

--capture <num>* : create an on-demand capture of the last <num> SYN packets and save it to disk or email it to the admin.

* The last 4 options can be executed regardless of the status of floodmon (active or not).

V - User configuration file :

The configuration file 'floodmon.conf' must be edited and then copied into the '/etc/' directory. It contains some variables that must be configured as soon as floodmon is installed (network, email etc) and others that are already configured by default.
If you make changes to this file while floodmon is running, use the # floodmon --reload command so that the daemon will reload it.

VI - Variables to configure :

INTERFACE : your network interface (ex : 'eth0', 'venet0'...).

EMAIL_ALERT : send an email when the alert reached a specific level. By default, it is level 2.

EMAIL_ALERT_FREQ : emails frequency during an attack. You will not receive more than one email every EMAIL_ALERT_FREQ minutes when your server is under attack. Default value is 30mn.

ADMIN_EMAIL : as the name says, the administrator email where the alert will be sent.

DUMP_SYNPACKETS : floodmon can attach to its email alert a capture of the last SYN packets received. The file uses 'pcap' format and can be open/read with a network analyzer like Wireshark (Windows · OS X · Linux).

SMS_ALERT : send a SMS alert to the admin. You must be using a service (VOIP etc) that allows you to send SMS from a simple URL (HTTP or HTTPS). The whole URL + its parameters should be included.

SMS_ALERT_FREQ : SMS alert frequency (in hours).

FLUSH_FREQ : null-routed IPs flush frequency. You shouldn't keep IPs blocked forever, thus this option will flush the routing table every FLUSH_FREQ minutes. By default, every 10mn.

NETMASK : network mask used to nullroute IPs (see NULLROUTE_SUBNET below). Value can only be 8 or 16 or 24. Default value : '16'.

LOG_LEVEL : log file verboseness. Default value : '2'.

NO_CONNTRACK : enable this variable if your server doesn't use the connection tracking table. Floodmon will not modify its values (size, buckets and timout). Default value : '0'.

VII - Variables configured by default :

Those variables are used to mitigate the attack and have been configured with the best values. They have been tested during both real and simulated attacks and have given some pretty good results. Don't change them unless you know what you are doing !

MAX_SYN (N == max number of SYN) :
This is the maximum number of SYN_RECV allowed for the current level. When that number is reached or exceeded, floodmon switches to the next alert level.
By default, 350 simultaneous SYN are allowed in the 1st level before the alert is triggered. Even if that value may seem a bit low, there shouldn't be any reason to change it, except of course if you receive too many false alerts.
For the second level, the maximum amount of SYN_RECV allowed is 500 and 1,000 for the 3rd level. For the 4th last level, 'MAX_SYN' value just doesn't matter but of course should be kept higher than the value of the previous (3rd) level.

SYN_COOKIES (0 = deactivated, 1 = activated) :
Syncookies (see syncookies.c, sysctl_net_ipv4.c and tcp_ipv4.c) have proved to be very effective during a SYN flood attack. Instead of keeping and filling up the backlog queue (tcp_max_syn_backlog) with the received SYN data, it is encrypted and sent along with the SYN/ACK packet and decrypted/verified upon the receipt of the client ACK.
Floodmon deactivates them and will only activate them when an attack will be detected. It should also be noted that even when enabled, syncookies are not used until the tcp_max_syn_backlog and the application listen queue are getting full. Finally, Linux is an OS that has also implemented some strong features to help to mitigate SYN flood attacks (minisock etc).

MAX_SYN_QUEUE (N == size of SYN backlog) :
The maximum number of queued connection requests which have still not received an acknowledgement from the connecting client (tcp_max_syn_backlog). When the queue is full, any new connection attempt will be dropped by the kernel (unless syncookies are activated). In Linux, its size is hard-coded to a value of 256 (cf request_sock.c) and adjusted at runtime according to the server available RAM (1024 if more than 256MB of RAM).
This value is quite low, hence floodmon adjust it from 10,000 to 200,000 depending on the current alert level.

SOMAXCONN ( N == backlog size) :
Limit of socket listen() backlog, known in userspace as SOMAXCONN.
When an application calls the listen() function, it gives 2 parameters : sockfd (file descriptor that refers to a socket) and backlog (queue length for completely established sockets waiting to be accepted). If backlog has a higher value than SOMAXCONN, it will be lowered to the value of SOMAXCONN.
During a SYN flood attack, SOMAXCONN is as important as tcp_max_syn_backlog : if there are enough warm entries in syn queue, new connection attempts will be dropped (see tcp_ipv4.c).
In Linux, SOMAXCONN size is hard-coded to a very low value : 128 (see socket.h). Fortunately, its value can be dynamically adjust in '/proc/sys/net/core/somaxconn'. Floodmon set it from 10,000 to 60,000 depending on the current level alert.


MAX_SYNACK_RETRY (N == maximum number of retransmissions - from 0 to max 255) :
The maximum number of times a SYN/ACK segment for a passive TCP connection will be retransmitted (tcp_synack_retries).
Upon receipt of a SYN packet, the server will reply with a SYN/ACK. If the client doesn't reply with a ACK packet, just like in the case of a SYN flood attack, the server will send it again several times. By default in Linux a SYN/ACK is retransmitted 5 times (see tcp.h) : first time 3s after the first one, then 6s after the first retransmission, then 12s after the 2nd one, 24s after the 3rd one and, finally, 48s after the 4th one. That last retransmission will only timeout after another 96s. That is a total of 189 seconds and also the reason why, during an attack, the netstat command will display so many SYN_RECV.
Floodmon will reduce the number of retransmissions to 3 (45s), and will keep lowering it at higher alert levels. In the last one, 4th level, it will not allow any SYN/ACK retransmission. Even if that seems a bit harsh, it is simply identical to the use of syncookies which also prevent any retransmission of a SYN/ACK packet.

Connection tracking table :
This table allows the firewall (netfilter) to keep track of one of more connections according to its state (NEW, ESTABLISHED, RELATED etc). Although very useful, it could overflow during a DDoS attack and any further incoming connection would be dropped by the kernel. In that case, kern.log and dmesg command would display : 'nf_conntrack: table full, dropping xx packet".
At start-up, floodmon will check which, if any, table the server is using (ip_conntrack which handles IPv4 or nf_conntrack which handles both IPv4 and IPv6). If your server doesn't use the connection tracking table, you are advised to set the NO_CONNTRACK variable in floodmon.conf.
Floodmon handles 3 variables related to that table :
LOOP_DELAY (N == seconds/milliseconds) :
Number of seconds/milliseconds floodmon will pause between each check of your system. By default, connections are checked every 15s in the 1st level, 1.5s in the 2nd one, 1s in the 3rd one and 0.5s in the last one.
Note that floodmon retrieves all connection states directly inside '/proc/net' and '/proc/sys/net' without using a third-party program and therefore, even if LOOP_DELAY has a very low value, it uses very little CPU and RAM (approx 2MB in standby).

NULLROUTE_SUBNET (0 == deactived, N == max number of IPs) :
Floodmon doesn't use iptables to block IPs, it uses the routing table to null-route them. Even if the routing table is not intended to replace the firewall, Floodmon uses it because, when a IP is null-routed, whether it is spoofed or not, the kernel will stop any further retransmission of SYN/ACK segments hence it will get rid of the half-open connections (SYN_RECV) quicker that way as they will be removed from the tcp_max_syn_backlog at the end of the current retransmission. Note that, unlike a firewall, the routing table can only block a destination, not a source therefore it is the response to the SYN packets that Floodmon will block, not the incoming SYN packets.
IPs are not blocked individually, but by entire subnets using /8, /16 or /24 network mask (see NETMASK above). The default value is 16, which means a whole class B (255.255.0.0 mask, CIDR /16). If there are more than NULLROUTE_SUBNET IPs from the same subnet, the whole class will be null-routed. For example, if you set NETMASK == 16 and NULLROUTE_SUBNET == 200, if there are more than 200 SYN from remote IP addresses 200.10.xx.xx, floodmon will block all IPs from 200.10.0.0 to 200.10.255.255
Null-routed IPs are flushed by default every 10mn (FLUSH_FREQ variable in 'floodmon.conf').

BEST_EFFORT (0 == deactived, 1 == actived):
This variable is used to modify miscellaneous TPC/IP options that could help during an attack. If you want more informations about them, use the command # man tcp.

VIII - Server setup :

When using floodmon, there are some important points you should take into consideration :

- Apache :
As mentionned above, any application calling the listen() function must pass two parameters, one being the size of the backlog queue. Apache v2.x* uses a backlog named DEFAULT_LISTENBACKLOG and hard-coded with a value of 511 (see mpm_common.h). However, as we already know, Linux will lower the application backlog value if it is higher than the SOMAXCONN value. So, when running Apache, it will request a backlog value of 511 but it will be lowered to 128 only, the SOMAXCONN default value.
That difference between Apache and Linux default values makes some problems : if you are under a SYN flood attack on port 80, you will notice that Apache will quickly become irresponsive. In fact it is not Apache but the kernel that will reject any new connection attempt because the listen() and SYN backlogs are both linked together when it comes to handle incoming connections. Apache isn't responsible (it will never know about the attack !) because, and that is a SYN flood attack goal, no connection is established. SYN packets will remain in the kernel half-open connection backlog.
Fortunately, DEFAULT_LISTENBACKLOG can be changed by adding the ListenBacklog directive to apache2.conf (or httpd.conf). Since floodmon gives generous values to both SOMAXCONN and tcp_max_syn_backlog (10,000 minimum), you are advised to greatly increase Apache backlog as well, specially when using floodmon if you want to increase your chances to handle an attack on port 80. A minimal value of 5120 seems to be a comfortable one, but it will depends on your server capacity, the number of visitors and of course, the magnitude of the attack you are facing as well.
You must edit your apache2.conf or httpd.conf and add the value you want to use. For example, with a value of 5120 :
  ListenBacklog 5120
Then modify SOMAXCONN value (only if floodmon is not already running !) :
  echo 5120 > /proc/sys/net/core/somaxconn
You must restart Apache after the modification.

* Apache v1.x uses the same variable since version 1.3 but named DEFAULT_LISTEN_BACKLOG and having a value of 512 (see httpd.h). It can also be changed with the ListenBacklog directive as described above.


- Nginx :
Nginx backlog can also be changed in the configuration file (inside the listen directive) :
  backlog=5120
For more info, see Nginx online help.
Change this value, change SOMAXCONN (only if floodmon is not already running !) and restart Nginx.


- Lighttpd :
Lighttpd backlog size is hardcoded to 1024 (128 * 8) and cannot be change in its configuration file. You must download the source code and edit network.c :
  if (-1 == listen(srv_socket->fd, 128 * 8)) {
    log_error_write(srv, __FILE__, __LINE__, "ss", "listen failed: ", strerror(errno));
    return -1;
  }
Change this value, recompile, then modify SOMAXCONN and restart Lighttpd.


If the attack targets another port (POP3 server, FTP, SMTP...) check the server doc to see if you can increase its backlog. Otherwise, you can always change its value in the source code and recompile (just look for a call to the listen() function).

- iptables :
Since version 0.9.4, floodmon does not use iptables any longer.

There are many iptables rules on the Internet to help to 'fight' against a SYN flood, some of which you really should avoid, especially if you use floodmon. For example :
  iptables -N syn-flood
  iptables -A INPUT -p tcp --syn -j syn-flood
  iptables -A syn-flood -m limit --limit 5/s -j RETURN
  iptables -A syn-flood -j DROP
That rule rates limit any new TCP connection to 5 per seconds. There is no real difference between using that rule and shooting yourself in the foot.
Even if a website receives only few hundreds visitors pers day, such a rule will affect them, slowing down their attempts to get connected, forcing them to retransmit SYN packets several times and, during an attack, those retransmissions will even increase the flood. Also, the SYN rate limit will not only affect the HTTP server but the whole TCP protocol (POP3, SMTP, FTP etc). An attacker could send an as little as 10 SYN/s flood, that's a ridiculous 80bits/s bandwidth, to block any access to the server, including SSH access. In other words, a script-kiddie could 'flood' the firewall only with his iPhone...
If you are using such a rule, you must remove it. It is not compatible at all with floodmon.

IX - Download :

floodmon.tgz - (c) Jerome Bruandet

View source


spam sucks, it's gay !