Logwatch is a very flexible and customizable log watching system for lazy system administrators. It checks the logfiles regularly and sends custom mail reports – very useful and way better than daily manual checking of the logs.

Unfortunately the configuration for MySQL, PHP and the Apache error_log is missing; so let me share the configuration scripts:

Logwatch configuration for PHP
Interestingly there is no logwatch configuration for PHP error_log files – and all a search revealed are some outdated files which don’t work anymore. Here are the adopted files that allow you to get notified about PHP errors:

• logfiles_php.conf – place this file in /etc/logwatch/conf/logfiles/php.conf and adopt the path to the php error_log file (or just use the Apache error_log)
• services_php.conf – place this file in /etc/logwatch/conf/services/php.conf
• scripts_php – place this file in /etc/logwatch/scripts/php and make it executable

Logwatch configuration for MySQL
Also a configuration for MySQL is missing in the default configuration; here are the configuration files:

• logfiles_mysql.conf – place this file in /etc/logwatch/conf/logfiles/mysql.conf and adopt the path to the MySQL logfile
• services_mysql.conf – place this file in /etc/logwatch/conf/services/mysql.conf
• scripts_mysql – place this file in /etc/logwatch/scripts/mysql and make it executable

Logwatch configuration for Apache’s error_log
Now hat really made me wonder is the at the httpd access_log is monitored but the error_log is left out, so no details about errors are included in the logs. This configuration ignores PHP errors and includes all httd errors in the logwatch output:

• logfiles_http-error.conf – place this file in /etc/logwatch/conf/logfiles/http-error.conf and adopt the path to the httpd error_log file accordingly
• services_http-error.conf – place this file in /etc/logwatch/conf/services/http-error.conf
• scripts_http-error – place this file in /etc/logwatch/scripts/http-error and make it executable

To test that the individual scripts work use this command:

/usr/sbin/logwatch –detail high –print –service $SERVICE –range today –debug 0

Replace $SERVICE with either php, mysql or http-error and set debug to 5 if you want to see more (but not too much) debugging information.

All the logwatch configuration scripts for MySQL, PHP and the Apache error_log can be found here.

Update: There was an error in the scripts_mysql file, fixed now.

On December 5th 2007 I could not believe my eyes when I had a look at the log files – someone was trying to send a severe amount of spam to my server. Someone? Yes, “someone” targeted his botnet against a hosted domain and this botnet performed a dictionary attack. So what did I do? What can you do?

First of all let’s see what happens in detail – a bad guy ™ who controls a botnet says he wants to spam a certain domain – now the bots start sending mails to that domain using random (dictionary based) recipients that don’t exist on the server, i.e.

• Firmi@domain.com
• Smalleyjducz@domain.com
• Mikk-Smollen@domain.com
• stephaniereka@domain.com
• Kimplin@domain.com
• jensenybl@domain.com
• Kajumova@domain.com
• …

By running the following command against the mail-log you can see that most of the sender IP’s are unique:

cat mail.log | grep NOQUEUE | awk -F[ '{print $3}' | awk -F] ‘{print $1}’ | sort | uniq -c | sort -n

You will also see that most IP’s are just contacting the server once.

So the first thing to do is blocking the “top 50″ (or top 100) of those IP’s using iptables. This helps to filter IP’s that are coming back on a regular basis (be sure you don’t filter any of your users!) and works, but compared to ~1000 unique IP’s that are ever-changing it does not really work out. This manually IP filtering shows up in the graph around 14:45 and 18:00.

You are lucky if you are running a greylisting service, that way ~90% of all mails are rejected the first time and the spam bots never come back to try a second time. This is also visible in the graph – usually there is about 1 mail per minute filtered, during the attack this number went up to 22/minute. What I did was increasing the greylisting time-out time to 3000 seconds (50 minutes) to block the bots even longer. But still some of them tried very often and did not get blocked by manually created iptables rules – they came through to the mail server.

Luckily catch-all was disabled on this domain, so the other 10% that came through have been rejected with a “user not found” message. I would estimate that not a single mail has been delivered to any inbox… so, as no mail hit Spamassassin and ClamAV (which are expensive processes) the load of the machine was still fine and it had reserves left even at the peak of the attack around 18:00.

What else to do? Actually, there is nothing you can do if you have such an attack except waiting until it’s over. Next steps would have been disabling the targeted domain in the mail server (which makes the denial of service attack successful) or shutting the service down – which also has an impact on other hosted domains. Both options are only the last resort and should be taken if the mails start coming through to Spamassassin and ClamAV and the server load increases dramatically. But it’s better shutting one service down than to loose the whole machine.

So, sit down, have a drink and wait.

The conclusion is that you can’t really do anything against such attacks as they are coming from a lot of different hosts – but the spammers will stop by themselves after a while and it’s going to be over soon. But you better be prepared – be sure you use a greylisting filter and catch-all is disabled. Also closely monitor the system load and be prepared to shut the service down (inform your users before you do that!).

Three additional comments: the spike at 21:15 was a user sending his evening-bunch of “funny fun mails” to a lot of recipients (he should use Youtube for those videos… *grml*). No user reported an issue with the server, it was responding as usual during the attack. Traffic did not increase rapidly during the attack.