Dictionary Attack

[shealey]

For several days now, our toaster has been brought to its knees by what is almost certainly the results of a dictionary attack using one of the domain names we host. Basically some scumbag has been sending out streams of spam with random forged sender addresses but using a real domain name. We're getting torrents of inbound mail which are the bounces from hundreds of legit mailservers, which are as a result getting past our RBL checks forcing the mailserver to handle them.

Can anyone offer advice on possible action we could take to get the toaster processing mail again?

We've got so many SYN_RECV connections waiting on port 25 to be handled that legit mail is not getting through, and clients are now complaining.
I've *reduced* the number of concurrent connections allowed because they were maxing out CPU resources and nothing was getting processed at all. This has allowed the box to actually start handling mail again, but nowhere near fast enough to cope.
I've currently got 4 RBL lookups, but I'm going to reduce this to the two which are making the most blocks. This should reduce the overheads of handling each connection.
We're using simscan + spamassassin + clamav - should I try using maildrop? Which is faster?

Sean.

[shealey]

Update:

The domain being targetted had a 'catchall' address configured, which meant that qmail was accepting messages to any random generated fake account at that domain.

The first course of action we tried was simply setting the catchall for the domain to 'delete'.

Unfortunately this doesn't quite work how you might think - the mailserver still accepts the message, feeds it through the spam/virus checks, but simply doesn't deliver it to any mailbox. The side effect of this is that Spamassassin still has to handle each and every one of the torrent of invalid messages before the MTA gets to the point where it decides to throw it away. An undesirable implication of this is that Spamassassin automatically updates its Bayesian scoring data from some of these messages if it decides they are clean, which ends up skewing the spam detection.

The second course of action was to reconfigure the catchall to bounce-no-mailbox.

Doing this is less desirable because it can potentially feed the bounce mail storm, and did increase the server load slightly even though Spamassassin was no longer getting hammered. Then I discovered there are two directives in the vpopmail tcp.smtp file which can be modified which control 'tarpitting':
CHKUSER_RCPT_LIMIT
CHKUSER_WRONGRCPT_LIMIT
(Refer to http://www.interazioni.it/opensource/chkuser/documentation/chkuser_settings.html)

The default settings were 15 and 3 respectively. Since I wanted to minimise the amount of time the server was wasting on each of these invalid incoming connections, I've reduced them to 5 and 1 - effectively saying if you try to deliver *anything* to an account that doesn't exist, we're not going to waste time talking to you any more. Immediately after recompiling the tcp.smtp.cdb database, the logs started showing streams of rejected connections (about 2 a second), and our load dropped noticeably. With the CPU resources that freed up, I was able to increase the number of concurrent connections we could handle which has resulted in a whole lot more legit mail making it through the blockade. It has been a delicate balancing act - it I allow slightly too many connections, the increase in load results in 'MySQL Server Has Gone Away' errors.

While researching this, I just found a writeup where a guy has put together an IP banning solution for this exact problem, which I'm going to have a go at when I get more time to work on the server later today:

http://www.bantcp.com/bantcp.htm

I hope some of this helps someone else who's been hit the same way.

Sean.

[matt]

The domain being targetted had a 'catchall' address configured, which meant that qmail was accepting messages to any random generated fake account at that domain.

The first course of action we tried was simply setting the catchall for the domain to 'delete'.

Unfortunately this doesn't quite work how you might think - the mailserver still accepts the message, feeds it through the spam/virus checks, but simply doesn't deliver it to any mailbox. The side effect of this is that Spamassassin still has to handle each and every one of the torrent of invalid messages before the MTA gets to the point where it decides to throw it away. An undesirable implication of this is that Spamassassin automatically updates its Bayesian scoring data from some of these messages if it decides they are clean, which ends up skewing the spam detection.

The second course of action was to reconfigure the catchall to bounce-no-mailbox.

Doing this is less desirable because it can potentially feed the bounce mail storm, and did increase the server load slightly even though Spamassassin was no longer getting hammered.

Correct, you do NOT want to set the catchall to "delete" unless you want to process the message, and then discard it. That's pretty expensive since processing requires virus and spam checking before you even queue the message. The message doesn't get deleted until later during delivery.

However, "bounce-no-mailbox" is the right solution. I don't understand what you mean by "it can feed the bounce mail storm." Apparently you don't understand how bounce-no-mailbox works on a Mail Toaster. With BNM, if the mailbox doesn't exist, it rejects the message during the SMTP conversation with a 550 error. The remote mail server, which accepted the undeliverable message in the first place, is stuck with the message and is now burdened with disposal instead of you. How does this "feed" a bounce mail storm?

Then I discovered there are two directives in the vpopmail tcp.smtp file which can be modified which control 'tarpitting':
CHKUSER_RCPT_LIMIT
CHKUSER_WRONGRCPT_LIMIT
(Refer to http://www.interazioni.it/opensource/chkuser/documentation/chkuser_settings.html)

The default settings were 15 and 3 respectively. Since I wanted to minimise the amount of time the server was wasting on each of these invalid incoming connections, I've reduced them to 5 and 1 - effectively saying if you try to deliver *anything* to an account that doesn't exist, we're not going to waste time talking to you any more. Immediately after recompiling the tcp.smtp.cdb database, the logs started showing streams of rejected connections (about 2 a second), and our load dropped noticeably. With the CPU resources that freed up, I was able to increase the number of concurrent connections we could handle which has resulted in a whole lot more legit mail making it through the blockade. It has been a delicate balancing act - it I allow slightly too many connections, the increase in load results in 'MySQL Server Has Gone Away' errors.

Tuning chkuser was certainly a good choice when coping with an attack like this. Good job.

Matt

[shealey]

However, "bounce-no-mailbox" is the right solution. I don't understand what you mean by "it can feed the bounce mail storm." Apparently you don't understand how bounce-no-mailbox works on a Mail Toaster. With BNM, if the mailbox doesn't exist, it rejects the message during the SMTP conversation with a 550 error. The remote mail server, which accepted the undeliverable message in the first place, is stuck with the message and is now burdened with disposal instead of you. How does this "feed" a bounce mail storm?

You're quite right in that I've somewhat misunderstood the bounce mechanism. Thanks for the correction.


Sean.

[shealey]

Update:

Now that I've fixed a problem with the TLS cert generation, the system is hardly noticing the steady stream of mail with invalid recipients. It seems that the reconfig discussed in this thread is doing the job nicely.

Sean.