TNPI - should I use RBLs?

home : internet : mail : toaster : filtering : RBL (real time blacklists) : should I use RBLs?

Should I use RBLs?

Grab a cut of coffee, it's SMTP lesson time.

The type of statistics you'll see in your logs regarding SMTP blocks will reflect several things of interest (that are related to your question). The first, and most obvious is how you are using the RBLs.

The most important setting in that regard is rbl_enable_soft_failure in your toaster-watcher.conf file. If you have soft failure enabled, then when you deny a mail agent access, you are doing so with a temporary failure, which amounts to saying "try again later". The idea behind that is that if it's a legit mail server, it gives them a chance to fix their problem and then try sending again.

I'm not in that camp. My email addresses have been plastered on web pages for years, on mailing lists, usenet forums, etc. Thus, my address(es) are included in many of the spammers databases so I'm going to be a target regardless. I want to stop them dead so I set rbl_enable_soft_failure to 0, which chooses a 553 error "sorry go away, permanent error, you spammer pig". This effectively tells the remote server, "don't even bother trying again, it won't work."

Reason #1: It costs me less to use RBLs.

Obviously with the latter arrangement, you'll see a lot fewer RBL blocks. Many spammers don't honor the 553 error and try again anyhow but that's quite OK. The majority of mail servers (themselves victims of naughty people) will get the 553 and bounce the message, no longer trying to contact me. This is the very same reason why it's better to use a SMTP 553 error instead of simply packet filtering their IP. Sure, they're blackholed, but they'll get a SMTP unreachable error and just keep trying. I want the spam traffic destined to my server to go away, not add to the cost of my system (via connectivity). When your server has thousands of mailboxes on it, these policy decisions make very real and measurable differences.

Doing a RBL lookup on a message at SMTP invocation time is extremely cheap. It's a simple DNS query so it's ultra lightweight. I may even have it cached from the last time the spammer tried connecting. Further, we'll probably do the lookup when SpamAssassin processes the message, so it costs me almost nothing. If I can deny a spam message early in the SMTP phase, before having to pass it to a content scanner (medium cost), virus scanner (medium to expensive), bayesian filtering (low cost), spamassassin network tests (low - medium), and then maildrop filtering, then it's well worth it to me.

Reason #2: Increase the cost to the spammer

The next reason for RBL usage is more political. The spammers revenue (that they charge advertisers) is based on message deliveries. They maintain databases of email accounts that they can deliver to. They have developed very sophisticated ways of determining if a message got delivered (well beyond "did the SMTP server accept it") and further, if it landed in a real persons mailbox. These include delivery notifications, embedded HTML web bugs, and other clever mechanisms.

Very few Mail User Agents are clever about dealing with spam messages. Unfortunately, the ones that are (like Mail.app) are not the most popular. A lions share of users use that icky software from Redmond which is a spammers dream come true. Even if I tag and bag the spam to a special folder, the end user may peruse through it, opening the message "just in case" and thus letting the spammer know the message was delivered successfully. Yay, spam 'em again!

So, if I can block the message entirely (not just from my mailbox, but from ever landing on my server) then the spammer will never get back any indication of success. This increases the likelihood of being removed from the naughty miscreants list and thus reduces the likelihood of getting spammed from him again in the future. This is good, very good.

Sounds great in theory right? So how does it play out in real life? I have quite a few "case studies" we'll call them, ranging from my own personal mail servers to mail systems with tens of thousands of mailboxes on them. In general, the spam to ham ratio of messages on the internet is rising. This trend has been steadily growing in the spammers favor for the past few years. However, in every case where the mail server has been consistently employing RBLs, the number of RBL blocks I see is gradually decreasing. This suggests that either RBLs are becoming less effective or some other unknown factor(s) are at work.

Let's look at my mail server as an example. Over time my message volume on my server has continued to rise (as I add more users), so we should expect the level of spam messages as well as ham messages to be steadily increasing (or the number of RBL blocks to increase). The effectiveness of my spam filters has been fairly constant (>98%) for the past six months yet the number of spam messages being delivered on my mail server has steadily dropped. What accounts for the difference?

Reason #3: Keeping spam messages away from end users

About a year ago I was watching my mail server logs and noticed that quite a few of my mail users (of whom I know personally) were getting extraordinarily high levels of spam being delivered to them. I then decreased my spam tolerance by blocking mail servers without DNS (which required whitelisting a fair number of mail servers), increasing my RBL usage. Immediately after that the number of RBL hits spiked and then began a gradual decline ever since. This reduction of traffic is due almost entirely to the RBL usage as I tested the reverse DNS blocking at various times and ultimately removed it.

Approximately six months later I did another round of decreased spam tolerance and enabling the spam filtering for these users, all of whom are not very sophisticated mail users. They don't know or care about mail servers, filtering, etc. They just want their email to work. Overnight the levels of spam in their inboxes dropped to almost nothing (a condition I'm accustomed to living with). This, of course made them all very happy. It also gave me a way to track the effectiveness of the RBLs, as I had a very accurate indicator of their effectiveness. RBLs only caught about half the spam, but that's still a remarkable achievement for something that costs so little to employ.

In summary, RBLs aren't a "one stop shopping" spam solution, but they are a very effective tool.

Reason #4: Spammers hate RBLs

There is good reason why RBL operators are targets of DoS attacks. These days running a RBL means making sure you and your upstream provider are prepared to deal with massive DoS attacks, sometimes lasting over a week. These DoS attacks can cost many thousands of dollars in excess bandwidth fees, lost revenues, and related costs. This is the reason for several of the RBL operators decisions to step out of the game, including osirusoft which was one of, if not the most effective free RBL. If RBLs weren't effective, they wouldn't be targets.

Unfortunately, the spammers have millions of advertising dollars at stake. When a RBL is as effective as Osirusoft, it measurably effects their bottom line, and thus becomes a target. If I can hurt the spammers where it counts (in their wallet), then in my book, that's A Very Good Thing.

Last modified on 4/28/05.