New Feature Survey

[matt]

I've been thinking about options for adding new features to Mail::Toaster. There are a few key areas that are ripe for improvement.

WEBMAIL: the webmail interface is already a step forward but, there are several problems with it. The authentication portion does not actually test to see if the user/pass is valid, it simply saves it. It should test first, and this requires writing a CGI to run on the server to verify authentication info. The JS in the browser would use AJAX to verify the auth parameters. The backend CGI must be written carefully to avoid security problems and it should also check the users postmaster status, only showing postmaster grade options (such as vqadmin, vpopmail CLI instructions, etc) to users who have authenticated with postmaster privileges.

Some time should be invested in enhancing the appearance of the interface as well. In writing it, I focused almost entirely upon the functionality. It should also look pretty.

ANTI-SPAM: Simscan is a very nice little program. It is significantly better than qmail-scanner, but its biggest problem is that it is married to ClamAV and SpamAssassin. ClamAV is quite good, and so is SpamAssassin but SA is very expensive (in terms of RAM and CPU). I would like to have other options for real-time (as in, during the SMTP conversation) message processing. For example, a pipeline where any one trigger can block a message, but all can be used individually or in aggregate.

On many servers, SA is way too resource intensive to be run for every message during the SMTP conversation. Instead, run dspam and/or bogofilter along with ClamAV during the SMTP conversation and use them to block 95% of the garbage. All three are small fast C programs and using them during the SMTP conversation would likely drop the CPU requirements of an email system by an order of magnitude. Then, use SpamAssassin during the mailbox delivery phase (called via maildrop). Expand the current training mechanisms to train each bayesian filtering system being used.

I would probably drop SA bayesian entirely. Not because it is poorly implemented, it actually works quite well but the time it takes to train SA bayes -vs- dspam is almost two orders of magnitude. SA still does all the other tests which make it an excellent filtering engine, but I think it best to limit its use to the messages that have made it past the lean mean, crap eating machines.

Another potential for saving lots of CPU cycles is to firewall IPs that send us virus/spam. A sensible default might be blocking an IP for 4 hours upon first offense. Upon sebsequent offense, use a quadratic block interval for their expiration date so the only blocks that last more than a few days are those that have repeatedly offended us.

SECURITY: Think toaster_setup.pl -s audit. The audit would check on a number of things, such as seeing if Apache is tightened down (set ServerSignature Off, ServerTokens ProductOnly, disable TRACE and TRACK, etc). Check SSHd and make sure it is not telling everyone about our server (the difference between this: 

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2006-10-07 14:08 CDT
Interesting ports on x.x.x.x:
Not shown: 1679 closed ports
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 4.2p1 (FreeBSD 20060930; protocol 2.0)
Service Info: OS: FreeBSD

and this:

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2006-10-07 14:07 CDT
Interesting ports on x.x.x.x:
Not shown: 1679 closed ports
PORT   STATE SERVICE VERSION
22/tcp open  ssh      (protocol 2.0)
1 service unrecognized despite returning data.

In addition, there are a number of hardening steps that can be done to FreeBSD to prevent other information leakage, like the OS and version. This is A Good Thing[TM] because script kiddies running around continually checking OS version and app versions with scanners to detech vulnerabilities.

FreeBSD has a great number of features for defending against network attacks (such as TCP_DROP_SYNFIN, DEVICE_POLLING, and of course, firewalling) but most are disabled by default. These should be checked to see if they are enabled and if not, provide a reference to where the operator can read about, understand, and implement them. Certain options can be implemented by the script automatically.

PASTEURIZE Mail Toaster

Perhaps it is time to cease development and put Mail::Toaster out to pasture. I have been getting quite a few job offers lately and all of them are reminders of how much earning power I am passing up while working on Mail::Toaster. I do enjoy working from home but the meager income I get from it is not enough for us to live on.

[LogicX]

On the topic of job-seeking; perhaps you could seek out a job with a mega-money company who enjoys throwing money at popular open source projects (*cough IBM *cough*)?

Its been known to happen, and maybe you could keep pumping out projects, make it even better, and maybe such a company would want to take clients that you do work for, and provide them with additional services, helping to turn your solution into one part of an all-around mail&hardware package of sorts?

You could proposition the mailing list for anyone with ties who could present such an idea, get your in touch with the right people.

[rainer_d]

Hm, difficult question.
I hope to be able to implement the toaster in a multi-host environment sometime 2007 (hopefully with a NetApp), so I have to see how much work it needs in that direction.
For the rest, it's pretty complete now, IMO.
Once the environment gets bigger, you need some sort of homegrown control-panel anyway and qmailadmin is of less help - but all the help-scripts etc. are really cool - I miss them every day on my non-toaster qmail install...


About the future of Mail::Toaster in general (and your personal future, too):
IMO, if you get a good job offer (well, a really good), you should take it. (If you think it brings you an advancement of some kind).

But before you take the job, do you think it would be possible to add some more record-types (AAAA, SRV etc.) to NicTool?
;-)

[Marc]

Anything that brings down the ram requirements is good news in my book.

On the job; I feel your pain. As much as I enjoy running my own shop, setting my own hours and knowing that every dollar coming in and out is hard fought and therefore just a little more satisfying I could make 3 to 4 times my current income working for someone else's company with someone else doing marketing and support and accounting and on and on and on.

MT was good enough in the 4.x versions and certainly in the 5.x incarnations that most of us users don't require too much support. You really either need a corparate sponsor with reasonably deep pockets or a ton more customers and I'm guessing that with Cpanel and Hsphere out there the market for MT is limited to folks who can appreciate it and have the modicum of skill required to use it. Which makes that number something fairly small I would think.

So as much as I hate to say it because knowing I can pull you in when there's trouble (and the quality of MT makes that VERY seldom) gives me the warm fuzzies the job must look tempting and I wouldn't blame you for taking the opportunity.

Any way it turns out though my personal thanks for MT as my life would be more difficult without it.

Marc

[TW_Adam]

Good day everyone
As a recent happy customer of Mr Matts skills, I would regret seeing this opportunity diminish by him getting a damn job.
But all the love of the community doen't pay the bills.

Have you thought about selling pre-setup turnkey boxes at all?
Just a thought because As a company we have to consider the cost of an actual body into our server side requirements.
Do the research on hardware compatability, warranty, tco and upgrade curve.  A turnkey box with a professional but generic interface would sell I think. Kind of plug it in, add domains and go sort of thing. (no doubt someone would try & host hotmail on it)

The only feature I see lacking with the toaster, is a clear interface to manage the many options available to the toaster owner. Like the RBL,s,  the tcp.smtp (real time blocking would be a godsend, as you explained...) (out side of a shell for non priv admins)

The toaster does a spectacular job, and isn't too too complex to work with as it is, even for a newbish admin like myself.  I have been reviewing the 2 older toasters in our mess, and I see alot of options that would have really helped reduce load and costs but were not being used.  A facility to see these options or configure them at a glance would be a feature worth having I think.

Awsome job on the toaster though.

Thanks Adam

[rlance]

Seems to me you should be able to find some corporate sponsors willing to support the project (you, for now) on the basis of your monitoring and improving their mail servers.

I would put your excellent development plan in the order of
1) cutting cycles for spam processing
2) webmail
3) security

[shealey]

Hi Matt,

On the subject of Anti Spam, I've been thinking along similar lines to how you describe banning spammers at the IP level after a 'first offence'.

An approach I'm experimenting with right now involves parsing offending source IP addresses out of the mailserver logs, and writing them into the database file of DJB's 'rbldns' server ...
(conveniently packaged along with tinydns!) along with a comment containing a date/timestamp which I can use later to remove or disable the entry. I am able to use the same rbl database software to whitelist stuff like the M$ servers to prevent them from getting accidentally blocked.

This does involve handling a connection on port 25 first before you can know that you want to turn the sender away though.

I figure the same thing can be achieved using another standard FreeBSD tool: ipfw2
The same kind of collection script as above could dynamically add 'deny' rules to block offending IPs. The man page says it can handle 65535 rules, so if you're only setting up each rule for a limited duration, it'd take a helluva mail storm to max it out!


Sean.

[matt]

An even better approach is using PF instead of IPFW.  I've used IPFW for years (like 6 or 7 it seems) and just recently switched to PF, because I wasted 4 days trying to get traffic from a FreeBSD jail (on localhost IPs 127.0.0.3-12) to the public internet via IPFW and natd. What a tangled ugly mess to debug.

But that's not the reason to use PF for this task. PF has the handy ability to create tables. This is from the PF manual:

A table is used to hold a group of IPv4 and/or IPv6 addresses. Lookups against a table are very fast and consume less memory and processor time than lists. For this reason, a table is ideal for holding a large group of addresses as the lookup time on a table holding 50,000 addresses is only slightly more than for one holding 50 addresses.

PF is quite wonderful and does so much more than IPFW dreams of.

[shealey]

Cool, sounds *ideal*

Sean.

[jerm]

Matt,

I'm adding another vote to Anti-spam.  making your afore mentioned changes to simscan would open of a world of flexability in a couple of ways.

first, it would allow the competant sysadmin to be much more creative in their spam-dodging rules.  That creativity could span the next great spam fighting tool, or at least let them customize things to fit their liking better.

It would also allow you the opportunity for income by giving the incompetant sysadmin more room to hang themselves..... win-win.