Vpopmail auth failing under heavy load

vantage255 · April 25, 2005, 02:23:29 PM

I am running a toaster based install that I have integrated into our control panel package. It has been working great untill we hit 11K domains. the load on the server is not particularly high (0.6 to 1.0) but each day, at the peak of the day, accounts are unable to auth. This will only happen for a few dozen people a day and I have not been able to reproduce it on any accounts that I have created on the server. I have seen the log messages though. They show that the account failed to auth and they give the password used. It is always the correct password though. I am able to log in after the fact usign the same information.

This is starting to get troubling.

Has anyone else had these issues in the past?

matt · April 25, 2005, 11:02:26 PM

And you are using what form of database backend?

And what version?

And what does the database servers query load look like?

And what version of Mail::Toaster?

And what version of vpopmail?

Your question is quite vague on details.

vantage255 · April 25, 2005, 11:18:33 PM

Sorry about the sparsity of info. I had been up for about 30 hours... (bad night)

I am running vpopmail 5.4.7
Mysql 4.1.7
I set the server up usign the mail::toaster install but have modified it a good deal away from there. The mail::toaster module is no longer called by anything and the only cron running is the clearopensmtp cron.

I am running SA out of .qmail files from indevidual users home directories and I am running ClamAV directly though the qmailqueue patch. A coworker has rewriten Qmail-ClamC and we have gained a good deal of speed here. (I have code if you are interested. It uses RAM now instead of writing to disk. It has proven stable over a few dozen high load boxes and is a LOT faster).

The MySQL server load is not terribly high. It is serving about 80 to 90 QPS durring busy times. Mytop is showing me about 300K Cache hits per second and 12 threads open with a key efficiancy of 99.6%

I guess this isnt truly a toaster question but more of a vpopmail question. This just seamed like the place to ask.

matt · April 26, 2005, 10:08:11 AM

vantage255 wrote on Tue, 26 April 2005 01:18

Sorry about the sparsity of info. I had been up for about 30 hours... (bad night)

I am running vpopmail 5.4.7
Mysql 4.1.7
I set the server up usign the mail::toaster install but have modified it a good deal away from there.

Some versions of vpopmail had a problem similar to what you describe. That was some time ago and 5.4.7 should be just fine.

Why are you running mysql 4.1.x? All the benchmarks I've seen thus far indicate that 4.0 is roughly 20% faster. What "new" feature of 4.1 do you need? I'd expect you to have better luck with 4.0.

http://www.tnpi.biz/internet/mail/toaster/faq/programs/mysql.html" target="_blank"> http://www.tnpi.biz/internet/mail/toaster/faq/programs/mysql .html

Quote:

I am running SA out of .qmail files from indevidual users home directories and I am running ClamAV directly though the qmailqueue patch. A coworker has rewriten Qmail-ClamC and we have gained a good deal of speed here. (I have code if you are interested. It uses RAM now instead of writing to disk. It has proven stable over a few dozen high load boxes and is a LOT faster).

So, say you're having load issues and want to quickly disable SA processing. How do you do that when you call it from .qmail files? Don't get me wrong, I like calling it from .qmail files, and that's an option in Mail::Toaster but I much prefer simscan now as I can instantly disable spam or clam processing. This can be important at times:

http://www.tnpi.biz/internet/mail/toaster/faq/joe-job.html" target="_blank">http://www.tnpi.biz/internet/mail/toaster/faq/joe-job.html

Quote:

The MySQL server load is not terribly high. It is serving about 80 to 90 QPS durring busy times. Mytop is showing me about 300K Cache hits per second and 12 threads open with a key efficiancy of 99.6%

I guess this isnt truly a toaster question but more of a vpopmail question. This just seamed like the place to ask.

That's a fairly significant MySQL query load, and I'd be looking at ways to significantly reduce that.

Matt

vantage255 · May 02, 2005, 04:07:06 PM

Matt,
I have reduced the load, Even tried an off server DB box. MySQL doesnt seam to be having a problem with this. It is responding very quickly.
I can run a mysql password query againt accounts that are currently not able to auth. I get the same password that vpopmail is rejecting.it returns it in .000sec.

I am leaning twords a vpopmail issue here. I am going to upgrade to vpopmail 5.4.10 in hopes of the 3 or 4 updates made to vchkpw will solve the issue. None of the changes mentioned in the cahngelog seam to be related to any known issue such as this though.

Your thoughts?

matt · May 12, 2005, 09:46:47 PM

Have you found any resolution to this yet?

Matt

vantage255 · May 12, 2005, 10:30:51 PM

Nope.

I am looking into the some load issues on the server. "sometimes" it seams like the load is very high due to the autoresponder script that I am currently being forced to use. The issue doesnt ALWAYS coinside with a high load but more often then not it seams to lately.

It is such an infrequent issue, and I cannot recreate it with my own accounts on the box, that I am having a lot of trouble narrowing down the world situation.

dorianfm · August 11, 2005, 02:27:35 AM

I've been having a similar problem, but occuring with much more voracity. It was in a non-toaster based vpopmail installation, but using a very similar set up. The Mail server is very low capacity (12 accounts), but the system also has a quite high capacity website running from it, which also uses MySQL...

I was getting lots of 'MySQL has gone away' messages from vchkpw ...

In the end I resolved vmysql.c changing

unsigned int timeout = 2;

to

unsigned int timeout = 10;

And it resolved the issue. You might find that for you simply making it 4 resolves the issues (though how cleanly, I do not know)

vantage255 · August 11, 2005, 12:49:55 PM

I resolved it... Sort of. I will look into this mod, What I ended up doing was moving all 10 of my mail servers to using a dedicated SA server. this cut all of the SA requests out of the local MySQL DB. Cut the load on mysql in half. Stopped all auth issues.

Up to 15K domains per server and looking good.

Looking to centralize auth next.