Mail Toaster, Qmail, and OpenSSL 0.9.8e Workaround and Fix

[D3s7]

This is an FYI for all who might run into a problem with some mail bouncing.. I had this problem myself and the posted link fixed it:

There is apparently a bug in qmail when it comes to OpenSSL 0.9.8e that effected me at least.

here is the link and the doc posted under it:

http://www.thegillis.net/content/view/41/31/

Mail Toaster, Qmail, and OpenSSL 0.9.8e Workaround and Fix       PDF        Print        E-mail
Contributed by Brian Gillis   
Apr 07, 2007 at 09:03 AM

Recently, a friend of mine sent an e-mail to me that bounced. After checking his server logs, he was receiving the following error:

TLS_connect_failed:_error:14077410:SSL_routines:SSL23_GET_SERVER_HELLO:sslv3_alert_handshake_failure

It was a very interesting error to say the least. After some digging, I found that the new OpenSSL breaks mail. They claimed that this was due to an API change, but looking at the change list between 0.9.8d and 0.9.8e, no API change was noted. After some digging, I found that the problem is a programming error in the TLS patch for qmail. Read on for details.
PROBLEM

The error appears is in netqmail-1.05 on my server. Since the problem appeared on the submission port 587, and after checking the supervise run file, the binary file that handles SSL is /var/qmail/bin/qmail-smtpd. Checking the corresponding qmail-smtpd.c file, there is a function tls_init() on line 1134. This is where it appears to initialize the SSL library.

Attaching a debugger when the e-mail is being sent, shows us that it does not execute without errors. At line 1196, there is an error detected, and there is a call to ssl_error_str(). This error function came back with the following error:

error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher

It turns out this error is from several calls back where the error code is not checked. The actual function call where this error occurs is the call to SSL_set_cipher_list(...). One of the parameters of this function is the list of ciphers that the server accepts.

Tracing back some more, we see that the parameter "ciphers" is built from one of several locations. First it can come from the environment variable "TLSCIPHERS" on line 1174. If the environment variable is not set, the file "control/tlsserverciphers" is checked for any specific ciphers to use. If that file doesn't exist, the cipher should be set to "DEFAULT". The problem must be occurring here.

On my server, the environment variable is not set, and the control file doesn't exist; however, when I looked at the value passed into the function, the value was actually the hostname of my server. Since this is obviously not a valid cipher encryption, the error is expected.
SOLUTION

After tracing through some more, my hostname was actually being copied as the cypher by the function control_readfile(...) on line 1176 where the custom cipher list can be specified. Looking at this function on line 90 of file control.c I noticed the first problem. Although there are 2 arguments passed into the function, there are actually 3 function parameters. Looking further into the logic of this function, it appears as though the function copies the contents of the specified file, parameter 2, into the string, parameter 1, that is passed in. The third parameter is only referenced when the file doesn't exist. When the very descriptive third parameter "flagme" is != 0, the hostname appears to be copied into the return value. When "flagme" is 0, the return value is an empty string.

Since there is no third parameter, and variables aren't initialized to 0, "flagme" will almost always be non-zero, making the returned string the hostname. The previous OpenSSL version probably ignored incorrect values and continued while the new version doesn't.

The workaround is easy. Add the file "control/tlsserverciphers" in the qmail folder. The contents should be a single line with the word DEFAULT. You can optionally specify DES-CBC3-SHA instead of DEFAULT which is a TLS cipher. Either will work. A copy of this file is found below.

The fix is also simple. In qmail-smtpd.c on line 1176, a third parameter of 0 should be added to the control_readfile(...) function. The line used to look like:

if (control_readfile(&saciphers, "control/tlsserverciphers") == -1)

but now it should read:

if (control_readfile(&saciphers, "control/tlsserverciphers", 0) == -1)

Files:

    * netqmail-fix.patch - A patch for netqmail-1.05 that fixes the OpenSSL 0.9.8e compatability issue. To use, after installing netqmail using the "toaster_setup.pl -s qmail" command, cd to the netqmail work folder. By default this is "/usr/local/src/mail/netqmail-1.05/netqmail-1.05". Then run the command "patch -p1 < /path/to/patch/netqmail-fix.patch". Then build using "make clean && make qmail-smtpd". Make a backup of the old binary using "cp -p /var/qmail/bin/qmail-smtpd /var/qmail/bin/qmail-smtpd.bak". Copy over the new file "cp qmail-smtpd /var/qmail/bin/qmail-smtpd". Verify the new file has the same permissions as the old file.
    * tlsserverciphers - A configuration file to put in the (qmail_home)/control folder. This is a workaround for the OpenSSL 0.9.8e compatability issue.

[jostreff]

For me it doesn't work
I have configured all prerequests make /var/qmail/control/ and patched my qmail-smtpd.c file and my supervise startup file looks like
cat /var/service/smtps/run
#!/bin/sh

#    NOTICE: This file is generated automatically by toaster-watcher.pl. Do NOT hand
#      edit this file. Edit toaster-watcher.conf instead and then run toaster-watcher.pl
#      to make your settings active. See perldoc toaster-watcher.conf

PATH=/var/qmail/bin:/usr/local/bin:/usr/bin:/bin
export PATH

SMTPSPORT=465
export SMTPSPORT
TLS_CERTFILE="/var/qmail/control/servercert.pem"
export TLS_CERTFILE
TLS_PROTOCOL=SSL3
export TLS_PROTOCOL
TLS_STARTTLS_PROTOCOL=TLS1
export TLS_STARTTLS_PROTOCOL
TLS_VERIFYPEER=NONE
export TLS_VERIFYPEER
TLSCIPHERS=DEFAULT
export TLSCIPHERS
#QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl"
#export QMAILQUEUE

if [ ! -f /var/qmail/control/rcpthosts ]; then
   echo "No /var/qmail/control/rcpthosts!"
   echo "Refusing to start SMTP listener because it'll create an open relay"
   exit 1
fi

exec /usr/local/bin/softlimit -m 15360000 /usr/local/bin/tcpserver -S -H -R -c20 -x /usr/local/vpopmail/etc/tcp.smtp.cdb -u 89 -g 89 0 smtps /usr/local/bin/couriertls -server -tcpd qmail-smtpd /usr/local/vpopmail/bin/vchkpw /usr/bin/true 2>&1

[jostreff]

Does anybody tried with netqmail-1.0.6 ? Any implementation in mailtoaster?