SASL Authentication with Sendmail 8.12.9, Redhat 9, BerkelyDB 4 and Cyrus-SASL v2

Last update: 27 May 2003

“I hate spam. I hate open relays for spam. I don’t want my mail server to be a relay, but I want my users to be able to use my mail server as a relay and their not all on my network.”

The above statement pretty much sums up what I was trying to achieve. Obviously you may well have different objectives and requirements for your server set-up but this might help you get started. It will show you how I configured my system to use Cyrus SASL v2, BerkeleyDB 4, Redhat 9 and Sendmail 8.12.9 to require users to authenticate themselves before being allowed to relay mail through my server. It is in no-way comprehensive or authoritative guide on setting up such systems but simply how I did it. For more detailed help with sendmail, visit the Official Sendmail Website or the comp.mail.sendmail newsgroup.

For more detailed information on SMTP AUTH with sendmail, check out Claus Außmann’s page on SMTP AUTH in sendmail 8.10-8.12. You may also want to read Jon Fullmer’s page on SMTP AUTH using SASL 1 and Berkeley DB3 which inspired me to write this guide.

What’s the plan?

The set-up described in this document enabled me to do the following:

  • Sendmail 8.12.9 as my system MTA.
  • Server is a closed relay and will only accept mail for hosted domains and users.
  • Users are able to use the mail server as an outgoing relay from any ISP or network they are on once authenticated.
  • System uses the users default password as authentication reading it from /etc/shadow.

What software did you use?

I tried to use the most recent versions I could find (as at time of writing). Software used was:

A quick word on Redhat 9 and Linux Threads…

As I was setting up my server from scratch to replace an old and very much out of date server, I stuck with Redhat Linux as I’ve used it for years and I’m comfortable with it and know my way around. Without reading up, I burned a set of ISO’s and installed but totally missed the news of the new NTPL Linux threads implementation. My ignorance of this caused me a unbelievable amount of frustration trying to compile Sendmail with the compiler generating a lot of errors I didn’t understand. I’m not a C++ programmer by any stretch and it took a lot of research and questions to solve the problems, therefore I have highlighted the relevant parts which related directly to Linux threads during compilation time. I don’t know how this effects other *nix variations or non-Redhat distro’s however so your on your own with solving any problems you may have.

Step 1 – Cleaning up

Although I deselected the software I didn’t want during the RedHat install, various bits and pieces crept it afterwards. I used RPM to um-install these and to make sure there was nothing that could cause conflicts, namely older versions of Sendmail, BerkelyDB and Cyrus-SASL.

Step 2 – Installing Berkeley DB 4.1.25

I’m still not 100% sure if I actually need BerkeleyDB specifically for the way I have configured my system but, as its possible to use it to create a separate authentication database for Sendmail AND as other applications I will install may use SASL and BerkeleyDB I decided to install it anyway.

I downloaded the latest tar.gz file from Sleepycat (see links above) and uncompressed it into /usr/local/src/ thus giving me the source in /usr/local/src/db-4.1.25.

At this point I cant re-enforce how important and useful it is to READ THE DOCUMENTATION. You’ll be suprised how much you might miss if you dont. Documentation for BerkeleyDB is located in /usr/local/src/db-4.1.25/docs/. I suggest you read chapter 22 on using it with Sendmail, chapter 26 on using it with Unix systems and part 8 of chapter 26 which mentions about building under Linux with threaded application support.

To build and install BerkeleyDB I did the following from the /usr/local/src/db-4.1.25/build_unix/ directory:

../dist/configure --prefix=/usr --enable-compat185 --enable-posixmutexes
make install

A note on the options in the configure command:

  • --prefix=/usr installs everything in to /usr/lib/ and /usr/include/ respectivley. As I had removed all previous versions of BerkeleyDB, I figured I wanted it in the standard library and include paths as I’d had problems in the past with compilers not finding libraries. If you dont set a prefix, everything ends up in /usr/local/BerkelyDB-4.1/ and you will need to tell the compiler where to find them when compiling other applications.
  • --enable-posixmutexes is to force Berkeley DB to use the POSIX pthread mutex interfaces. Normally which POSIX mutexes to use will be selected automatically on most systems but as I was using Redhat 9 with NTPL I put it in as a precaution. I also checked Makefile to ensure that the line LIBS contained the option -lpthread and that CPPFLAGS contained -D_REENTRANT.
  • --enable-compat185 I added “just-in-case” as it enables DB 1.85 API compatibility code into the library.

Step 3 – Installing Cyrus-SASL 2.1.13

I downloaded the latest tar.gz file from Carnegie Mellon (see links above) and uncompressed it into /usr/local/src/ thus giving me the source in /usr/local/src/cyrus-sasl-2.1.13.

Again, reading the docs located in /usr/local/src/cyrus-sasl-2.1.13/doc/ on installation I configured and built Cyrus-SASL with the followin commands:

./configure --prefix=/usr --enable-login --disable-krb4 --disable-gssapi --enable-pwcheck --with-dblib=berkely
make install

A note on the options in the configure command:

  • --prefix=/usr installs everything in to /usr/lib/sasl2 which as I mentioned before is my preferred location.
  • --enable-login is *VERY* important if your users are going to be using Entourage, Microsoft Outlook or Outlook Express. To quote from the SASL documentation:
    The LOGIN mechanism is a non-standard, undocumented plaintext mechanism. It’s included in the SASL distribution purely for sites that need it to interoperate with old clients; we don’t support it. Don’t enable it unless you know you need it.
    If you dont enable login as an authentication method, users of these mails clients wont be able to authenticate. SASL v2 also supports NTLM (MS Outlook, Exchange) but again, this is unsupported and needs to be enabled. If you need this, read the documentation for more details.
  • --disable-krb4 --disable-gssapi was used as I’m not a kerberos site.
  • --enable-pwcheck as I want to use the pwcheck function.
  • --with-dblib=berkely as I want to use BerkeleyDB.

Step 4 – Installing Sendmail 8.12.9

Sendmail caused me the biggest headache of all during the set-up process, mainly because of issues of it not compiling with the linux threads library properly.

I downloaded the latest tar.gz file from (see links above) and uncompressed it into /usr/local/src/ thus giving me the source in /usr/local/src/sendmail-8.12.9/.

The first thing you need to do is set-up your site config file, which you can create using a text editor such as vi, emacs or pico. The file needs to be called site.config.m4 and exist in /usr/local/src/sendmail-8.12.9/devtools/Site/.

My site.config file looks as follows:

APPENDDEF(`conf_sendmail_ENVDEF', `-DSASL=2')
APPENDDEF(`conf_sendmail_LIBS', `-lsasl2')
APPENDDEF(`conf_sendmail_LIBS', `-ldb-4')
APPENDDEF(`conf_sendmail_LIBS', `-lpthread')
APPENDDEF(`confLIBDIRS', `-L/usr/lib/sasl2')
APPENDDEF(`confINCDIRS', `-I/usr/include/sasl')
APPENDDEF(`confLIBS', `-lpthread')

These options are very important to getting sendmail to compile properly and pay special attention to the direction of the apostrophes!

The -DSASL=2 and -lsasl2 options tells Sendmail we are using Cyrus SASL v2. confLIBDIRS and confINCDIRS tells the compiler where to find the Cyrus SASL libraries. -ldb-4 tells sendmail that we are using BerkeleyDB 4.

The two lines with -lpthread are what caused me the biggest headache and took me a while to find out. Without these, on Redhat 9 you may find that either Sendmail or ALL of the Sendmail utilities fail to compile with an error message such as:

err.o(.text+0x123f): In function `sm_errstring':
: `sys_errlist' is deprecated; use `strerror' or `strerror_r' instead err.o(.text+0x1236): In function `sm_errstring':
: `sys_nerr' is deprecated; use `strerror' or `strerror_r' instead
/usr/lib/ undefined reference to `pthread_condattr_setpshared'
/usr/lib/ undefined reference to `pthread_mutexattr_destroy'
/usr/lib/ undefined reference to `pthread_mutexattr_setpshared'
/usr/lib/ undefined reference to `pthread_mutexattr_init'
/usr/lib/ undefined reference to `pthread_mutex_trylock'
collect2: ld returned 1 exit status
make: *** [sendmail] Error 1

Without telling Sendmail to use the Linux threads library, the compiler fails to include them which is what was causing the compiler to crash. These two lines may not be needed on other Linux distro’s.

Now, with the site.config file ready, I moved to the /usr/local/src/sendmail-8.12.9/ and build and install Sendmail with:

sh Build -c
sh Build install

The next step is to build your file which will vary depending on what options you want. Information on these options can be found here. I moved to the /usr/local/src/sendmail-8.12.9/cf/cf directory and created a file called You will probably want more options than this, but to get Sendmail working with SASL and PLAIN plus LOGIN authentication you need at least the following lines:

define(`confAUTH_OPTIONS', `A')dnl

I then compiled the file into with the following command:

m4 ../m4/cf.m4 >

Once that was done I moved the file to /etc/mail.

Step 5 – Setting up SASL for Sendmail and the authentication daemon.

Unlike the previous version, SASL v2 now has an authentication daemon which you use as an interface between Sendmail and the /etc/shadow file. More information on this can be found in the Cyrus SASL documentation under upgrading (of all places!).

The first stage in setting this up is to create a application config in you SASL library directory. In my case I created a file called Sendmail.conf in /usr/lib/sasl2. My file contained the following:

pwcheck_method: saslauthd

Which tells Sendmail to use the authentication daemon for password checking.

You then need to start the SASL authentication daemon (this should be done at boot usually) telling it which method of authentication you wish to use. In my case, Im authenticating against the /etc/shadow file so I start the daemon with:

/usr/sbin/saslauthd -a shadow

The first time you run it, you may get an error message about a missing directory called /var/state/saslauthd. If this is the case just create that directory and make sure it is read/writeable by the user you intend to run saslauthd as.

If all went well you should now have saslauthd running in your process list!

Step 6 – Testing it all.

I confess, I cheated and used my mail client for this and didnt hack the hard way until my excitement that it was working died down a bit.

First up, start (or restart) Sendmail as a daemon and telnet to your server on port 25. Below is a sample dialog of my connection to my mail server. Lines prefixed with ‘>’ denote my input:

> telnet localhost 25 ESMTP Sendmail 8.12.9/8.12.9; Tue, 27 May 2003 20:33:57 +0200
> EHLO localhost Hello localhost.localdomain [], pleased to meet you
250-SIZE 4194304
250 HELP
334 VXNlcm5hbWU6
>334 UGFzc3dvcmQ6
235 2.0.0 OK Authenticated

As you can see, the server reports that it accepts PLAIN and LOGIN as authentication methods. The garbled lines starting with 334 is the Username: and Password: prompts. These are Base64 encoded and for testing you will need to encrypt your username and password this way. You can do this using this Base64 Encoder and Decoder.

If you get any errors, check /var/log/maillog and /var/log/messages as this is where Sendmail and saslauthd log to.


Well, thats it! I hope it worked for you or at least got you someway towards making your own implementation on your system. As I said its by no means comprehensive but covers the basics and what I did.

I welcome comments, additions, corrections or technical feedback however I am no expert and you are better asking for help from the members of comp.mail.sendmail if you encounter any problems.

I appreciate that this method is not secure, does not properly encrypt passwords but for what it does its fine for me. Please dont mail me with arguments about using SSL or trying to convince me that there are better MTA’s than Sendmail. I really don’t care. This solution works for me and helps close another potential open relay and is another step in the fight against spam.