What features does the Toaster support?
SMTP Mail Server (SMTP-AUTH, TLS, tarpitting, RBL)
http://www.tnpi.biz/internet/mail/toaster/patches/
Virtual Domain Hosting w/delegated administration
Virtual Domain Users
Mailing List (Ezmlm)
AutoResponder
Web Based E-Mail (multiple options)
Web Based Domain Administration
Mail Filtering (rblsmtpd, SpamAssassin, Qmail-Scanner)
SMTP roaming via SMTP-AUTH, POP-AUTH, & IMAP-AUTH
POP3, POP3-SSL
IMAP, IMAP-SSLWhat programs will be installed?
Qmail 1.03 : Secure, fast, and reliable MTA
MySQL (3,4) : Database engine
ucspi-tcp : Inetd replacement
daemontools : Tools for managing daemons
ezmlm-idx : Mailing list management
Apache (1,2) : Secure HTTP server
Apache mods : mod_php, mod_ssl, mod_perl
OpenLDAP : LDAP client
vpopmail : Virtual Domain module
qmailadmin : Vpopmail domain administration CGI
autorespond : Auto Responder & vacation
gdbm : Database routines
ispell : Spell checking for web mail
qmailadmin : Web Control Panel Interface
sqwebmail : Web based email
squirrelmail : Web based email
courier-IMAP : IMAP & POP3 servers
maildrop : procmail like mail deliver agent
spamassassin : spam filtering program
clamAV : Virus scanning software
qmailscanner : content scanner for qmail
maillogs : mail log processor
rrdutil : MRTG like graphs
Mail::Toaster: Perl modules for mail systems
MATT::Bundle : Perl modules for unix systemsWhat are the mail retrieval options?
pop3://mail.example.com
imap://mail.example.com
pop3s://mail.example.com
imaps://mail.example.com
http://mail.example.com/cgi-bin/sqwebmail
https://mail.example.com/cgi-bin/sqwebmail
http://mail.example.com/squirrelmail
https://mail.example.com/squirrelmailMail::Toaster::CGI showcases the toasters functionality. See it live at: http://mail.cadillac.net/
How do I manage mail options for a domain?
Point a web broswer at https://mail.example.com/cgi-bin/qmailadmin
Where do I find the maildomain script?
maildomain perl script - Front end to nearly everything you can do with the vpopmail programs and a bunch of handy fixup utilities for qmail systems. http://localhost.tnpi.biz/internet/mail/toaster/maildomain/maildomain.pl
UPDATE: This is now part of Mail::Toaster and has been renamed mailadmin.
How do I convert Multilog date stamps to human readable date stamps?
tai64nlocal < logfile > qmaillog.tmpHow do I find statistics about my mail queue:
1. qmail queue | more
2. qqtool -a list
3. rrdutil
4. isoqlog
5. Manually feed qmail-send output through matchup and pipe
the results through a processor (all one one line):
/usr/local/qmailanalog/bin/matchup <
/var/log/mail/send/current |
/usr/local/qmailanalog/bin/z*
where z* is one of the following:
zddist, zdeferrals, zfailures, zoverall, zrecipients,
zrhosts, zrxdelay, zsenders, zsuccesses, zsuidsHow do I enable the sqwebmail calendar?
echo "local" > /usr/local/share/sqwebmail/calendarmode
When will you add support for my platform?
The scripts are extensible in anticipation of using them on platforms other than FreeBSD where they were developed. My next "supported" OS will likely be Mac OS X and then probably Linux. Linux fans might want to check out Bill Shupp's toaster pages. Here's a couple URLs for you: http://www.shupp.org/toaster/ and http://qmailtoaster.clikka.com/.
How do I block connections from servers with no reverse DNS?
Contents of ~vpopmail/etc/tcp.smtp (fix wrapped lines):
127.:allow,RELAYCLIENT=""
209.218.8.2:allow
=:allow
:allow,RBLSMTPD="Blocked - Reverse DNS queries for
your IP fail. You cannot send me mail."
#:allow,RBLSMTPD="-Blocked - Reverse DNS queries for
your IP fail. You cannot send me mail."
line 1. Obvious, allows localhost to relay.
line 2. Allows traffic from the inter7 email list
line 3. Matches any mail message with reverse dns.line 4. Matches what's left (no reverse DNS). By setting the RBLSMTPD environment variable, we actually get to pass a message back to the mail server we're blocking telling them why. That gives them a chance to fix it before the messages bounces.
They'll get a message like this in their mail logs:
Mar 27 08:40:43 seattle qmail: 1048783243.397888 info msg 6469: bytes 258 from qp 13226 uid 0 Mar 27 08:40:43 seattle qmail: 1048783243.438981 starting delivery 533: msg 6469 to remote matt@simerson.net Mar 27 08:40:43 seattle qmail: 1048783243.979048 delivery 533: deferral: 207.89.154.94_does_not_like_recipient./ Remote_host_said:_451_Blocked_ Reverse_DNS_queries_for_your_IP_failed. You_cannot_send_me_mail. Giving_up_on_207.89.154.94./
line 5. If you want to be a little more aggressive about it, use the 5th line instead of the fourth. Notice the '-' character in there. That tells rblsmtpd to return a permanent error (ie, don't try again!)
Mar 27 08:42:40 seattle qmail: 1048783360.776812 info msg 6475: bytes 250 from qp 13464 uid 0 Mar 27 08:42:40 seattle qmail: 1048783360.805534 starting delivery 534: msg 6475 to remote matt@simerson.net Mar 27 08:42:41 seattle qmail: 1048783361.259737 delivery 534: failure: 207.89.154.94_does_not_like_recipient. Remote_host_said:_553_Blocked_-_Reverse_DNS_queries_for_your_IP_fail. You_cannot_send_me_mail. Giving_up_on_207.89.154.94./ Mar 27 08:42:41 seattle qmail: 1048783361.269637 bounce msg 6475 qp 13467 Mar 27 08:42:41 seattle qmail: 1048783361.270564 end msg 6475
Notice that in the second case, the message bounces immediately. It's your mail server, you have to decide what policy you think is best. Bouncing messages seems to get more attention, and gets it faster than deferring connections.
That's all there is to it. Of course, that assumes you are running rblsmtpd as part of your smtp invocation.
How do I use SpamAssassin & ClamAV?
See the Filtering section at http://www.tnpi.biz/internet/mail/toaster/filter.shtml
Whats the best way to view/expire messages from the queue?
Use qqtool: http://www.tnpi.biz/internet/mail/qqtool/
Why are IMAP connections getting reset?
See this post to the mailing list for a possible explanation: http://matt.simerson.net/cgi-bin/ezmlm.cgi?mss:2006:pnonldigcpgkfpehlcgo
How do I delete old messages in the Spam folders?
find /usr/local/vpopmail/domains/*/*/Maildir/.SPAM/cur -type f -mtime +30 -exec rm -f {} \;
deletes files older than 30d
no guarantee, that just came off from the top of my head :) haven't tested, but worth the try :D regards, Andryan.
UPDATE: Matt thinks is a partial solution, as it won't find files within numbered buckets which you'll find on any domains with a large number of users.
Where is the UCSPI-TCP patch?
http://www.tnpi.biz/internet/mail/toaster/patches/tcpserver-mysql.shtml
The patch adds MySQL query support to tcpserver. The patch will allow tcpserver to query a Mysql server for each new connection and set RELAYCLIENT based on the presence of the remote IP in the table. This is a very useful and popular patch.
How can I modify the Sqwebmail interface?
The html temples live in /usr/local/share/sqwebmail/html/
How can I modify the Qmailadmin interface?
The html templates live in /usr/local/share/qmailadmin/html/
What httpd.conf changes are needed?
See contrib/httpd.conf-2.0.diff in the Mail::Toaster distribution.
Can you explain the Anti-Spam options please?
There are several anti-spam mechanisms in place. The first is RBLs.
Layer 1: RBLs
RBL's that deny SMTP connections to your server have no concept of email addresses. They only know whether the remote address is a blacklisted or not. So, if someone is blacklisted, you only have two choices:
a) use the blacklist and deny connections b) use the blacklist and whitelist specific IP's c) don't use the blacklistI choose b. I strongly desire to block volumes of crap before my SMTP server has to deal with it. In the few cases where a remote server is misconfigured and gets blacklisted, I have no problems being one of many servers they can't send mail to until they get it fixed. This is a form of policing the Internet and making careless or just plain naughty mail server owners become good netizens.
If a RBL is unjustly blocking too much for your organizations policies, you'll want to disable it.
Layer #2: Patches to qmail-smtpd.
We block invalid users (based on qmail-chk-usr patch, and have the badmailfrom, badrcptto, and other patches which you'll need to read up on to understand how to use. However, there isn't (as far as I know) any whitelisting. Even if there was, I wouldn't consider it beneficial.
Layer #3: Qmail-scanner.
If you've chosen to use it. It can filter based on external programs (ClamAV, SpamAssassin, etc) as well as things like attachment extensions, keywords, etc. It's very powerful, but has substantial resource requirements so use it with care.
The aforementioned layers are at the system level and settings there apply to your entire mail server. If you must employ different policies for different users, then you must do one of the following:
a) run multiple SMTP servers (on unique IPs) with different policies b) choose a "lowest common denominator" policyThe final layer of protection is at the mailbox level. This is where the user gets to opt into using SpamAssassin, and routing suspected spam into a .Spam folder. If there is to be user selectable preferences, this is where they belong, and eventually I'll work in some Squirrelmail -> SpamAssassin integration so users can tweak their preferences.
There are other mechanisms available for fighting spam but none that are easily implemented or that don't have other nasty side affects.
Layer 4: Maildrop
Maildrop is much like procmail and is used for the same reasons. After the message has passed through all the checks above, we can do some very sophisticated processing of the message. For example, the default mailfilter I wrote and include with the toaster does the following:
1. process messages < 256k through SpamAssassin 2. Process any user specific rules 3. verify users mail quotas 4. delete any message with a spam score > 12 5. Deliver spam message to Spam folder 6. bounce message if Mailbox is over quota
You have some very powerful tools included with the Mail::Toaster that will let you deal with each message in any fashion that you'd like. The only limits are your knowledge of the available tools and you are encouraged to learn them.
Is portupgrade -a safe?
Simple answer: No.
Long answer. I never recommend portupgrade -a on a production server. There are a variety of reasons for this:
Broken Binaries
Read the portupgrade man page for details.
Overwrites custom binaries
The toaster setup installs some customized binaries such as the heavily modified qmail package. If you let portupgrade "update" qmail, then the special anti-spam features go away. Ouch.
Does not honor make options
You'll note that toaster-watcher.conf has lots of options for configuration how certain toaster components are installed. We'll take MySQL for example. You might want it installed with ssl or linuxthreads. The toaster setup script honors those settings, but after portupgrade "updates" your MySQL, those special features are gone.
Because of the aforementioned reasons, many folks end up breaking their mail systems after running portupgrade. The best advise is have a development server to test on first. If it doesn't break your dev server, then it'll probably work find on your production one. Be very careful updating any programs installed via toaster_setup.pl. If you do upgrade something and break your toaster, run through the setup again, and it should repair the damage.
Why does SpamAssassin take so long to check messages?
First, you need to understand what a "normal" processing time is. On my server, I do DCC, Pyzor, Razor2, and the rest of the "normal" checks. If I have RBL checking on, then processing a message takes a couple seconds. If I disable RBL checks, messages processing drops to less than a second. This is on a pretty old server (Dual PIII 650) with 1GB of RAM.
If your processing time is extraordinarily different, That could be one of several reasons:
DNS timeouts
By default, SpamAssassin does RBL lookups against several blacklists. Make sure your DNS works well. If you are doing RBL checks via SMTP, then you may want to disable the RBL lookups in SpamAssassin. You can do that by adding "skip_rbl_checks 1" to /usr/local/etc/mail/spamassassin/local.cf.
If a RBL that SA is trying to contact is down, it may also introduce an extended timeout. Disabling the RBL checks is a good way to diagnose this. I run all the RBL checks at the SMTP level so I can deny the messages and thus avoid having to process them. As such, it makes little sense to have SA also running RBL checks.
Firewall blocking
SpamAssassin can use DCC, Razor2, and Pyzor. Those modules require contacting network servers. If your firewall rules are blocking those connections, this will introduce a timeout that hangs SA for 5 seconds per check. So, if your message processing time takes 11 seconds and it should be taking about 1, then you are likely blocking a couple checks. I added the following rules to my IPFW firewall:
# Allow DCC (6277) & Pyzor (24441) ${fwcmd} add allow udp from ${oip} to any 6277,24441 ${fwcmd} add allow udp from any 6277,24441 to ${oip} 1024-65535
How do I set up per-user SpamAssassin preferences?
I did it as follows:
Create the MySQL database
http://eu.spamassassin.org/full/2.6x/dist/sql/README
I named my database "spamassassin" and the table "userpref" as shown in the SA docs. The table name isn't significant, just make sure you are consistent in how you configure SA and sasql's access to it.
Create a MySQL user and password for access to the spamassassin database. I did so with a grant such as: GRANT ALL PRIVILEGES ON spamassassin.* TO spamassassin@'locahost' IDENTIFIED BY 'secretword';
Install SASQL
http://www.squirrelmail.org/plugin_view.php?id=167
Install it as you would any other squirrelmail plugin. Edit sasql_conf.php and set the DSN.
Install WebUserPrefs
http://www.pipegrep.net/webuserprefs/
I did it as follows:
cd /usr/local/www/mail fetch http://... tar -xzf webuserprefs-0.5.tar.gz mv webuserprefs-0.5 webuserprefs cd webuserprefs vi config.phpEdit the prefs_source to "db", authorization to "squirrelmail" and set the database info. Point your browser at http://mail.example.com/webuserprefs/ and volia. The catch is that you must be logged in via squirrelmail in order to use it. The alternate solution is to use IMAP or POP3 but then you have to recompile PHP with IMAP support.
Install Pear-DB
You'll find it in /usr/ports/databases/pear-DB.
I had mod_php4 installed before and this silly port was convinced it needed the lang/php4 port installed. Since I had another need for the PHP cli, I honored it's insistence by deinstalling www/mod_php4.
Modify spamd's flags. The best way is to modify /etc/rc.conf and add the following lines:
spamd_enable="YES" spamd_flags="-a -d -v -q -x -r /var/run/spamd.pid"
What needs to be in /etc/rc.conf?
Here's the relevant portions of mine:
enable_quotas="YES"
check_quotas="YES"
clamav_clamd_enable="YES"
spamd_enable="YES"
spamd_flags="-a -d -v -q -x -r /var/run/spamd.pid"
ntpdate="YES"
xntpd_enable="YES"
xntpd_flags="-p /var/run/ntpd.pid"
sendmail_enable="NONE"
snmpd_enable="YES"Should I use RBLs?
Grab a cut of coffee, it's SMTP lesson time.
The type of statistics you'll see in your logs regarding SMTP blocks will reflect several things of interest (that are related to your question). The first, and most obvious is how you are using the RBLs.
The most important setting in that regard is rbl_enable_soft_failure in your toaster-watcher.conf file. If you have soft failure enabled, then when you deny a mail agent access, you are doing so with a temporary failure, which amounts to saying "try again later". The idea behind that is that if it's a legit mail server, it gives them a chance to fix their problem and then try sending again.
I'm not in that camp. My email addresses have been plastered on web pages for years and years, on mailing lists, usenet forums, etc. Thus, my address(es) are included in many of the spammers databases so I'm going to be a target regardless. I want to stop them dead so I set rbl_enable_soft_failure to 0, which chooses a 553 error "sorry go away, permanent error, you spammer pig" style. This tells the remote server, don't even bother trying again, it won't work.
Obviously with the latter arrangement, you'll see a lot fewer (than if using the former) RBL blocks. Many of the spammers don't honor the 553 error and try again anyhow but that's quite OK. The majority of mail servers (themselves victims of naughty people) will get the 553 and bounce the message, no longer trying to contact me. All is well. This is the very same reason why it's better to use a SMTP 553 error instead of simply packet filtering their IP. Sure, they're blackholed, but they'll get a SMTP unreachable error and just keep trying. I want the spam traffic destined to my server to go away, not add to the cost of my system (via connectivity). When your server has thousands of mailboxes on it, these policy decisions make very real and measurable differences.
Doing a RBL lookup on a message at SMTP invocation time is cheap. Extremely cheap. It's a simple DNS query so it's very very very lightweight. I may even have it cached from the last time the spammer tried. It costs almost nothing. If I can deny a spam message there, before having to pass it to qmail-scanner (medium cost), through the virus scanner (medium to expensive), bayesian filtering (low cost), spamassassin network tests: blacklists, DCC, pyzor, razor (low - medium), and then maildrop filtering, then you it's worth it to me.
So, reason #1: It costs me less to use RBLs.
The next reason for RBL usage is more political. The spammers revenue (that they charge advertisers) is based on message deliveries. They maintain databases of email accounts that they can deliver to. They have developed very sophisticated ways of determining if a message got delivered (well beyond "did the SMTP server accept it") and further, if it landed in a real persons mailbox. These include delivery notifications, embedded HTML (the web server logs then reflect success), and other clever mechanisms.
Very few Mail User Agents are very clever about dealing with spam messages. Unfortunately, the ones that are (like Mail.app) are not the most popular. A lions share of users use that icky software from Redmond which is a spammers dream come true. Even if I tag and bag the spam to a special folder, they end user may peruse through it, opening the message to verify and thus letting the spammer know the message was delivered successfully. Yay, spam 'em again!
So, if I can block the message entirely (not just from my mailbox, but from every landing on my server) then the spammer will never get back any indication of success. This increases the likelihood of being removed from the naughty miscreants list and thus reduces the likelihood of getting spammed from him again in the future.
Sounds great in theory right? So how does it play out in real life? I have quite a few "case studies" we'll call them, ranging from my own personal mail servers to mail systems with tens of thousands of mailboxes on them. In general, the spam to ham ratio of messages on the internet is rising, but in every case my mail servers (with RBLs), the number of RBL blocks I see is gradually decreasing. This suggests that either RBLs are becoming less effective or some other unknown factor(s) are at work.
So, let's look at my mail server as a case study. Over time my message volume on my server has continued to rise, so we should expect the level of spam messages as well as ham messages to be steadily increasing (or the number of RBL blocks to increase). The effectiveness of my spam filters has been fairly constant (>98%) for the past six months yet the number of spam messages being delivered on my mail server has steadily dropped. What accounts for the difference?
Reason #2: Keeping spam messages way from end users
About a year ago I was watching my mail server logs and noticed that quite a few of my mail users (of whom I know personally) were getting extraordinarily high levels of spam being delivered to them. I then decreased my spam tolerance by blocking mail servers without DNS (which required whitelisting a fair number of mail servers), increasing my RBL usage. Immediately after that the number of RBL hits spiked and then began a gradual decline ever since. This reduction of traffic is due almost entirely to the RBL usage as I tested the reverse DNS blocking at various times and ultimately removed it.
Approximately six months later I did another round of decreased spam tolerance and enabling the spam filtering for these users, all of whom are not very sophisticated mail users. They don't know or care about mail servers, filtering, etc. They just want their email to work. Overnight the levels of spam in their inboxes dropped to almost nothing (a condition I'm accustomed to living with). This, of course made them all very happy. It also gave me a way to track the effectiveness of the RBLs, as I had a very accurate indicator of their effectiveness. RBLs only caught about 50% of the spam, but that's still a remarkably good achievement for something that costs so little to employ.
So, RBLs aren't a "one stop shopping" spam solution, but they are a very effective tool.
Reason #3: Spammers hate RBLs
There is good reason why RBL operators are targets of DoS attacks. These days running a RBL means making sure you and your upstream provider are prepared to deal with massive DoS attacks, sometimes lasting over a week. These DoS attacks can cost many thousands of dollars in excess bandwidth fees, lost revenues, and related costs. This is the reason for several of the RBL operators decisions to step out of the game, including osirusoft which was one of, if not the most effective free RBL. If RBLs weren't effective, they wouldn't be targets.
Unfortunately, the spammers have millions of advertising dollars at stake. When a RBL is as effective as Osirusoft, it measurably effects their bottom line, and thus becomes a target. If I can hurt the spammers where it counts (in their wallet), then in my book, that's A Very Good Thing.
Matt
How do I get QmailScanner Statistics to work?
I did as follows:
cd /usr/local/www/data
mkdir qss; cd qss
download tar archive from:
http://sourceforge.net/projects/qss/
tar -xzf qss-2.0.2.tar.gz
vi config.php Change the start date to the earliest date shown in your log file (head /var/spool/qmailscan/quarantine.log). Change the logFile variable to point to "/var/spool/qmailscan/quarantine.log". Then open up the permissions a bit on the quarantine so qmail-scanner stats can read it:
chmod o+x /var/spool/qmailscan
chmod o+r /var/spool/qmailscan/quarantine.logQmail-Scanner changed the logging date format at version 1.20 but qmailscanner stats doesn't have a fix for it yet. If you log files with dates in this format: 23-09-2003 then it'll work great. If you are using a newer version of qmail-scanner, you'll notice that your log entries are not showing up. I have both (before/after upgrade) so I fixed qmailscanner stats it by editing the index.php as follows (The if..else block is my addition):
vi index.php
if ( eregi("(^[0-9]+)", $val[0]) )
{
# Calc the date timestamp
$date = explode("/",$val[0]);
$dateT = $date[0];
$date[0] = $date[1];
$date[1] = $dateT;
$date = strtotime(implode("/",$date));
}
else
{
$date = strtotime ($val[0]);
};Point your browser at http://mail.example.com/qss/ and voila! If you can't get it working, just look at mine and pretend: https://mail.cadillac.net/qss/. ;-)
2 POD Errors
The following errors were encountered while parsing the POD:
- Around line 261:
=back without =over
- Around line 269:
'=item' outside of any '=over'