tnefclean in Kolab mini-Howto

Fixing TNEF (winmail.dat) attachments in the Kolab (Postfix) mailserver

Author: Nik Trevallyn-Jones
Date: November 2007
Version: 1.0

The problem

Microsoft Outlook can use a proprietary "encapsulation" format (called TNEF) for email attachments. Outlook users are becoming more aware of the problems with this format (eg complete lack of interoperability with any other mail reader) and many have configured Outlook not to use TNEF, but there are still Outlook users out there who are sending TNEF attachments and are either completely surprised that recipients cannot read them, or simply have to use this format for other (usually Microsoft enterprise) processing.

This HowTo describes how to configure the postfix mail server in Kolab to convert TNEF attachments into MIME attachments before delivery of the email.

The fix

An inline filter is used to unpack the known (ie, reverse-engineered) TNEF content types, and convert them to MIME attachments, making them interoperable with all standard mail readers.

Summary

There are a number of programs available which can convert at least some of the types of TNEF content to standard MIME types:

tnefclean
This is a perl script which uses a binary (C-based) tnef decoder.
http://www.dread.net/~striker/tnefclean/
killtnef
This is a perl script which uses other perl modules to unpack the TNEF attachments
http://www.cpan.org/authors/id/H/HI/HIGHTOWE/killtnef-1.0.1.pl

Both of these operate as an inline filter, and/or as a mail delivery agent. None of them operate as a pass-through SMTP processor. However, Postfix has limited support for inline filters, and Kolab already makes use of this feature (see kolabfilter and kolabmailboxfilter).

Of these two, I am currently using tnefclean, although the killtnef code looks cleaner. I will test killtnef at some stage in the future. The primary advantage of killtnef is that it uses only Perl modules (as far as I can tell), whereas tnefclean uses a binary TNEF unpacker.

So this document details a couple of ways of integrating the tnef filter of your choice into the email processing of a Kolab Postfix mail server.

Multiple approaches

I am sure that there are other possible solutions in addition to the ones I've explored and detailed in this document. I am certainly not claiming that these are the best approaches. I am merely documenting what worked for me, and how I did it. Feel free to modify my approach to suit your own preferences or installation.
The two approaches documented here are:

inline filter as part of the mail delivery (ie final) step.
Advantage: simplicity;
disadvantage: less efficient, and difficult to get working
There are two primary downsides to this approach:
1. the tnef processing is performed as the last step, whereas it would be best performed as the first step. TNEF decoding is non-trivial processing, and usually involves multiple temporary files (typically at least one per attachment). Many virus scanners also unpack TNEF attachments to scan the contents, but leave the mail unmodified. So if TNEF conversion is done at the end, then on many installations, it will have been performed twice.
2. an additional process is spawned for each email, which is also a significant performance overhead.
SMTP processor in the postfix processing chain
Advantage: efficiency, and works reliably;
disadvantage: introduces an addititional daemon.
This approach suffers from neither of the disadvantages of the previous approach. As part of the SMTP processing chain, the tnef conversion can be performed first, meaning that virus scanners don't need to repeat the processing to scan attchments. In addition, the SMTP processor is a long-lived daemon, and so repeatedly spawning new processes is avoided.

SMTP processor

Due to its performance advantages, and the fact I could get it working reliably, I will detail this approach first. This is the arrangement I have deployed on my live Kolab installation.

Ingredients

You will need:

a tnef decoder
I use tnfclean
http://www.dread.net/~striker/tnefclean/
an SMTP proxy with filtering abilities
I use smtpprox-filter
This is a perl script I created by modifying smtpprox-contentfilter which itself is based on the smtpprox perl script.

The original site for smtpprox and published derivitives is here:
http://bent.latency.net/smtpprox/

And this page has a link to smtpprox-contentfilter, which is here:
http://bent.latency.net/smtpprox/smtpprox-contentfilter

My enhanced smtpprox-filter can be downloaded from here:
http://nik.homelinux.net/files/smtpprox-filter.tz

Note: that smtpprox-contentfilter and smtpprox-filter can also be configured to run the clamd virus scanner and spamassassin spam filter. However, the amavis daemon used by Kolab to run these two filters supports other sophisticated features, so I have chosen not to replacce amavis, but rather augment it. If you consider that you don't need the added features of amavis, then feel free to replace it completely with smtpprox-filter.
access to an admin account
I use root
A text editor
I use vi
A small amount of sticky-tape or blue-tack.

Steps

Login or su to your admin account. If you're using su, don't forget to use the '-' option: su -
install your tnef filter
The installation script for tnefclean installs the executables in /usr/local/bin. I actually manually copied them into a new directory which I called /kolab/extras.
You need to edit a few paths at the top of tnefclean.pl. In addition, you will either need to change its name to tnefclean, or make sure you invoke it as tnefclean.pl.
If you've moved the tnefclean executables to a new location, then you will definitely need to change these paths!
install your smtp proxy
I use smtpprox-filter which relies on a few perl modules:
1. MSDW::SMTP (this is part of smtpprox and is also included with smtpprox-filter)
2. Net::Telnet (this is on CPAN).

Note 1. To make your smtp proxy run tnefclean as a filter, invoke it with the -f (filter) option.
Configure your smtp proxy
You can run smtpproc-filter with command-line arguments, but it is generally more robust if you create a config file for it. The default location for the config file is: /etc/smtpprox-filter.conf. Change the code to locate is elsewhere.
A sample config file is included with smtpprox-filter. The listern and talk directives must match the configuration changes you make to postfix.

Note: Since you can use command-line options to override the values in the config file, set the values in the config file to what you will be using in production. Then for testing, you can start smtp-filter on different ports by using command-line the options --listen and --talk.
test the smtp proxy installation
run the smtp proxy using an unused port for listening (eg localhost:1001), and a running mailserver (eg localhost:25).

Eg for smtpprox-filter: smtpprox-filter --listen=127.0.0.1:1001 --talk=127.0.0.1:25
Obviously, if your email server is somewhere else, or if you want to inject directly into amavis, then choose a different --talk value.
Telnet into your smtp proxy (eg localhost 1001), and inject an email into it. Make sure the email is correctly delivered. Use something like the following:
```
# telnet localhost 1001
HELO my.domain
MAIL FROM: <me@my.domain>
RCPT TO: <me@my.domain>
DATA
From: <me@my.domain>
To: <me@mydomain>
Subject: test

test email
.
QUIT
telnet> close
```
If the above doesn't make sense to you, I recommend that you read up on the SMTP protocol. Of course, you can also just skip this step.
If you do complete this step, make sure that the email is correctly delivered, and view the headers of the delivered email, and make sure any TNEF processing headers are there.
smtpprox-filter inserts 2 headers:
```
X-TNEF-Fixed-By: smtpprox-filter (message unmodified)
X-Scanned-By: smtpprox-filter
```
Configure your smtp proxy daemon startup
I created a proper startup script in /etc/init.d. You may prefer to do it some other way. I would really like to investigate getting postfix to start and stop the daemon, if that is possible.
start the smtp proxy
On my system, this is: "service smtpprox-filter start"
now connect your smtp proxy into you postfix server filter-chain using the following steps
edit /kolab/etc/kolab/template/main.cf.template;
find the line which configures all mail to be sent to 127.0.0.1:10024 for scanning by amavis; copy this line; comment out the original; and change the copy to send to 127.0.0.1:10023
```
Eg:
# insert TNEF processing into the content filtering chain (port 10023)
#content_filter=smtp-amavis:[127.0.0.1]:10024
content_filter=smtp-amavis:[127.0.0.1]:10023
```
make identical corresponding changes to /kolab/etc/postfix/main.cf
restart the kolab postfix server
/kolab/etc/rc postfix restart
now send an email to yourself and make sure that it is delivered correctly.
View the headers of the received email, and ensure that all the TNEF processing headers are there. On smtpprox-filter, the following headers should be there:
```
X-TNEF-Fixed-By: smtpprox-filter (messafe unmodified)
X-Scanned-By: smtpprox-filter
```
I have found live testing of TNEF email a little harder, because forwarding a TNEF tainted email causes the winmail.dat file to be enclosed inside a MIME attachmemt, causing tnefclean to detect it, but not convert it.

inline filter in the delivery chain

I tried this approach first, and while it was easy to configure and tests indicated all was well, tnef mail somehow was not processed. Further testing by injecting emails and watching the postfix log seemed to indicate that an email could take one of two possible paths for delivery, and the path I had modified seemed only to be taken by emails I injected into the system, while other emails appeared to take a different path. However, since I had already decided to test the smtp proxy approach, I did not continue debugging this configuration.

Ingredients

You will need:

a tnef filter (as before)
I use tnefclean

a small wrapper script for your tneffilter
I used the following (which I saved as /kolab/extra/tnefpipe):

#!/bin/sh
#
#  simple script to filter mail through tnefclean to whatever command is found on the commandline
#

TNEFPATH=/kolab/extra

# debug
#echo "tnefpipe" >> /tmp/tnefpipe.log

# create the pipeline
$TNEFPATH/tnefclean.pl -f | $@

access to an admin account
I use root
A text editor
I use vi
A small amount of sticky-tape or blue-tack.

Steps

Preceed as follows:

Login or su to your admin account (as above).
install your tnef filter, eg tnefclean, (as above)
This is tneflcean we are talking about, not tnefpipe.
Edit /kolab/etc/kolab/templates/master.cf.template Locate the definition for the kolabmailboxfilter, and have it call your script instead of kolabmailboxfilter. Do this by copying the first line of the definition, commenting the original, and modifying the copy.
Eg:
```
#kolabmailboxfilter     unix  -       n       n       -       -       pipe user=kolab-n flags=n argv=/kolab/bin/php

kolabmailboxfilter     unix  -       n       n       -       -       pipe user=kolab-n flags=n argv=/kolab/extra/tnefpipe
    /kolab/bin/php
```
The first (commented) line is a single line, even if it has wrapped in this document, and/or in your editor
Note: The primary change is argv=/kolab/bin/php has been changed to argv=/kolab/extra/tnefpipe, and /kolab/bin/php is now the first line of the following command.
Note: The rest of the comand is left untouched. So the entire definition for the kolabmailboxfilter becomes:
```
#kolabmailboxfilter     unix  -       n       n       -       -       pipe user=kolab-n flags=n argv=/kolab/bin/php

kolabmailboxfilter     unix  -       n       n       -       -       pipe user=kolab-n flags=n argv=/kolab/extra/tnefpipe
    /kolab/bin/php
    -c /kolab/etc/apache/php.ini
    -f /kolab/etc/resmgr/kolabmailboxfilter.php
    --
    -h @@@fqdnhostname@@@
    -s ${sender}
    -r ${recipient}
    -c ${client_address}
```
The tnefpipe script simply pipes its input through tnefclean -f (running as a filter), and pipes the output into whatever command it finds on the command line (which is basically whatever kolabmailboxfilter was doing before).
Edit /kolab/etc/postfix/master.cf and make identical corresponding changes.
restart the postfix process
/kolab/etc/rc postfix restart
Test your handiwork by enabling (uncommenting) the debug line in tnefpipe and sending an email to yourself. Make sure the email is delivered correctly, and that the debug file shows that tnefpipe was being invoked.
The line to uncomment is:
```
#echo "tnefpipe" >> /tmp/tnefpipe.log
```
Disable the debug line once you are happy that everything is working.

As I stated at the beginning of this section, this worked and passed all my tests, but incoming emails with TNEF attachments weren't being converted, and the logs seemed to indicate they had not been delivered using kolabmailboxfilter but had instead been delivered by kolabfilter.
I haven't debugged this any further, but if you get the same results as I did, then feel free to either post on the kolab forums for help, or perhaps try routing kolabfilter through tnefpipe instead of, or possibly as well as kolabmailboxfilter.