Manitou-Mail Blog

Version 1.3.0 released

daniel — Thu, 04 Oct 2012 20:40:14 +0000

Version 1.3.0 is now available for download, with major improvements to the full text search and the ability to import mailbox archives from within the user interface. Here is what’s new in more detail:

Accent-insensitive search, word exclusion, better word parser, smaller inverted index.
User interface to submit and control server-side import of mailbox files.
Combine headers and body in the same view for better usability.
Add email syntax checks in composer.
Bugfixes: set forwarded status on forwarded messages.
Help system: use a simple integrated browser instead of the external Qt assistant program.
Integrated indexing of html parts.
Text extractors on attachments replace indexer plugins.
Add partial and full reindexing capabilities to manitou-mgr.

The user interface is available as an installer for MS-Windows and a disk image for Mac OS X. For Linux, precompiled packages are available for the latest Debian and Ubuntu distributions, for both x86 and amd64 architectures.
Users of other operating systems are encouraged to compile from source and open issues at github if problems arise.

To upgrade from previous versions, it’s strongly suggested to replace attachments indexers by words extractors (see the previous blog post: Upgrading attachments indexers). Once done, it’s also suggested to rebuild entirely the inverted index. See Upgrading manitou-mdx in the documentation.

Upgrading attachments indexers

daniel — Wed, 26 Sep 2012 13:36:26 +0000

Version 1.3.0 is ready to be released with some major improvements in the full-text search and full-text indexer. A couple of packaging issues are still being worked on, but the code won’t change significantly from the 1.3.0 tags on the master branches at github, for both manitou-mail-mdx and manitou-mail-ui.

The list of changes from the previous version is currently visible in the NEWS file in the sources.

It should be noted that 1.3.0 will require some adjustments in how attachments in various formats are indexed with the help of user-supplied scripts.

Up to now, indexer plugins were used to index the contents of attached files in PDF or DOC, or even HTML formats.

Starting from 1.3.0, this is no longer desirable: they should be integrated with a new method called “words extractors” and declared in manitou-mdx main configuration file. Example:

index_words_extractors = application/pdf: /opt/scripts/pdf2text \
	application/msword: /opt/scripts/word2text

The user-supplied scripts should extract words from the contents in custom format passed to their standard input, and output these words encoded in utf-8 to the standard output.

For installs that didn’t index attachments with plugins, it doesn’t matter. Upgrading to 1.3.0 will just cause starting to index HTML contents, which it does now internally by default, so no manual action is required.

On the other hand, for installs that used indexer plugins, a preliminary step to upgrade to 1.3.0 would be to convert these to word extractor scripts. It’s nothing particularly difficult. As an example, here is a ready-to-use script that extracts words from MS-Word files with antiword.

#!/bin/sh

t=$(tempfile --suffix=.doc) || exit 1
trap "rm -f -- '$t'" EXIT
cat >>$t
antiword -i1 "$t" || exit 1

rm -f -- "$t"
trap - EXIT
exit 0

This is a preliminary step because it will be recommended to rebuild the inverted word index when upgrading to 1.3.0 , and doing this involves reindexing attachments as well.
Up to version 1.2, that was not possible with indexer plugins. That’s one of the reasons why plugins get deprecated as a way to index attachments contents. In addition, the lack of integration with the words vectors cache was a performance drag, and 1.3 solves that as well with its words extractors method.

Exim4 and its pipe_transport unset error

daniel — Mon, 03 Sep 2012 13:16:59 +0000

On Debian systems, Exim4 in its default configuration does not allow piping of an incoming mail into a program defined in /etc/aliases. Instead of launching the program it will report this type of error in /var/log/exim4/mainlog:

system_aliases defer (-30): pipe_transport unset in system_aliases router

Yet that’s the method suggested in manitou-mdx documentation’s “Delivering incoming mail into files“ section, and it has the advantage of being quite standard across most MTAs. This is also a problem for popular mail software such as MailMan, as mentioned in this Ubuntu issue.

The solution is similar to the one mentioned in the above issue, except that it’s better to create a custom configuration file rather than editing exim4.conf.template, according to update-exim4.conf manpage.

In the simplest case where a split configuration for Exim is not used (dc_use_split_config=’false’), the fix is as simple as creating /etc/exim4/exim4.conf.localmacros containing:

SYSTEM_ALIASES_PIPE_TRANSPORT = address_pipe

SYSTEM_ALIASES_USER and SYSTEM_ALIASES_GROUP may be specified too if the defaults are not suitable, but only SYSTEM_ALIASES_PIPE_TRANSPORT is strictly necessary.

If a split configuration is used, the line should go into a file under /etc/exim4/conf.d, e.g. /etc/exim4/conf.d/main/000_localmacros

SQL functions for tags

daniel — Wed, 29 Feb 2012 19:56:08 +0000

Tags in Manitou-mail are hierarchical, for several reasons such as the ability to mimic folders. There are pros and cons of this choice, but from the point of view of SQL querying, tree-like structures are clearly more complicated than flat structures. Here are two functions in the wiki that could be of help to compare tags across hierarchies:

tag_path(tag_id) returns the full hierarchical path of a tag, with -> as the separator between branches.
tag_depth(tag_id) returns the depth of the tag inside its hierarchy, starting at 1

As an example of use, in the custom queries of the user interface, we could use this query:
select mail_id from mail_tags mt join tags t on (mt.tag=t.tag_id) where tag_path(t.tag_id) ilike 'ParentTag->%'
to retrieve any message tagged with any tag whose top-level ancestor is ‘ParentTag’, no matter how deep the tag is inside the hierarchy (child, grandchild, grand grandchild…)

Attachment uploader reloaded

daniel — Fri, 24 Feb 2012 18:43:06 +0000

The attachment uploader is a plugin that solves the problem of attachments that are too big to be transferred by mail. It requires no specific action from the person who composes the mail: on the client side, the user attaches the files as usual. This is important because the users are often not aware that sending huge files by e-mail is troublesome.

The upload happens on the server side: before passing outgoing messages to the delivery service, the plugin checks for attachments bigger than the configured size, and when found, transfers them to a web server, under a random-generated directory name. Inside the message, the contents are replaced by the URL pointing to them.

Until recently, the attachments had to be sent by FTP to the web server. What’s new is that it can now be SSH instead, provided that the Net::SFTP::Foreign Perl module is installed on the server.

The new version of the plugin comes with manitou-mdx 1.2.0 and is named attach_uploader_ssh.
(source code on github)

To activate the plugin and connect it to a sender’s identity, it needs to be declared in manitou-mdx configuration file.
Example:

[mymail@domain.tld]
outgoing_plugins = attach_uploader_ssh({host=>"www.myserver.tld", login=>
"sshuser", base_url=>"http://attached.myserver.tld", path=>"/var/www/attached.myserver.tld", maxsize=>2000000})

There is a “password” field that could be used but in this example it is assumed that an SSH key lets the script connect without a password. This is just one of the multiple security options and choices that open up when switching from FTP to SSH.

And here is how a picture larger that 2Mb would appear in the message (as a text/plain part)

The attached file is available at:

http://attached.myserver.tld/JXlafBybAKGrwE5yyOIbKA/PICT34.JPG

Moving the source code to github

daniel — Tue, 10 Jan 2012 13:31:07 +0000

Starting with v1.2.1 currently under development, subversion is replaced by the more modern git source control tool.

It’s also the opportunity to split the source code into two distinct repositories for the user interface and mdx, since they can be worked on independently. A third repository should follow for the documentation.

The master branches for both programs are accessible at:
https://github.com/manitou-mail/manitou-mail-ui and https://github.com/manitou-mail/manitou-mail-mdx

The subversion repository at SourceForge.Net is still accessible, but from now on it will probably see commits only on releases.

The github service also has an issue tracker, a wiki and other things that might get used over time, we’ll see.

Version 1.2.0 released

daniel — Sat, 31 Dec 2011 17:34:22 +0000

Version 1.2.0 is now available for download.
What’s new?

More functions are available in the filter expressions: age(), date(), date_utc(), now(), now_utc(), identity(), rawsize().
New equality and inequality operators for numeric comparisons.
A mdx plugin to modify headers of outgoing mail with dynamic contents.
A new incarnation of the attach_uploader plugin that supports ssh for file transfers.
Migration to new Qt assistant and compatibility fixes for Qt-4.7 and 4.8.
Drag and drop for attached files in the mail composer.
A new dynamic column for recipients in the messages list.
Many improvements in the filter editor: filters can now be reordered and tested from within the interface, and the user interface for actions has been redesigned.
New filter actions: Set identity, set header, remove header.

Quick resend functionality

daniel — Fri, 25 Nov 2011 16:51:23 +0000

Sometimes a message that has been previously sent needs to be sent again. The normal way to do that is to recompose a new message by copying the contents of the old one. This leads to a new message with identical contents except for the Date and Message-Id header fields.
However, there is a quicker way to re-send an outgoing message without the need to create a duplicate of the original: if the Sent and Archived bit of the message status are cleared, manitou-mdx will just pick up again the message for sending as if it was new. To clear these bits, use the Message->Properties command and check No in the boxes drawn with the red border in the screenshot:

Acting on all tagged messages except some

daniel — Sat, 29 Oct 2011 20:03:30 +0000

Recently I wanted to reduce the size of my main manitou-mail database, and thus I’ve decided to delete all the messages I’ve received from some mailing-lists. I know these messages are archived elsewhere, so that I could re-import them if needed anyway.
But I didn’t like the idea of deleting also the messages that I’ve sent to these mailing-lists, because it would have broken the rule I’ve adopted of keeping all sent messages. Also on second thought, I thought it would be best to include also the whole threads in which I’ve participated, so that the context of the messages would still be available (BTW, the entire thread to which a message belongs can be recalled in the user interface by the contextual menu command: “Show thread on new page”).

So the question was, how to select all the messages tagged with certain tags, but excluding all threads for which at least one message has the Sent status? As usual, the database and the SQL come to help. First I looked up the tag_id’s of the tags corresponding to the mailing-lists, let’s say they were 3,6 and 10. And then I just expressed in SQL the sentence above. The result is:

SELECT mt.mail_id FROM mail_tags mt JOIN mail m1 USING (mail_id)
 WHERE tag IN (3,6,10) AND NOT EXISTS
 (SELECT 1 FROM mail m2 WHERE m2.thread_id=m1.thread_id AND m2.status&128!=0)

After issuing this query with the selection dialog, with the Limit To field empty to ensure that all messages are retrieved, all that was needed to accomplish the task was to select all the messages in the resulting list (Ctrl-A) and hit the Del key.

Manitou-Mail for Mac OS X

daniel — Sat, 01 Oct 2011 22:24:30 +0000

For you Mac users out there, I’m glad to announce that the Manitou-Mail interface is now available for Mac OS X, thanks to Adebe Networks that funded the work for the port and TLK Games who lent their Mac machine. The application bundle is available here: manitou-1.1.0.dmg (42Mb, with Qt-4.7 and libpq5 compiled with ssl support). Enjoy!