IMAP "semi-asynchronous" backwards search in batches

[lennartack]

IMAP searches can take a very long time if you have a large inbox. In my inbox it is common that the HTTP timeout occurs before the search has finished. This feature is intended to resolve this problem and would greatly enhance user experience, in my opinion.

If you turn on "backwards" search, the IMAP search command will use an RFC 3501 message sequence range to search the n most recent messages in the folder. Once the web client receives a reply, it will send a search request for the next n emails in the folder. I have hardcoded n to 500 for body and full text searches and 5000 for other searches.

Open issues:

1. Everything is sorted DESC according to the internal IMAP server arrival date and the user cannot change this. This should still be made clear in the UI.
2. There is no implementation for multi folder search yet. Perhaps this should be disabled and made clear in the UI.
3. Backwards search with threads listing is still buggy. The threads implementation looks complicated and I have run out of time to study it.
4. The feature is only available with the Elastic skin.
5. The length n of the sequence range is set client-side, which is a bit awkward. It should probably be a server-side configuration parameter instead.
6. Maybe it should also be implemented for contacts search.

[alecpl]

I didn't review yet, but I think I'd prefer this to be implemented as a plugin. This probably makes most sense on a slow imap server. There are setups that have no problem in searching even very big folders. The open issues make it not ready and questionable.

[lennartack]

I didn't review yet, but I think I'd prefer this to be implemented as a plugin. This probably makes most sense on a slow imap server. There are setups that have no problem in searching even very big folders. The open issues make it not ready and questionable.

With a plugin implementation, would it be possible to run the backwards search with the normal search form in the UI? That is all that matters to me.

If you provide some feedback on the code and the issues I will continue this project once I have time.

[bsdice]

IMAP searches can take a very long time if you have a large inbox

I just stumbled on this ticket. Not sure if you use dovecot or something else, if dovecot, try the dovecot-fts-xapian fulltext-search plugin. On Arch, e.g. from https://archlinux.org/packages/community/x86_64/dovecot-fts-xapian/ with configuration similar to grosjo/fts-xapian@1e6d3b2

Anecdotally I have around 160.000 messages in IMAP Archive/ folders going back to 1995, in total ~24 GB for the Maildir. Using this plugin for dovecot I could swear fulltext search will return results faster than a Google search. I also use attachment decoders to make PDF, XLS etc. searchable. Previously I relied on Thunderbird, and Outlook before that, to find stuff in my Inbox, but this solution provides superior search results faster compared to those two fat clients. The server is usually lightly loaded and not even particularly beefy, a Xeon E3 1275 v6 with 64 GB ECC RAM from 5 years ago and an Intel P3520 PCIe SSD the same age.