Suggestions for IMAP session handling for mass/bulk-processing incoming mails? (IMAP IdleClient)

[rklec]

Using the IdleClient as a start I was wondering what the best strategy for maintaining a connection would be.

My requirements are basically:

• mass email processing
• processing any incoming mail as soon as it has been received
• I then may want to do different actions, like categorizing (aka sorting them in different folders) etc.

As such IdleClient and IMAP seem to be a good starting point.
The basic problem of SMTP IDLE seems to be you cannot do any other commands (other thane to end the idle state.

Aka/Thus:

You cannot query the ImapClient or an ImapFolder from within event callbacks.

#749 (comment)

So, to solve this, when a message is received the IdleClient just stops being idle and re-uses the connection to, in this case, download a message summary (aka the subject in this case) and display them.

But what, if I now want to process the whole message. I guess:

• downloading and processing a whole message may take some time
• thus, if the connection is re-used, the client would not be informed about new messages, as it is not in the IDLE state, would it?
• thus, this could cause messages not being processed/skipped
• Also, the client needs to be resilient against re-processing messages in case of failure/abrupt shutdown etc. while processing a message?

In #749 about connection issues I already cited, you delivered different solution ideas, but in the end, said:

Most mail client software that supports IDLE uses multiple connections.

They'll use 1 connection for IDLE and another connection to actually fetch the messages and such.

#749 (comment)

Thus, is this the recommend solution? Open a new IMAP connection for each mail (which would likely also allow simple, parallelization, given that each mail is only dependent on itself and can be processed while other emails are still being checked for).

Also ImapClient currently just uses the Index/ID of the messages to check which message, has been processed (in this case just printed to STDOUT) already. However, [instead of the MessageId it seems to be better to save the UniqueIds of the messages, apparently, in a production environment.
I feel like this may be noted in the example, that in production, you may want to do it a little differently there.

So is this approach correct or does it miss anything important?

[jstedfast]

But what, if I now want to process the whole message. I guess:
downloading and processing a whole message may take some time

Depends on what kind of processing you are doing, but yea.

thus, if the connection is re-used, the client would not be informed about new messages, as it is not in the IDLE state, would it?

Not while you are processing the message, but you will be notified once your code re-engages with the ImapClient by sending pretty much any other command to the IMAP server.

thus, this could cause messages not being processed/skipped

That's not true. Once you finish processing your message, presumably you'd call client.Idle/Async() (or client.NoOp()) again and then you'd be notified of any new messages since the last notification you got.

Also, the client needs to be resilient against re-processing messages in case of failure/abrupt shutdown etc. while processing a message?

The universal problem of all software systems that have to synchronize data between 2 (or more) locations.

Thus, is this the recommend solution? Open a new IMAP connection for each mail (which would likely also allow simple, parallelization, given that each mail is only dependent on itself and can be processed while other emails are still being checked for).

I'm not sure I'd recommend that approach. You could need way too many connections to be reasonable. At most I'd probably have 2 connections, 1 for idle and 1 for processing messages.

Also ImapClient currently just uses the Index/ID of the messages to check which message, has been processed (in this case just printed to STDOUT) already.

The approach in the IdleClient example is using indexes/counts as a means of determining where to start getting information about new messages that have arrived (which are the only messages the IdleClient sample doesn't know about yet because it started off doing a complete FETCH of the Inbox before it started the IDLE loop).

In your scenario, how you handle this will depend on what kind of state you need to keep about the messages.

The IdleClient sample was written from the perspective of an email client where the client application needs to have a complete mirror of what exists on the IMAP server but your scenario sounds like more of an automated message-processing pipeline.

If your processing app saves the last processed message UID such that if it crashes and respawns, it can load that UID back again, it could probably do something like this:

// If we know the UID of the last message that we processed, then the next message to process will be +1
var nextUid = new UniqueId (lastProcessedUid.Id + 1);

// Create a range of potential messages to process
var range = new UniqueIdRange (nextUid, UniqueId.MaxValue);

// See which messages actually exist
var uids = client.Inbox.Search (range, SearchQuery.All);