Update Message and move it to an archive folder

[rogatec]

Hey @jstedfast , I know this question was somehow answered in #338, but it's very old and closed and I stumbled over this issue this week and I might miss here something.

Currently I implement a feature to move processed emails to an archive folder.

I open the INBOX folder and get all messages in it. After that I am storing relevant information in a dictionary, so I later have the relation between the MessageId and the UID.

After some other business logic I am trying to update and move the processed e-mails into an archive folder and additionally add a header information (as a "flag" to be sure this mail was processed).

Now I am currently struggling to update the message and afterwards move it - my current situation is, that the AppendAsync function does not only update the message, but additionally adds a "duplicate" - I've never seen this before

// the folder is 'INBOX'
await folder.OpenAsync(FolderAccess.ReadWrite, _token);
// Get all uids in the inbox folder
var uidList = await folder.SearchAsync(SearchQuery.All, _token);

foreach (var id in uidList)
{
    // get the MimeMessage
    var message = await folder.GetMessageAsync(id, _token);
    
    // ... some other stuff happening here
    
    // add the MessagId and the uid to a dictionary
    _mailDictionary.TryAdd(messageId, id);
}

// create the archive folder
var toplevelFolder = _mailClient.GetFolder(_mailClient.PersonalNamespaces[0]);
_archiveFolder = await toplevelFolder.CreateAsync("Archive", true, _token);

// ... lots of logic happens now in other classes / files etc.

// now assuming processing was successful I want to flag it and move it
message.Headers.Add("X-UNIQUE-FLAG", "processed"); 

// update the message with the new header element
// Here is the issue - it does not actually update the message, but creates a new Email element in the INBOX folder
var uid = await folder.AppendAsync(message, MessageFlags.Seen, _token);

// move message from inbox to archive folder
// Here it's moving the original one to the archive folder, but as the AppendSync above creates a duplicate, it stays in the INBOX
await folder.MoveToAsync(_mailDictionary[message.MessageId], _archiveFolder, _token);

In the other issue mentioned at top - you gave the hint, that I can update the message header and then append it directly to the new folder. Is this still the way to go? And to ensure that the Email is gone from the INBOX I afterwards need to ExpungeAsync it from the inbox?

I am using MailKit 4.3.0 with .NET 8

[rogatec]

Welll I found #985 - guess this answers my question - if not please reopen it with further tipps. Thanks anyways.

[jstedfast]

Hi @rogatec

Yes, you can download the message, add a custom header to it, and then Append the message to the Archive folder, but as your comment above the AppendAsync suggests, it does not "update" the existing message on the server. Instead, it appends a brand new message to the folder.

IMAP has a feature called "keywords" (which are just custom MessageFlags) that might be better for this kind of thing because it would require less work to implement (assuming the IMAP server allows custom keywords - which most do these days).

There are 2 ways to figure out if an IMAP server supports setting (persistent) custom keywords on a message:

1. Use a ProtocolLogger and look at the response to the SELECT command (which gets sent when you OpenAsync(FolderAccess.ReadWrite, ...)) and look for the PERMANENTFLAGS value to see if it contains \*. If it does, then it is supported. Otherwise, it isn't.
2. Check the PermanentFlags property of the folder to see if it has the MessageFlags.UserDefined bit set:

if (folder.PermamentFlags.HasFlag (MessageFlags.UserDefined)) {
    ...
}

Assuming that the PermanentFlags allows setting custom defined keywords, you can set a custom keyword like this:

// The newer API is ImapFolder.Store(...)
// Silent=true just means that we don't need the server to send us a notification that the message keywords changed
// as a result of this Store request.
var request = new StoreFlagsRequest (StoreAction.Add, MessageFlags.None) {
    Silent = true
};
request.Keywords.Add ("$MyProcessedKeyword");

await folder.StoreAsync (uid, request, _token);

See Also:

• https://mimekit.net/docs/html/Overload_MailKit_Net_Imap_ImapFolder_Store.htm
• https://mimekit.net/docs/html/Overload_MailKit_Net_Imap_ImapFolder_StoreAsync.htm
• https://mimekit.net/docs/html/T_MailKit_StoreFlagsRequest.htm

Another way using the older API (which in MailKit v4.x uses an Extension method) would look like this:

var keywords = new HashSet<string> (new [] { "$MyProcessedKeyword" });
await folder.AddFlagsAsync (uid, MessageFlags.None, keywords, true, _token);

See Also:

• https://mimekit.net/docs/html/Overload_MailKit_IMailFolderExtensions_AddFlags.htm
• https://mimekit.net/docs/html/Overload_MailKit_IMailFolderExtensions_AddFlagsAsync.htm

Custom keywords, by convention at least, typically start with a $. You should also consider naming this custom keyword such that it won't conflict with any other client or processing app that may also be running against the folder. So pick something that will be unique.

Hopefully that helps you decide how to proceed with your implementation.

[rogatec]

Wow, what a great answer, thanks for the hint with the keywords - didn't know that.
I'll refactor my code to use the new API.
Thank you very much!

[jstedfast]

I forgot to mention this before, but I would also advise against mapping Message-IDs to UniqueIds. Everyone assumes that Message-IDs are guaranteed to be unique, but there is no safeguard on that which means there is always a risk of collisions (and a malicious hacker could easily exploit the fact that they could craft a message that duplicates a Message-ID as well).

I would instead create a class that groups the MimeMessage that you download/process with the UniqueId that it has.

[rogatec]

Yeah you're right.
Maybe instead of creating a dictionary with key: MessageId and value: UniqueId I would just create a IList<ProcessMessage>, where the object definition looks like this:

public record ProcessMessage(MimeMessage message, UniqueId uid);

Sad thing is, that the MimeMessage object can be quite large (regarding when it contains large attachments)

[jstedfast]

Can I ask some questions that may help you to think about the problem you are trying to solve and maybe rethink its design a bit?

1. Do you really need to process the full MimeMessage? Can you get away with less?
2. Is it necessary to download a batch of messages to process as opposed to download 1, process 1, repeat?
3. If you do need the full MimeMessage, could you write it to disk and only include a FileName and UniqueId in your ProcessMessage record? If you can do this, I would recommend GetStream/Async() instead of GetMessage() - that will save you a bit of unnecessary processing time.

[rogatec]

Thanks for asking:

The purpose of the app I am maintaing is to get emails from mailboxes where invoices are send to (95%+ with PDF attachments, <5% zip archves).
The original mail with those attachments need to be processed in a ecm system and later archived (regulatory provisions - so the complete message and it's attachments need to be in the destination system).

Instead of processing mail by mail I implemented an eventbus and queueing system (with Masstransit and RabbitMQ) because the ecm system has a bottleneck (only one connection allowed via gRPC, no parallel exeuction) - therefore I download all messages, validate, prepare and process them and send it into the queue and then wait until they are in the destination system.

Afterwards I have a redundant implementation to ensure that I do not process them again (because we're talking about invoices, nobody wants to pay twice) - this redundance is first to mark the mail as processed - currently with my added header implementation, secondly I store them in a database (but here is a problem, because I asume the MessageId is unqiue,, an as you mentioned, it is not).

Now with the new archiving implementation I also want to cleanup the inbox, so after the process the inbox is empty and I don't have to check hundreds/thousand of emails again if they are already processed (check the header, check the database entry).

So in short I need several informations from the email, like subject, body, mail date, size, recipient, sender, copy cats. But the gRPC API wants the complete message and separated it's attachments each as a binary stream to process it correctly, I need to store them in specific properties and additionally I need to fill other properties with the email details (subject, body etc.).

---

I also thought about storying the complete message on the file system - but then I also need to implement additional checks (thinking about the disk is full etc.) - and besides on premise usage on windows machines the app now also runs in a k8s environment where more hardware means more 💵
But yes - this would definitly improve the memory usage

---

The performance (speed) of the app, getting the messages and process them is in my opinion really good - I did a test with around 1500 emails from a test server and they were all in the queue in no time.
But indeed the memory usage is a problem, because of the transformation of the message and it's attachments to a byte array (at least just before I call the ecm-gRPC)