Separate mime tree from large attachments

README.md

@@ -4,64 +4,76 @@
 
 This is a very early preview of an IMAP server built with Node.js, MongoDB and Redis. Node.js runs the application, MongoDB is used as the mail store and Redis is used for ephemeral actions like publish/subscribe or caching.
 
-### Goals of the Project
+## Goals of the Project
 
 1. Build a scalable IMAP server that uses clustered database instead of single machine file system as mail store
 2. Push notifications. Your application (eg. a webmail client) should be able to request changes (new and deleted messages, flag changes) to be pushed to client instead of using IMAP to fetch stuff from the server
 3. Provide Gmail-like features like pushing sent messages automatically to Sent Mail folder or notifying about messages moved to Junk folder so these could be marked as spam
 
-### Supported features
+## Supported features
 
 Wild Duck IMAP server supports the following IMAP standards:
 
-* The entire **IMAP4rev1** suite with some minor differences from the spec. Intentionally missing is the `\Recent` flag as it does not provide any real value, only makes things more complicated. RENAME works a bit differently than spec describes.
-* **IDLE** – notfies about new and deleted messages and also about flag updates
-* **CONDSTORE** and **ENABLE** – supports most of the spec, except metadata stuff which is ignored
-* **STARTTLS**
-* **NAMESPACE** – minimal support, just lists the single user namespace with hierarchy separator
-* **UNSELECT**
-* **UIDPLUS**
-* **SPECIAL-USE**
-* **ID**
-* **AUTHENTICATE PLAIN** and **SASL-IR**
+- The entire **IMAP4rev1** suite with some minor differences from the spec. Intentionally missing is the `\Recent` flag as it does not provide any real value, only makes things more complicated. RENAME works a bit differently than spec describes.
+- **IDLE** – notfies about new and deleted messages and also about flag updates
+- **CONDSTORE** and **ENABLE** – supports most of the spec, except metadata stuff which is ignored
+- **STARTTLS**
+- **NAMESPACE** – minimal support, just lists the single user namespace with hierarchy separator
+- **UNSELECT**
+- **UIDPLUS**
+- **SPECIAL-USE**
+- **ID**
+- **AUTHENTICATE PLAIN** and **SASL-IR**
 
-### FAQ
+## FAQ
 
-#### Does it work?
+### Does it work?
 
 Yes, it does. You can run the server and get a working IMAP server for mail store, LMTP and/or SMTP servers for pushing messages to the mail store and HTTP API server to create new users. All handled by Node.js and MongoDB, no additional dependencies needed.
 
-**Isn't it bad to use a database as a mail store?**
+### What are the killer features?
+
+1. Start as many instances as you want. You can start multiple Wild Duck instances in different machines and as long as they share the same MongoDB and Redis settings, users can connect to any instances. This is very different from more traditional IMAP servers where a single user always needs to connect (or proxied) to the same IMAP server. Wild Duck keeps all required state information in MongoDB, so it does not matter which IMAP instance you use.
+2. Super easy to tweak. The entire codebase is pure JavaScript, so there's nothing to compile or anything platform specific. If you need to tweak something then change the code, restart the app and you're ready to go. If it works on one machine then most probably it works in every other machine as well.
+3. Works on Windows. At least if you get MongoDB and [Redis](https://github.com/MSOpenTech/redis) running first.
+
+### Isn't it bad to use a database as a mail store?
 
 Yes, historically it has been considered a bad practice to store emails in a database. And for a good reason. The data model of relational databases like MySQL does not work well with tree like structures (email mime tree) or large blobs (email source).
 
 Notice the word "relational"? In fact documents stores like MongoDB work very well with emails. Document store is great for storing tree-like structures and while GridFS is not as good as "real" object storage, it is good enough for storing the raw parts of the message.
 
-#### Is the server scalable?
+### Is the server scalable?
 
 Not yet. These are some changes that need to be done:
 
-1. Separate attachments from indexed mime tree and store these to GridFS. Currently entire message is loaded whenever a FETCH or SEARCH call is made (unless body needs not to be touched, for example if only FLAGs are checked). This also means that the message size is currently limited. MongoDB database records are capped at 16MB and this should contain also the metadata for the message.
-2. Optimize SEARCH queries to use MongoDB queries. Currently only simple stuff (flag, internaldate, not flag, modseq) is included in query and more complex comparisons are handled by the application but this means that too much data must be loaded from database (unless it is a very simple query like "SEARCH UNSEEN" that is already optimized)
-3. Optimize FETCH queries to load only partial data for BODY subparts
-4. Parse incoming message into the mime tree as a stream. Currently the entire message is buffered in memory before being parsed.
-5. Add quota handling. Every time a user gets a new message added to storage, the quota counter should increase. If only a single quota root would be used per account then implementing rfc2087 should be fairly easy. What is not so easy is keeping count on copied and deleted messages (there's a great technique for this described in the [mail.ru blog](https://team.mail.ru/efficient-storage-how-we-went-down-from-50-pb-to-32-pb/)).
+1. Optimize SEARCH queries to use MongoDB queries. Currently only simple stuff (flag, internaldate, not flag, modseq) is included in query and more complex comparisons are handled by the application but this means that too much data must be loaded from database (unless it is a very simple query like "SEARCH UNSEEN" that is already optimized)
+2. Optimize FETCH queries to load only partial data for BODY subparts
+3. Parse incoming message into the mime tree as a stream. Currently the entire message is buffered in memory before being parsed.
+4. Add quota handling. Every time a user gets a new message added to storage, the quota counter should increase. If only a single quota root would be used per account then implementing rfc2087 should be fairly easy. What is not so easy is keeping count on copied and deleted messages (there's a great technique for this described in the [mail.ru blog](https://team.mail.ru/efficient-storage-how-we-went-down-from-50-pb-to-32-pb/)).
 
-#### What are the killer features?
+### How does it work?
 
-1. Start as many instances as you want. You can start multiple Wild Duck instances in different machines and as long as they share the same MongoDB and Redis settings, users can connect to any instances. This is very different from more traditional IMAP servers where a single user always needs to connect (or proxied) to the same IMAP server. Wild Duck keeps all required state information in MongoDB, so it does not matter which IMAP instance you use.
-2. Super easy to tweak. The entire codebase is pure JavaScript, so there's nothing to compile or anything platform specific. If you need to tweak something then change the code, restart the app and you're ready to go. If it works on one machine then most probably it works in every other machine as well.
+Wild Duck tries to keep minimal state for sessions to be able to distribute sessions between different hosts. Whenever a mailbox is opened the entire message list is loaded as an array of UID values. The first UID in the array element points to the message #1 in IMAP, second one points to message #2 etc.
+
+Actual update data (information about new and deleted messages, flag updates and such) is stored to a journal log and an update beacon is propagated through Redis pub/sub whenever something happens. If a session detects that there have been some changes in the current mailbox and it is possible to notify the user about it (eg. a NOOP call was made), journaled log is loaded from the database and applied to the UID array one action at a time. Once all journaled updates have applied then the result should match the latest state. If it is not possible to notify the user (eg a FETCH call was made), then journal log is not loaded and the user continues to see the old state.
 
 ## Usage
 
+Assuming you have MongoDB and Redis running somewhere.
+
 **Step 1.** Get the code from github
 
-    $ git clone git://github.com/wildduck-email/wildduck.git
-    $ cd wildduck
+```
+$ git clone git://github.com/wildduck-email/wildduck.git
+$ cd wildduck
+```
 
 **Step 2.** Install dependencies
 
-    $ npm install
+```
+$ npm install
+```
 
 **Step 3.** Modify [config file](./config/default.js)
 
@@ -69,7 +81,9 @@ Not yet. These are some changes that need to be done:
 
 **Step 5.** Run the server
 
-    npm start
+```
+npm start
+```
 
 **Step 6.** Create an user account (see [below](#create-user))
 
@@ -81,8 +95,8 @@ Users can be created with HTTP requests
 
 Arguments
 
-  * **username** is an email address of the user
-  * **password** is the password for the user
+- **username** is an email address of the user
+- **password** is the password for the user
 
 **Example**
 
@@ -113,7 +127,7 @@ Create an email account and use your IMAP client to connect to it. To send mail
 node examples/push-mail.js username@example.com
 ```
 
-This should "deliver" a new message to the INBOX of *username@example.com* by using the built-in SMTP maildrop interface. If your email client is connected then you should promptly see the new message.
+This should "deliver" a new message to the INBOX of _username@example.com_ by using the built-in SMTP maildrop interface. If your email client is connected then you should promptly see the new message.
 
 ## License
 

config/development.js

@@ -21,14 +21,14 @@ module.exports = {
         enabled: true,
         port: 3424,
         host: '0.0.0.0',
-        maxMB: 5
+        maxMB: 25
     },
 
     smtp: {
         enabled: true,
         port: 3525,
         host: '0.0.0.0',
-        maxMB: 5
+        maxMB: 25
     },
 
     api: {

examples/push-message.js

@@ -17,12 +17,17 @@ const transporter = nodemailer.createTransport({
 });
 
 transporter.sendMail({
+    envelope: {
+        from: 'andris@kreata.ee',
+        to: [recipient]
+    },
     from: 'andris@kreata.ee',
     to: recipient,
     subject: 'Test message [' + Date.now() + ']',
     text: 'Hello world! Current time is ' + new Date().toString(),
     html: '<p>Hello world! Current time is <em>' + new Date().toString() + '</em></p>',
     attachments: [{
-        path: __dirname + '/swan.jpg'
+        path: __dirname + '/swan.jpg',
+        filename: 'swän.jpg'
     }]
 });

imap-core/lib/indexer/indexer.js

@@ -9,10 +9,9 @@ const BodyStructure = require('./body-structure');
 const createEnvelope = require('./create-envelope');
 const parseMimeTree = require('./parse-mime-tree');
 const LengthLimiter = require('../length-limiter');
-// const ObjectID = require('mongodb').ObjectID;
+const ObjectID = require('mongodb').ObjectID;
 const GridFs = require('grid-fs');
-
-// TODO: store large attachments to GridStore
+const libmime = require('libmime');
 
 class Indexer {
 
@@ -21,7 +20,9 @@ class Indexer {
         this.fetchOptions = this.options.fetchOptions || {};
 
         this.database = this.options.database;
-        this.gridstore = new GridFs(this.database, 'attachments');
+        if (this.database) {
+            this.gridstore = new GridFs(this.database, 'attachments');
+        }
 
         // create logger
         this.logger = this.options.logger || {
@@ -215,6 +216,84 @@ class Indexer {
         return parseMimeTree(rfc822);
     }
 
+    /**
+     * Parses structured MIME tree from a rfc822 message source
+     *
+     * @param  {String|Buffer} rfc822 E-mail message as 'binary'-string or Buffer
+     * @return {Object} Parsed mime tree
+     */
+    storeAttachments(messageId, mimeTree, sizeLimit, callback) {
+        let walk = (node, next) => {
+
+            let continueProcessing = () => {
+                if (Array.isArray(node.childNodes)) {
+                    let pos = 0;
+                    let processChildNode = () => {
+                        if (pos >= node.childNodes.length) {
+                            return next();
+                        }
+                        let childNode = node.childNodes[pos++];
+                        walk(childNode, processChildNode);
+                    };
+                    setImmediate(processChildNode);
+                } else {
+                    setImmediate(next);
+                }
+            };
+
+            if (node.body && node.size > sizeLimit) {
+                let attachmentId = new ObjectID();
+                let contentType = node.parsedHeader['content-type'] && node.parsedHeader['content-type'].value || 'application/octet-stream';
+                let fileName = (node.parsedHeader['content-disposition'] && node.parsedHeader['content-disposition'].params && node.parsedHeader['content-disposition'].params.filename) || (node.parsedHeader['content-type'] && node.parsedHeader['content-type'].params && node.parsedHeader['content-type'].params.name) || false;
+
+                if (fileName) {
+                    try {
+                        fileName = libmime.decodeWords(fileName);
+                    } catch (E) {
+                        // failed to parse filename, keep as is (most probably an unknown charset is used)
+                    }
+                }
+
+                let returned = false;
+                let store = this.gridstore.createWriteStream(attachmentId, {
+                    fsync: true,
+                    content_type: contentType,
+                    metadata: {
+                        messages: [messageId],
+                        fileName,
+                        contentType,
+                        created: new Date()
+                    }
+                });
+
+                store.once('error', err => {
+                    if (returned) {
+                        return;
+                    }
+                    returned = true;
+                    callback(err);
+                });
+
+                store.on('close', () => {
+                    if (returned) {
+                        return;
+                    }
+                    returned = true;
+
+                    node.body = false;
+                    node.attachmentId = attachmentId;
+
+                    return continueProcessing();
+                });
+
+                store.end(Buffer.from(node.body, 'binary'));
+            } else {
+                continueProcessing();
+            }
+        };
+        walk(mimeTree, callback);
+    }
+
     /**
      * Generates IMAP compatible BODY object from message tree
      *

imap.js

@@ -10,7 +10,7 @@ const MongoClient = mongodb.MongoClient;
 const ImapNotifier = require('./imap-notifier');
 const imapHandler = IMAPServerModule.imapHandler;
 const bcrypt = require('bcryptjs');
-//const fs = require('fs');
+const ObjectID = require('mongodb').ObjectID;
 const Indexer = require('./imap-core/lib/indexer/indexer');
 
 // Setup server
@@ -563,7 +563,19 @@ server.onExpunge = function (path, update, session, callback) {
                 if (!message) {
                     return cursor.close(() => {
                         this.notifier.fire(username, path);
-                        return callback(null, true);
+
+                        // delete all attachments that do not have any active links to message objects
+                        database.collection('attachments.files').deleteMany({
+                            'metadata.messages': {
+                                $size: 0
+                            }
+                        }, err => {
+                            if (err) {
+                                // ignore as we don't really care if we have orphans or not
+                            }
+
+                            return callback(null, true);
+                        });
                     });
                 }
 
@@ -577,12 +589,28 @@ server.onExpunge = function (path, update, session, callback) {
                     if (err) {
                         return cursor.close(() => callback(err));
                     }
-                    this.notifier.addEntries(username, path, {
-                        command: 'EXPUNGE',
-                        ignore: session.id,
-                        uid: message.uid,
-                        message: message._id
-                    }, processNext);
+
+                    // remove link to message from attachments (if any exist)
+                    database.collection('attachments.files').updateMany({
+                        'metadata.messages': message._id
+                    }, {
+                        $pull: {
+                            'metadata.messages': message._id
+                        }
+                    }, {
+                        multi: true,
+                        w: 1
+                    }, err => {
+                        if (err) {
+                            // ignore as we don't really care if we have orphans or not
+                        }
+                        this.notifier.addEntries(username, path, {
+                            command: 'EXPUNGE',
+                            ignore: session.id,
+                            uid: message.uid,
+                            message: message._id
+                        }, processNext);
+                    });
                 });
             });
         };
@@ -644,6 +672,8 @@ server.onCopy = function (path, update, session, callback) {
                         });
                     }
 
+                    let sourceId = message._id;
+
                     sourceUid.unshift(message.uid);
                     database.collection('mailboxes').findOneAndUpdate({
                         _id: target._id
@@ -666,7 +696,7 @@ server.onCopy = function (path, update, session, callback) {
                         let uidNext = item.value.uidNext;
                         destinationUid.unshift(uidNext);
 
-                        message._id = null;
+                        message._id = new ObjectID();
                         message.mailbox = target._id;
                         message.uid = uidNext;
 
@@ -679,11 +709,27 @@ server.onCopy = function (path, update, session, callback) {
                             if (err) {
                                 return callback(err);
                             }
-                            this.notifier.addEntries(username, target.path, {
-                                command: 'EXISTS',
-                                uid: message.uid,
-                                message: message._id
-                            }, processNext);
+
+                            // remove link to message from attachments (if any exist)
+                            database.collection('attachments.files').updateMany({
+                                'metadata.messages': sourceId
+                            }, {
+                                $push: {
+                                    'metadata.messages': message._id
+                                }
+                            }, {
+                                multi: true,
+                                w: 1
+                            }, err => {
+                                if (err) {
+                                    // should we care about this error?
+                                }
+                                this.notifier.addEntries(username, target.path, {
+                                    command: 'EXISTS',
+                                    uid: message.uid,
+                                    message: message._id
+                                }, processNext);
+                            });
                         });
                     });
                 });
@@ -759,7 +805,8 @@ server.onFetch = function (path, options, session, callback) {
                     query: options.query,
                     values: session.getQueryResponse(options.query, message, {
                         logger: this.logger,
-                        fetchOptions: {}
+                        fetchOptions: {},
+                        database
                     })
                 }));
 
@@ -1059,6 +1106,7 @@ server.addToMailbox = (username, path, meta, date, flags, raw, callback) => {
     let envelope = server.indexer.getEnvelope(mimeTree);
     let bodystructure = server.indexer.getBodyStructure(mimeTree);
     let messageId = envelope[9] || uuidV1() + '@wildduck.email';
+    let id = new ObjectID();
 
     // check if mailbox exists
     database.collection('mailboxes').findOne({
@@ -1075,44 +1123,36 @@ server.addToMailbox = (username, path, meta, date, flags, raw, callback) => {
             return callback(err);
         }
 
-        // check if message with same Message-ID exists
-        database.collection('messages').findOne({
-            mailbox: mailbox._id,
-            messageId
-        }, (err, message) => {
+        // acquire new UID
+        database.collection('mailboxes').findOneAndUpdate({
+            _id: mailbox._id
+        }, {
+            $inc: {
+                uidNext: 1
+            }
+        }, {}, (err, item) => {
             if (err) {
                 return callback(err);
             }
 
-            if (message) {
-                // message already exists, skip
-                return callback(null, true, {
-                    uidValidity: mailbox.uidValidity,
-                    uid: message.uid
-                });
+            if (!item || !item.value) {
+                // was not able to acquire a lock
+                let err = new Error('Mailbox is missing');
+                err.imapResponse = 'TRYCREATE';
+                return callback(err);
             }
 
-            // acquire new UID
-            database.collection('mailboxes').findOneAndUpdate({
-                _id: mailbox._id
-            }, {
-                $inc: {
-                    uidNext: 1
-                }
-            }, {}, (err, item) => {
+            let mailbox = item.value;
+
+            // calculate size before removing large attachments from mime tree
+            let size = server.indexer.getSize(mimeTree);
+
+            // move large attachments to GridStore
+            server.indexer.storeAttachments(id, mimeTree, 50 * 1024, err => {
                 if (err) {
                     return callback(err);
                 }
 
-                if (!item || !item.value) {
-                    // was not able to acquire a lock
-                    let err = new Error('Mailbox is missing');
-                    err.imapResponse = 'TRYCREATE';
-                    return callback(err);
-                }
-
-                let mailbox = item.value;
-
                 let internaldate = date && new Date(date) || new Date();
                 let headerdate = mimeTree.parsedHeader.date && new Date(mimeTree.parsedHeader.date);
 
@@ -1121,13 +1161,14 @@ server.addToMailbox = (username, path, meta, date, flags, raw, callback) => {
                 }
 
                 let message = {
+                    _id: id,
                     mailbox: mailbox._id,
                     uid: mailbox.uidNext,
                     internaldate,
                     headerdate,
                     flags,
                     unseen: !flags.includes('\\Seen'),
-                    size: server.indexer.getSize(mimeTree),
+                    size,
                     meta,
                     modseq: 0,
                     mimeTree,
@@ -1171,7 +1212,9 @@ module.exports = done => {
 
         database = db;
 
-        server.indexer = new Indexer();
+        server.indexer = new Indexer({
+            database
+        });
 
         // setup notification system for updates
         server.notifier = new ImapNotifier({

indexes.js

@@ -78,3 +78,7 @@ db.messages.createIndex({
 db.messages.createIndex({
     uid: -1
 });
+
+db['attachments.files'].createIndex({
+    'metadata.messages': 1
+});

package.json

@@ -27,6 +27,7 @@
     "grid-fs": "^1.0.1",
     "joi": "^10.2.2",
     "libbase64": "^0.1.0",
+    "libmime": "^3.1.0",
     "mailparser": "^2.0.2",
     "mongodb": "^2.2.24",
     "nodemailer-fetch": "^2.1.0",