nodemailer/wildduck
1
1
Fork 0
mirror of https://github.com/nodemailer/wildduck.git synced 2024-11-10 17:47:07 +08:00
Code Issues Projects Releases Packages Wiki Activity

Added mailbox paging

Browse source
This commit is contained in:
Andris Reinman 2017-04-03 16:59:04 +03:00
parent fe9277bc7d
commit 6e40ea3b77
6 changed files with 555 additions and 39 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

199
README.md
View file

@ -11,7 +11,7 @@ Wild Duck is a distributed IMAP server built with Node.js, MongoDB and Redis. No
1. Build a scalable and distributed IMAP server that uses clustered database instead of single machine file system as mail store
2. Allow using internationalized email addresses
3. Provide Gmail-like features like pushing sent messages automatically to Sent Mail folder or notifying about messages moved to Junk folder so these could be marked as spam
4. Add push notifications. Your application (eg. a webmail client) should be able to request changes (new and deleted messages, flag changes) to be pushed to client instead of using IMAP to fetch stuff from the server
4. Provide parsed mailbox and message data over HTTP. This should make creating webmail interfaces super easy, no need to parse RFC822 messages to get text content or attachments
## Similar alterntives
@ -56,6 +56,7 @@ Yes, it does. You can run the server and get a working IMAP server for mail stor
3. Works almost on any OS including Windows. At least if you get MongoDB and Redis ([Windows fork](https://github.com/MSOpenTech/redis)) running first.
4. Focus on internationalization, ie. supporting email addresses with non-ascii characters
5. `+`-labels: _андрис+ööö@уайлддак.орг_ is delivered to _андрис@уайлддак.орг_
6. Access messages both using IMAP and HTTP API. The latter serves parsed data, so no need to fetch RFC822 messages and parse out html, plaintext content or attachments. It is super easy to create a webmail interface on top of this.
### Isn't it bad to use a database as a mail store?
@ -126,11 +127,16 @@ NODE_ENV=production npm start
### Step 5\. Create an user account
See see [below](#create-user) for details about creating new user accounts
See see [below](#http-api) for details about creating new user accounts
## Manage user
## HTTP API
Users can be managed with HTTP requests against Wild Duck API
Users, mailboxes and messages can be managed with HTTP requests against Wild Duck API
TODO:
1. Expose counters (seen/unseen messages, message count in mailbox etc.)
2. Search messages
### POST /user/create
@ -366,18 +372,191 @@ The response for successful operation should look like this:
}
```
### DELETE /message
### GET /mailbox/:id
Deletes a message from a mailbox.
List messages in a mailbox.
Arguments
Parameters
- **id** is the mailbox ID
- **size** is optional number to limit the length of the messages array (defaults to 20)
- **before** is an optional paging number (see *next* in response)
- **after** is an optional paging number (see *prev* in response)
Response includes the following fields
* **mailbox** is an object that lists some metadata about the current mailbox
* **id** is the mailbox ID
* **path** is the folder path
* **next** is an URL fragment for retrieving the next page (or false if there are no more pages)
* **prev** is an URL fragment for retrieving the previous page (or false if it is the first page)
* **messages** is an array of messages in the mailbox
* **id** is the message ID
* **date** is the date when this message was received
* **hasAttachments** is a boolean that indicates if this messages has attachments or not
* **intro** includes the first 256 characters from the message
* **subject** is the message title
* **from** is the From: field
* **to** is the To: field
* **cc** is the Cc: field
* **bcc** is the Bcc: field
The response for successful listing should look like this:
```json
{
"success": true,
"mailbox": {
"id": "58dbf87fcff690a8c30470c7",
"path": "INBOX"
},
"next": "/mailbox/58dbf87fcff690a8c30470c7?before=34&size=20",
"prev": false,
"messages": [
{
"id": "58e25243ab71621c3890417e",
"date": "2017-04-03T13:46:44.226Z",
"hasAttachments": true,
"intro": "Welcome to Ryan Finnie's MIME torture test. This message was designed to introduce a couple of the newer features of MIME-aware MUAs, features that have come around since the days of the original MIME torture test. Just to be clear, this message SUPPLEMENT…",
"subject": "ryan finnie's mime torture test v1.0",
"from": "ryan finnie <rfinnie@domain.dom>",
"to": "bob@domain.dom"
}
]
}
```
### GET /message/:id
Retrieves message information
Parameters
- **id** is the MongoDB _id as a string for a message
- **mailbox** is optional Mailbox id. Use this to verify that the message is located at this mailbox
**Example**
```
curl "http://localhost:8080/message?id=58d8299c5195c38e77c2daa5"
curl "http://localhost:8080/message/58d8299c5195c38e77c2daa5"
```
Response message includes the following fields
- **id** is the id of the message
- **headers** is an array that lists all headers of the message. A header is an object:
- **key** is the lowercase key of the header
- **value** is the header value in unicode (all encoded values are decoded to utf-8)
- **date** is the receive date (not header Date: field)
- **mailbox** is the id of the mailbox this messages belongs to
- **flags** is an array of IMAP flags for this message
- **text** is the plaintext version of the message (derived from html if not present in message source)
- **html** is the HTML version of the message (derived from plaintext if not present in message source)
- **attachments** is an array of attachment objects. Attachments can be shared between messages.
- **id** is the id of the attachment
- **fileName** is the name of the attachment. Autogenerated from Content-Type if not set in source
- **contentType** is the MIME type of the message
- **disposition** defines Content-Disposition and is either 'inline', 'attachment' or _false_
- **transferEncoding** defines Content-Transfer-Encoding
- **related** is a boolean value that states if the attachment should be hidden (_true_) or not. _Related_ attachments are usually embedded images
- **sizeKb** is the approximate size of the attachment in kilobytes
#### Embedded images
HTML content has embedded images linked with the following URL structure:
attachment:MESSAGE_ID/ATTACHMENT_ID
For example:
<img src="attachment:aaaaaa/bbbbbb">
To fetch the actual attachment contents for this image, use the following url:
http://localhost:8080/message/aaaaaa/attachment/bbbbbb
#### Example response
The response for successful operation should look like this:
```json
{
"success": true,
"message": {
"id": "58d8299c5195c38e77c2daa5",
"mailbox": "58dbf87fcff690a8c30470c7",
"headers": [
{
"key": "delivered-to",
"value": "andris@addrgw.com"
}
],
"date": "2017-04-03T10:34:43.007Z",
"flags": ["\\Seen"],
"text": "Hello world!",
"html": "<p>Hello world!</p>",
"attachments": [
{
"id": "58e2254289cccb742fd6c015",
"fileName": "image.png",
"contentType": "image/png",
"disposition": "attachment",
"transferEncoding": "base64",
"related": true,
"sizeKb": 1
}
]
}
}
```
### GET /message/:mid/attachment/:aid
Retrieves an attachment of the message
Parameters
- **mid** is the message ID
- **aid** is the attachment ID
**Example**
```
curl "http://localhost:8080/message/58d8299c5195c38e77c2daa5/attachment/58e2254289cccb742fd6c015"
```
### GET /message/:id/raw
Retrieves RFC822 source of the message
Parameters
- **id** is the MongoDB _id as a string for a message
- **mailbox** is optional Mailbox id. Use this to verify that the message is located at this mailbox
**Example**
```
curl "http://localhost:8080/message/58d8299c5195c38e77c2daa5/raw"
```
### DELETE /message/:id
Deletes a message from a mailbox.
Parameters
- **id** is the MongoDB _id as a string for a message
- **mailbox** is an optional Mailbox id. Use this to verify that the message to be deleted is located at this mailbox
**Example**
```
curl -XDELETE "http://localhost:8080/message/58d8299c5195c38e77c2daa5"
```
The response for successful operation should look like this:
@ -385,7 +564,9 @@ The response for successful operation should look like this:
```json
{
"success": true,
"id": "58d8299c5195c38e77c2daa5"
"message":{
"id": "58d8299c5195c38e77c2daa5"
}
}
```

346
api.js
View file

@ -9,6 +9,8 @@ const tools = require('./lib/tools');
const MessageHandler = require('./lib/message-handler');
const db = require('./lib/db');
const ObjectID = require('mongodb').ObjectID;
const libqp = require('libqp');
const libbase64 = require('libbase64');
const server = restify.createServer({
name: 'Wild Duck API',
@ -673,15 +675,23 @@ server.get('/user/mailboxes', (req, res, next) => {
});
});
server.del('/message', (req, res, next) => {
// FIXME: if listing a page after the last one then there is no prev URL
// Probably should detect the last page the same way the first one is detected
server.get('/mailbox/:id', (req, res, next) => {
res.charSet('utf-8');
const schema = Joi.object().keys({
id: Joi.string().hex().lowercase().length(24).required()
id: Joi.string().hex().lowercase().length(24).required(),
before: Joi.number().default(0),
after: Joi.number().default(0),
size: Joi.number().min(1).max(50).default(20)
});
const result = Joi.validate({
id: req.params.id
id: req.params.id,
before: req.params.before,
after: req.params.after,
size: req.params.size
}, schema, {
abortEarly: false,
convert: true,
@ -696,8 +706,13 @@ server.del('/message', (req, res, next) => {
}
let id = result.value.id;
let before = result.value.before;
let after = result.value.after;
let size = result.value.size;
messageHandler.del(id, (err, success) => {
db.database.collection('mailboxes').findOne({
_id: new ObjectID(id)
}, (err, mailbox) => {
if (err) {
res.json({
error: 'MongoDB Error: ' + err.message,
@ -705,12 +720,136 @@ server.del('/message', (req, res, next) => {
});
return next();
}
if (!mailbox) {
res.json({
error: 'This mailbox does not exist',
id
});
return next();
}
res.json({
success,
id
let query = {
mailbox: mailbox._id
};
let reverse = false;
let sort = {
uid: -1
};
if (req.params.before) {
query.uid = {
$lt: before
};
} else if (req.params.after) {
query.uid = {
$gt: after
};
sort = {
uid: 1
};
reverse = true;
}
db.database.collection('messages').findOne({
mailbox: mailbox._id
}, {
fields: {
uid: true
},
sort: {
uid: -1
}
}, (err, entry) => {
if (err) {
res.json({
error: 'MongoDB Error: ' + err.message,
id
});
return next();
}
if (!entry) {
res.json({
success: true,
mailbox: {
id: mailbox._id,
path: mailbox.path
},
next: false,
prev: false,
messages: []
});
return next();
}
let newest = entry.uid;
db.database.collection('messages').find(query, {
uid: true,
mailbox: true,
internaldate: true,
headers: true,
hasAttachments: true,
intro: true
}).sort(sort).limit(size).toArray((err, messages) => {
if (err) {
res.json({
error: 'MongoDB Error: ' + err.message,
id
});
return next();
}
if (reverse) {
messages = messages.reverse();
}
let nextPage = false;
let prevPage = false;
if (messages.length) {
if (after || before) {
prevPage = messages[0].uid;
if (prevPage >= newest) {
prevPage = false;
}
}
if (messages.length >= size) {
nextPage = messages[messages.length - 1].uid;
if (nextPage <= 0) {
nextPage = false;
}
}
}
res.json({
success: true,
mailbox: {
id: mailbox._id,
path: mailbox.path
},
next: nextPage ? '/mailbox/' + id + '?before=' + nextPage + '&size=' + size : false,
prev: prevPage ? '/mailbox/' + id + '?after=' + prevPage + '&size=' + size : false,
messages: messages.map(message => {
let response = {
id: message._id,
date: message.internaldate,
hasAttachments: message.hasAttachments,
intro: message.intro
};
message.headers.forEach(entry => {
if (['subject', 'from', 'to', 'cc', 'bcc'].includes(entry.key)) {
response[entry.key] = entry.value;
}
});
return response;
})
});
return next();
});
});
return next();
});
});
@ -791,6 +930,197 @@ server.get('/message/:id', (req, res, next) => {
});
});
server.get('/message/:id/raw', (req, res, next) => {
res.charSet('utf-8');
const schema = Joi.object().keys({
id: Joi.string().hex().lowercase().length(24).required(),
mailbox: Joi.string().hex().lowercase().length(24).optional()
});
const result = Joi.validate({
id: req.params.id,
mailbox: req.params.mailbox
}, schema, {
abortEarly: false,
convert: true,
allowUnknown: true
});
if (result.error) {
res.json({
error: result.error.message
});
return next();
}
let id = result.value.id;
let mailbox = result.value.mailbox;
let query = {
_id: new ObjectID(id)
};
if (mailbox) {
query.mailbox = new ObjectID(mailbox);
}
db.database.collection('messages').findOne(query, {
mimeTree: true,
size: true
}, (err, message) => {
if (err) {
res.json({
error: 'MongoDB Error: ' + err.message,
id
});
return next();
}
if (!message) {
res.json({
error: 'This message does not exist',
id
});
return next();
}
let response = messageHandler.indexer.rebuild(message.mimeTree);
if (!response || response.type !== 'stream' || !response.value) {
res.json({
error: 'Can not fetch message',
id
});
return next();
}
res.writeHead(200, {
'Content-Type': 'message/rfc822'
});
response.value.pipe(res);
});
});
server.get('/message/:message/attachment/:attachment', (req, res, next) => {
res.charSet('utf-8');
const schema = Joi.object().keys({
message: Joi.string().hex().lowercase().length(24).required(),
attachment: Joi.string().hex().lowercase().length(24).required()
});
const result = Joi.validate({
message: req.params.message,
attachment: req.params.attachment
}, schema, {
abortEarly: false,
convert: true,
allowUnknown: true
});
if (result.error) {
res.json({
error: result.error.message
});
return next();
}
let message = result.value.message;
let attachment = result.value.attachment;
let query = {
_id: new ObjectID(attachment),
'metadata.messages': new ObjectID(message)
};
db.database.collection('attachments.files').findOne(query, (err, messageData) => {
if (err) {
res.json({
error: 'MongoDB Error: ' + err.message,
attachment,
message
});
return next();
}
if (!messageData) {
res.json({
error: 'This message does not exist',
attachment,
message
});
return next();
}
res.writeHead(200, {
'Content-Type': messageData.metadata.contentType
});
let attachmentStream = messageHandler.indexer.gridstore.createReadStream(messageData._id);
attachmentStream.once('error', err => res.emit('error', err));
if (messageData.metadata.transferEncoding === 'base64') {
attachmentStream.pipe(new libbase64.Decoder()).pipe(res);
} else if (messageData.metadata.transferEncoding === 'quoted-printable') {
attachmentStream.pipe(new libqp.Decoder()).pipe(res);
} else {
attachmentStream.pipe(res);
}
});
});
server.del('/message/:id', (req, res, next) => {
res.charSet('utf-8');
const schema = Joi.object().keys({
id: Joi.string().hex().lowercase().length(24).required(),
mailbox: Joi.string().hex().lowercase().length(24).optional()
});
const result = Joi.validate({
id: req.params.id,
mailbox: req.params.mailbox
}, schema, {
abortEarly: false,
convert: true,
allowUnknown: true
});
if (result.error) {
res.json({
error: result.error.message
});
return next();
}
let id = result.value.id;
let mailbox = result.value.mailbox;
let query = {
_id: new ObjectID(id)
};
if (mailbox) {
query.mailbox = new ObjectID(mailbox);
}
messageHandler.del(query, (err, success) => {
if (err) {
res.json({
error: 'MongoDB Error: ' + err.message,
id
});
return next();
}
res.json({
success,
id
});
return next();
});
});
module.exports = done => {
let started = false;

2
examples/push-message.js
View file

@ -34,7 +34,7 @@ function send() {
},
from: 'Kärbes 🐧 <andris@kreata.ee>',
to: 'Ämblik 🦉 <' + recipient + '>, andmekala@hot.ee, Müriaad Polüteism <müriaad@müriaad-polüteism.dev>',
to: 'Ämblik 🦉 <'+recipient+'>, andmekala@hot.ee, Müriaad Polüteism <müriaad@müriaad-polüteism.org>',
subject: 'Test ööö message [' + Date.now() + ']',
text: 'Hello world! Current time is ' + new Date().toString(),
html: '<p>Hello world! Current time is <em>' + new Date().toString() + '</em> <img src="cid:note@example.com"/> <img src="http://www.neti.ee/img/neti-logo-2015-1.png"></p>',

21
imap-core/lib/indexer/indexer.js
View file

@ -301,10 +301,12 @@ class Indexer {
}
let disposition = (parsedDisposition && parsedDisposition.value || '').toLowerCase().trim() || false;
let isInlineText = false;
// If the current node is HTML or Plaintext then allow larger content included in the mime tree
// Also decode text/html value
if (['text/plain', 'text/html'].includes(contentType) && (!disposition || disposition === 'inline')) {
if (['text/plain', 'text/html', 'text/rfc822-headers', 'message/delivery-status'].includes(contentType) && (!disposition || disposition === 'inline')) {
isInlineText = true;
if (node.body && node.body.length) {
let charset = parsedContentType.params.charset || 'windows-1257';
let content = node.body;
@ -329,7 +331,12 @@ class Indexer {
content = content.toString();
}
if (contentType === 'text/plain') {
if (contentType === 'text/html') {
htmlContent.push(content.trim());
if (!alternative) {
textContent.push(htmlToText.fromString(content).trim());
}
} else {
textContent.push(content.trim());
if (!alternative) {
htmlContent.push(marked(content, {
@ -340,17 +347,12 @@ class Indexer {
smartypants: true
}).trim());
}
} else if (contentType === 'text/html') {
htmlContent.push(content.trim());
if (!alternative) {
textContent.push(htmlToText.fromString(content).trim());
}
}
}
}
// remove attachments and very large text nodes from the mime tree
if (node.body && (node.size > 300 * 1024 || disposition === 'attachment')) {
if (node.body && (!isInlineText || node.size > 300 * 1024)) {
let attachmentId = new ObjectID();
let fileName = (node.parsedHeader['content-disposition'] && node.parsedHeader['content-disposition'].params && node.parsedHeader['content-disposition'].params.filename) || (node.parsedHeader['content-type'] && node.parsedHeader['content-type'].params && node.parsedHeader['content-type'].params.name) || false;
@ -390,7 +392,8 @@ class Indexer {
}
});
if (!['text/plain', 'text/html'].includes(contentType) || disposition === 'attachment') {
// do not include text content, multipart elements and embedded messages in the attachment list
if (!isInlineText && contentType.split('/')[0] !== 'multipart' && !(contentType === 'message/rfc822' && (!disposition || disposition === 'inline'))) {
// list in the attachments array
response.attachments.push({
id: attachmentId,

14
indexes.json
View file

@ -66,6 +66,12 @@
"uid": 1,
"modseq": 1
}
}, {
"name": "newer_first",
"key": {
"mailbox": 1,
"uid": -1
}
}, {
"name": "mailbox_flags",
"key": {
@ -102,12 +108,6 @@
"mailbox": 1,
"size": 1
}
}, {
"name": "by_uid",
"key": {
"mailbox": 1,
"uid": 1
}
}, {
"name": "by_headers",
"key": {
@ -116,7 +116,7 @@
"headers.value": 1
}
}, {
"name": "bhas_attachments",
"name": "has_attachments",
"key": {
"mailbox": 1,
"hasAttachments": 1

12
lib/message-handler.js
View file

@ -201,7 +201,7 @@ class MessageHandler {
if (maildata.attachments && maildata.attachments.length) {
message.attachments = maildata.attachments;
message.hasAttachments = true;
}else{
} else {
message.hasAttachments = false;
}
@ -210,6 +210,10 @@ class MessageHandler {
if (maildata.text) {
message.text = maildata.text.replace(/\r\n/g, '\n').trim();
message.text = message.text.length <= maxTextLength ? message.text : message.text.substr(0, maxTextLength);
message.intro = message.text.replace(/\s+/g, ' ').trim();
if (message.intro.length > 256) {
message.intro = message.intro.substr(0, 256) + '…';
}
}
if (maildata.html) {
message.html = this.cleanHtml(maildata.html.replace(/\r\n/g, '\n')).trim();
@ -272,10 +276,8 @@ class MessageHandler {
});
}
del(messageId, callback) {
this.database.collection('messages').findOne({
_id: typeof messageId === 'string' ? new ObjectID(messageId) : messageId
}, (err, message) => {
del(query, callback) {
this.database.collection('messages').findOne(query, (err, message) => {
if (err) {
return callback(err);
}