Summary:
This swaps out our generic IMAP threading mechanism to use the threading
headers on the message instead of the prior way of grouping by subject
and then differentiating based on participants, as that design was
somewhat driven by what we could accomplish easily given legacy data
schema decisions and has serious caveats, such as different threads between
the same people with the same subject being misthreaded together. With K2, we
have free reign to change the data format, so we can do it right.
The algorithm is super simple:
- Define "references" as the union of the Message-Id, In-Reply-To, and
References headers on a message, filtered for valid RFC2822 Message-IDs
- On message sync, if any element of the new message's references
matches any element of an existing message's references, thread them
together
In order to accomplish this, we need to store References in a way that
allows each element to be indexed for fast lookup. That meant either
using the sqlite JSON1 extension + expression-based indices, or creating
a new table. I chose the latter as a time-tested and simple solution,
since we don't need the flexibility of JSON here.
Test Plan: manual - unit tests coming
Reviewers: khamidou, evan, juan
Reviewed By: evan, juan
Differential Revision: https://phab.nylas.com/D3651
Summary:
We may find it useful at some point to be able to tell which messages in
a user's mailbox were sent using N1/the REST API vs Nylas Mail.
Test Plan: manual
Reviewers: evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3631
Summary:
Well-behaved MUAs don't do this, but it is totally legal and we should
handle it.
The Python sync engine handles this properly also:
7db949fec9/sync-engine/inbox/util/addr.py (L26-L47)
(Found this out while scratching my head over the output format of
mimelib.parseHeaders---turns out it's an array for a reason:
> let to = mimelib.parseHeaders(`To: Christine Spang <spang@nylas.com>
... To: Foo Bar <foo@example.com>`);
undefined
> to
{ to:
[ 'Christine Spang <spang@nylas.com>',
'Foo Bar <foo@example.com>' ] }
)
Test Plan: unit test included
Reviewers: halla, jackie, mark
Reviewed By: mark
Differential Revision: https://phab.nylas.com/D3625
Summary:
We have another function called extractContacts, in its own file,
extract-contacts.js, which is used to create Contact objects. This has
confused me a number of times and also leads to grep collisions.
This patch also makes the snippet unit tests pass again after a recent
API change.
Test Plan: included unit tests, manual
Reviewers: halla, evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3623
Summary:
Sync operations are mostly bound by I/O and the imap connection.
What we believe that is mostly affecting cpu and battery life is that node’s event
loop is being hosed with cpu intensive message processing operations.
To alleviate this, we do a few things:
- Restore a global message processing queue to process messages serially and meter cpu usage (message processing continues to be a fire and forget call from within sync operations)
- Move actual cpu intensive work to the message processing queue, i.e. `MessageFactory.parseFromImap`
- Keep track of message processing queue length, and skip sync operations if queue is too big to prevent massive memory consumption
This commit also renames the package from new-message-processor to
message-processor, given that now it processes both new and existing
messages, and we like to minimize confusion.
Test Plan: manual
Reviewers: spang, khamidou, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3602
Summary:
Because of the way we were attempting to parse contacts from
From/To/Cc/Bcc headers by converting them to JSON with a regex, we were
erroneously breaking contacts that contained commas in quoted names into
multiple contacts. This could result in things like parsing multiple
addresses for the From: header, incorrectly!
To resolve the problem, replace our homegrown logic with mimelib's
seemingly excellent parseAddresses(), which handles this and a myriad of
other cases correctly.
Fixes: T7370
Test Plan: unit tests included
Reviewers: mark
Reviewed By: mark
Differential Revision: https://phab.nylas.com/D3565
Will fix these once I've finished up the current slew of bugfixes I'm
working on---kind of a pain to ensure they're passing in all
intermediate states.
Summary:
This was leading us to put funny things like 'Nylas !' in some snippets that used
tags like <i> and <b> for text formatting. This is probs a teeny little bit slower
than the previous version since it invokes a callback on a lot more nodes, but we
can't really fix this issue without knowledge of the preceding tag name.
Test Plan: unit test included!!
Reviewers: evan, jackie
Reviewed By: jackie
Differential Revision: https://phab.nylas.com/D3553
Summary:
This fixes multiple issues, including snippets telling you you
ought to look at the HTML as well as cruft like HTML entities
and CSS in snippets.
Test Plan: unit tests included o.O
Reviewers: juan
Reviewed By: juan
Subscribers: evan
Differential Revision: https://phab.nylas.com/D3500
Summary:
This commit also fixes snippets for HTML-only messages to strip out HTML
tags, and makes us preserve whitespace for plaintext emails by
displaying them in <pre class="nylas-plaintext"> tags, and makes us log
messages that fail to parse at all to a tempdir.
The only issue I found with using <pre> tags for plaintext email was
that some lines may trigger scrolling, so there is an associated commit
(D3484) that changes the CSS for <pre class="nylas-plaintext"> to wrap
lines.
In the future, we can add regression tests to this test suite whenever
we fix parsing bugs.
Test Plan: unit tests included
Reviewers: bengotow
Reviewed By: bengotow
Differential Revision: https://phab.nylas.com/D3483
Sometimes imap mailboxes aren't properly flagged with their roles, so we
check the display names against known variations to see if we can find any
missing roles.
