Commit graph

20 commits

Author SHA1 Message Date
Christine Spang e1271cf25c [local-sync] Give Nylas Mail messages a different Message-ID from API messages
Summary:
We may find it useful at some point to be able to tell which messages in
a user's mailbox were sent using N1/the REST API vs Nylas Mail.

Test Plan: manual

Reviewers: evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3631
2017-01-11 15:51:12 -08:00
Christine Spang 9eb1aef139 [local-sync] Properly handle multiple occurrences of From/To/Cc/Bcc headers
Summary:
Well-behaved MUAs don't do this, but it is totally legal and we should
handle it.

The Python sync engine handles this properly also:
7db949fec9/sync-engine/inbox/util/addr.py (L26-L47)

(Found this out while scratching my head over the output format of
mimelib.parseHeaders---turns out it's an array for a reason:

> let to = mimelib.parseHeaders(`To: Christine Spang <spang@nylas.com>
... To: Foo Bar <foo@example.com>`);
undefined
> to
{ to:
   [ 'Christine Spang <spang@nylas.com>',
     'Foo Bar <foo@example.com>' ] }

)

Test Plan: unit test included

Reviewers: halla, jackie, mark

Reviewed By: mark

Differential Revision: https://phab.nylas.com/D3625
2017-01-10 15:07:36 -08:00
Christine Spang a234570118 [local-sync] Rename extract{Snippet,Contacts} in message-factory
Summary:
We have another function called extractContacts, in its own file,
extract-contacts.js, which is used to create Contact objects. This has
confused me a number of times and also leads to grep collisions.

This patch also makes the snippet unit tests pass again after a recent
API change.

Test Plan: included unit tests, manual

Reviewers: halla, evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3623
2017-01-10 10:41:35 -08:00
Juan Tejada 83ef8c12b3 [local-sync] Restore global queue for message processing to improve perf
Summary:
Sync operations are mostly bound by I/O and the imap connection.
What we believe that is mostly affecting cpu and battery life is that node’s event
loop is being hosed with cpu intensive message processing operations.

To alleviate this, we do a few things:

- Restore a global message processing queue to process messages serially and meter cpu usage (message processing continues to be a fire and forget call from within sync operations)
- Move actual cpu intensive work to the message processing queue, i.e. `MessageFactory.parseFromImap`
- Keep track of message processing queue length, and skip sync operations if queue is too big to prevent massive memory consumption

This commit also renames the package from new-message-processor to
message-processor, given that now it processes both new and existing
messages, and we like to minimize confusion.

Test Plan: manual

Reviewers: spang, khamidou, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3602
2017-01-06 14:28:33 -08:00
Christine Spang 40c7b09e27 [local-sync] Use mimelib to parse contacts
Summary:
Because of the way we were attempting to parse contacts from
From/To/Cc/Bcc headers by converting them to JSON with a regex, we were
erroneously breaking contacts that contained commas in quoted names into
multiple contacts. This could result in things like parsing multiple
addresses for the From: header, incorrectly!

To resolve the problem, replace our homegrown logic with mimelib's
seemingly excellent parseAddresses(), which handles this and a myriad of
other cases correctly.

Fixes: T7370

Test Plan: unit tests included

Reviewers: mark

Reviewed By: mark

Differential Revision: https://phab.nylas.com/D3565
2016-12-29 15:48:12 -08:00
Christine Spang e30b299b39 [local-sync] Mark message factory tests for fixing later
Will fix these once I've finished up the current slew of bugfixes I'm
working on---kind of a pain to ensure they're passing in all
intermediate states.
2016-12-29 10:59:17 -08:00
Christine Spang de1b67287c fix(snippet-parsing): Don't add extraneous spaces after text format tags
Summary:
This was leading us to put funny things like 'Nylas !' in some snippets that used
tags like <i> and <b> for text formatting. This is probs a teeny little bit slower
than the previous version since it invokes a callback on a lot more nodes, but we
can't really fix this issue without knowledge of the preceding tag name.

Test Plan: unit test included!!

Reviewers: evan, jackie

Reviewed By: jackie

Differential Revision: https://phab.nylas.com/D3553
2016-12-23 18:26:35 -08:00
Juan Tejada 7c701c6369 [local-sync] Fix date parsing and tests
Make sure that we use the header date for our date field because that's
the one we can control and depend on for message id generation
2016-12-19 15:35:50 -08:00
Christine Spang c6a80efc44 [local-sync] Fix parseFromImap specs
Lots has changed for the better! Tests work again also.
2016-12-16 14:42:26 -08:00
Evan Morikawa ee831ba0aa [local-private, local-sync] temporarily disable specs
[local-private] Disable more broken specs

Disable specs

Reenable local-sync tests
2016-12-15 12:57:24 -05:00
Christine Spang c214ba1e34 [local-sync] Parse DOM to extract snippets
Summary:
This fixes multiple issues, including snippets telling you you
ought to look at the HTML as well as cruft like HTML entities
and CSS in snippets.

Test Plan: unit tests included o.O

Reviewers: juan

Reviewed By: juan

Subscribers: evan

Differential Revision: https://phab.nylas.com/D3500
2016-12-13 16:32:22 -08:00
Christine Spang a23c68092e [local-sync] Add specs for message parsing
Summary:
This commit also fixes snippets for HTML-only messages to strip out HTML
tags, and makes us preserve whitespace for plaintext emails by
displaying them in <pre class="nylas-plaintext"> tags, and makes us log
messages that fail to parse at all to a tempdir.

The only issue I found with using <pre> tags for plaintext email was
that some lines may trigger scrolling, so there is an associated commit
(D3484) that changes the CSS for <pre class="nylas-plaintext"> to wrap
lines.

In the future, we can add regression tests to this test suite whenever
we fix parsing bugs.

Test Plan: unit tests included

Reviewers: bengotow

Reviewed By: bengotow

Differential Revision: https://phab.nylas.com/D3483
2016-12-07 07:25:28 -08:00
Ben Gotow aed1d59916 [local-sync] fix(folder-list): Support children when parent has no attribs 2016-12-05 16:20:17 -08:00
Ben Gotow 1e3b346c94 [local-sync] feat(specs): Add basic tests for folder sync 2016-12-05 16:07:49 -08:00
Ben Gotow 269d61171a [local-sync] fix(specs): New specs for threading 2016-12-05 15:00:37 -08:00
Ben Gotow 30c8bedd7a [local-sync] fix(specs): run npm test in local-sync dir 2016-12-05 12:16:53 -08:00
Evan Morikawa d01b4ffe77 [isomorphic-core] Uppdate refs nylas-core => isomorphic-core 2016-11-23 09:12:45 -08:00
Evan Morikawa 106d7e57a1 Rename DatabaseConnector to LocalDatabaseConnector and move 2016-11-22 17:51:43 -08:00
Evan Morikawa 951feae657 [local-sync]: moved spec into similarly named folder and updated refs 2016-11-22 16:08:10 -08:00
Halla Moore 845139826b [local-sync] Check for missing Category roles by localized display names
Sometimes imap mailboxes aren't properly flagged with their roles, so we
check the display names against known variations to see if we can find any
missing roles.
2016-11-22 15:31:35 -08:00