Summary:
Because of JavaScript's asynchronous nature, it is possible that we will
be processing several downloaded messages concurrently. This can lead to
calling extractContacts() in an interleaved fashion, which it was not
designed to handle. It looks up which contacts are already in the
database and then performs inserts or updates accordingly, assuming
nothing has changed in the contacts table in between---which is not
true! If several messages have similar contacts, an insert race can
cause one of the inserts to throw an unhandled exception.
We fix this by simply catching the unique constraint error, and falling
back to an update instead. (There's not really a better way to deal
with write races other than to enforce that we process contacts from
messages serially, as transactions are of no help here.)
This commit also removes extractContacts()'s return value, which is not
currently used and I found confusing.
Test Plan: manual
Reviewers: juan, evan, mark, jackie
Reviewed By: jackie
Differential Revision: https://phab.nylas.com/D3555
Summary:
This replaces the API delta stream with a direct in-memory one
Addresses T7300
Test Plan: manual
Reviewers: jackie, halla, juan
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D3548
Summary:
This diff solves a few separate issues from T7313, T7316, T7282, and it refactors
the send code a little bit.
Initially, the problem that led to this diff was generating message ids that
wouldn't collide (which was causing errors in the message-processor). Collisions
in ids were being caused by messages that contained the exact same participants,
subject and date (most likely due bots or scripts sending emails in quick
succession)
To prevent collisions this commit adds the `message-id` header as part of the
database message id, and ensures that we set it correctly before sending, and
that it remains consistent through send, multi-send, and the sync loop.
During the refactor and review, I removed some code that assumed that we were
syncing drafts (which we aren't), and also fixes a few other known and
unknown issues around sending, message creation, and tracking, like assigning
the correct date header (we were previously assigning the draft creation date
from within N1), fixing the tracking regex, among other smaller bugs/typos.
Will address inline TODOs in a separate diff
Test Plan: TODO!!! I will add tests in another diff
Reviewers: evan, halla, jackie, khamidou
Reviewed By: halla, jackie
Differential Revision: https://phab.nylas.com/D3507
We were getting sql unique constraint violation errors for ids because
we were attempting to create objects with the same id within the same
transaction.
This commit ensures that only attempt to write a contact with the same
id once, and that we check for all exsiting contacts before hand.
For file, we ensure that we don't attempt to write 2 files with the same
id more than once
We were using the stripped version of label/folder names for the id
hash, e.g. [Gmail]/Drafts would be Drafts.
However, we can't do this because it might collide with other names. e.g. if the
user created a Drafts label, it would end up colliding with [Gmail]/Drafts
Minor lint fix
Summary:
Fixes T7283. We weren't creating deltas for contacts because we were inserting contacts using `UPSERT`, which requires us to add another sequelize hook. Unfortunately, support for this hook is only available on sequelize 4.x.
I didn't want to upgrade our sequelize version right now and cause of bunch of mysterious failures, so I've simply changed the `UPSERT` to be either an `INSERT`or `UPDATE`. We didn't really need it anyway.
Test Plan: Checked that N1 was receiving contact deltas.
Reviewers: bengotow
Reviewed By: bengotow
Maniphest Tasks: T7283
Differential Revision: https://phab.nylas.com/D3481
Also renames JSONType() -> buildJSONColumnOptions() and
JSONARRAYType() -> buildJSONARRAYColumnOptions() to prevent passing
those return values in as just the type value instead of the entire
options object.
Since message IDs are now static but there's no good way to generate
static thread IDs while syncing an account from newest message first,
we give threads the ID of any message on that thread and, when setting
metadata, look up the local thread ID by first going through the
message table.
Summary:
This diff adds persistent unique ids for messages and contacts. For messages, we just take a hash of the headers. For contacts, we hash the contact's email address.
This diff bundles a couple of tiny fixes too, like always trying to restart an account's sync, even after an exception.
Note that since there's no reliable way to have persistent ids for threads, we'll have to change our code to use message ids instead. Alas, that's a story for another (massive) diff.
Test Plan: Tested manually.
Reviewers: bengotow
Reviewed By: bengotow
Differential Revision: https://phab.nylas.com/D3468
This will only generate 1 delta per new message instead of 2
Adds a MessageFactory to create messages from imapmessage objects, and
unifies and cleans up that logic
Also modified the message-processor to be more suited for local sync,
made emphasis that it's only for processing new messages, and got rid of
the while "pipeline" concept
- TODO message-processor specs are broken
