Summary:
We were seeing JS blocking in snippet extraction of up to 2k ms. This
is because we were walking the entire DOM of a message and extracting
all text, regardless of message size---and using our own homegrown
DOM walker function.
To remedy this, use the standard TreeWalker from the Chrome browser
APIs (which in benchmarks looks 2-4x faster) and also exit out of
the DOM walking process once we've accumulated enough text to create
a snippet. Informal eyeballing of timing metrics for this function suggests
the new implementation is something like 10-100x faster for some messages.
As a bonus, we get to delete some code and end up with a cleaner
implementation!
Test Plan: old unit tests yaay
Reviewers: juan
Reviewed By: juan
Subscribers: evan
Differential Revision: https://phab.nylas.com/D3543
Summary:
Save metadata correctly by reassigning an object to value.
Since account IDs are different between N1 and N1 Cloud, use just the message ID, which should be unique.
Test Plan: Tested locally.
Reviewers: evan, juan
Reviewed By: evan, juan
Differential Revision: https://phab.nylas.com/D3524
Summary:
This is a refactor of the auth APIs to use async/await. Gmail Auth is
pretty confusing and I wanted to make it cleaner to read and easier to
use. This is also part of the general API upgrade to modern ES6
This also fixes the Gmail auth error we saw at showcase
Test Plan: manual
Reviewers: halla, jackie, mark, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3535
Summary:
See https://github.com/mscdex/node-imap/issues/585 for details.
This issue was causing us constantly run the sync loop without pauses,
i.e. every next sync loop was scheduled immediately.
Currently, when we receive a new `'mail'`event, we trigger a new sync loop. Previously, when this happened while a sync loop was already in progress we would just ignore the event. However, my recent patch keeps track of how many times we tried to start a sync loop while one was already in progress. If the number of times this happens is > 0, it will schedule the next sync loop immediately (as opposed to waiting a constant amount seconds before the next loop).
The problem is that this new logic is making the sync worker always schedule the next sync loop immediately (without pausing for a few seconds). This is due to the following chain of events (assume we are just syncing `all` and `trash` folders):
This commit is a temporary workaround to this problem.
Test Plan: manual
Reviewers: evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3537
Summary:
Associated N1 Diff: D3530
This commit converts multi-send from a 3 step process into a 2 step
process
The first step creates the base message and sends a message per
recipient, each with its customized message body for tracking.
The second step reconciles all sent messages, specifically removing any
sent messages created by gmail, and saving the correct message to the
sent folder
This commit also ensures that we run the send tasks immediately by
ensuring we restart the sync loop if its already running
Test Plan: Manual
Reviewers: evan, jackie, halla
Reviewed By: jackie, halla
Differential Revision: https://phab.nylas.com/D3529
Summary:
This is a small patch but it's pretty complex, because of the numbers of moving parts. Gmail has two types of tokens, access and refresh tokens. Access tokens have a limited shelf life of one hour. After that they expire and you need to use your refresh token to get a new one.
We've decided to do the access token generation on the server, because we don't feel comfortable giving our users both our Google client id and secret. To do that, I've added an endpoint, `/gmail/auth/refresh` which returns a valid access token as well as an expiration date for the token.
The only place where we handle token expiration is in the sync workers. Before trying opening a new connection we check if our access token is expired. If yes, we get a new one from the API. If there's an issue doing this, we notify N1 using `NylasAPIHelpers.handleAuthenticationFailure`.
There's a second patch for N1 with tiny related fixes.
Test Plan: Tested manually. Will need to test more in the real world.
Reviewers: evan, jackie, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3522
Summary:
- Don't fail if there's no subject, just set it to `(no subject)`
- Support "BINARY" content-transfer-encoding. (This really means
that there is no encoding, so it's simple to add support for it)
Test Plan: tested locally
Reviewers: jackie
Reviewed By: jackie
Differential Revision: https://phab.nylas.com/D3528
Summary:
Add internal state to the sync worker to allow for it to be interrupted
and restarted.
The concept in this commit is that if we've tried to trigger a sync enough times
while its already in progress, bail and start over.
Usually, we manually trigger sync loops when we queue a new SyncbackTasks,
so that the newly queued task gets executed. This is necessary because the only
way to run SyncbackTasks is via the sync loop, for consistency and simplicity
reasons.
For example, we might run into a case where we queue a SendMessage task,
and we want it to be executed ASAP, but if we're in the middle of a
syncing a mailbox with a ton of folders, we wont get to the SendMessage
task after some considerable time.
Specifically this commit makes it so:
- If the number of sync attempts while in progress is > 0, make sure we schedule the next sync immediately
- If we reach a threshold of sync attempts while in progress, interrupt sync and restart
Test Plan: todo :(
Reviewers: mark, spang, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3520
Summary:
We were doing all of this fancy filtering and sorting to determine which
tasks to run, but in the end, we were just running the whole unsorted
list (using the wrong variable)
Also extracted getting the list of tasks into its own function, for
easier unit tests (it should really have some)
And, also wanted to make sure other people looked at this code, since I
believe no one has before.
Test Plan: todo
Reviewers: mark, khamidou, spang, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D3516
Summary: See description at T7327
Test Plan: Manual, but this should have unit tests
Reviewers: mark, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3513
Summary:
Move sendmail-client and errors to isomorphic-core, given that they will
probably be used by cloud-workers (plugin backends) and cloud-api
Depends on D3510
Test Plan: Manual
Reviewers: halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D3512
Summary:
Associated N1 diff: D3511
Convert send endpoints to use syncback tasks for consistency with how we
perform other imap operations, but primarily:
- So that it triggers a sync loop immediately and we pick up changes quickly
- To keep track of various send operations as a single unit (e.g. sending + saving to sent folder or deleting from sent)
This commit also fixes SyncbackRequest error handling and processing in
N1-- previously we were saving error fields to the syncbackRequests with
a format that didn't match N1's API error and which wasn't properly
serializable. (Also rename HTTPError to APIError)
Test Plan: Todo/Manual
Reviewers: jackie, halla, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3510
Summary:
I've found this useful for generating test cases and am tired
of adding and removing this code!
Test Plan: inspect output of /tmp/k2-parse-output
Reviewers: juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3518
Summary: This was understandably causing some messages to fail to display correctly.
Test Plan: unit tests are already broken for message parsing -- will fix in follow up diff
Reviewers: juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3517
Summary:
This diff solves a few separate issues from T7313, T7316, T7282, and it refactors
the send code a little bit.
Initially, the problem that led to this diff was generating message ids that
wouldn't collide (which was causing errors in the message-processor). Collisions
in ids were being caused by messages that contained the exact same participants,
subject and date (most likely due bots or scripts sending emails in quick
succession)
To prevent collisions this commit adds the `message-id` header as part of the
database message id, and ensures that we set it correctly before sending, and
that it remains consistent through send, multi-send, and the sync loop.
During the refactor and review, I removed some code that assumed that we were
syncing drafts (which we aren't), and also fixes a few other known and
unknown issues around sending, message creation, and tracking, like assigning
the correct date header (we were previously assigning the draft creation date
from within N1), fixing the tracking regex, among other smaller bugs/typos.
Will address inline TODOs in a separate diff
Test Plan: TODO!!! I will add tests in another diff
Reviewers: evan, halla, jackie, khamidou
Reviewed By: halla, jackie
Differential Revision: https://phab.nylas.com/D3507
Summary:
We forgot to refactor `publicId` to just `id`, which was breaking contact
deltas in N1.
Test Plan: Run locally
Reviewers: khamidou, evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3508
Summary: See title
Test Plan: Run locally
Reviewers: juan, evan
Reviewed By: juan, evan
Maniphest Tasks: T7281
Differential Revision: https://phab.nylas.com/D3498
Summary:
Allow other fields to be passed in with participants, so that Contact
objects can be passed in as JSON. Also fix an error check.
Test Plan: tested locally
Reviewers: jackie
Reviewed By: jackie
Differential Revision: https://phab.nylas.com/D3504
Summary:
This fixes multiple issues, including snippets telling you you
ought to look at the HTML as well as cruft like HTML entities
and CSS in snippets.
Test Plan: unit tests included o.O
Reviewers: juan
Reviewed By: juan
Subscribers: evan
Differential Revision: https://phab.nylas.com/D3500
Summary: See title
Test Plan: Ran it locally
Reviewers: khamidou, juan, evan
Reviewed By: juan, evan
Differential Revision: https://phab.nylas.com/D3496
Summary:
1) Send the custom body, rather than the generic body
2) Extract contacts correctly so that the saved sent message has all the
participants, rather than just the last one
Test Plan: Tested locally
Reviewers: jackie
Reviewed By: jackie
Differential Revision: https://phab.nylas.com/D3499
Summary:
This commit fixes the following bugs in message parsing:
- we were unilaterally decoding MIME bodies as UTF-8; instead, decode according
to the charset data in the mimepart header
- '7bit' content-transfer-encoding means us-ascii, NOT utf-7
- only interpret valid content-transfer-encodings (previously we were trying
to treat various charsets as transfer-encodings)
- clearer naming: s/values/parsedMessage/
- unify snippet cleanup between plaintext & stripped HTML (merging
whitespace etc.)
Test Plan: units tests coming
Reviewers: juan
Differential Revision: https://phab.nylas.com/D3491
We were getting sql unique constraint violation errors for ids because
we were attempting to create objects with the same id within the same
transaction.
This commit ensures that only attempt to write a contact with the same
id once, and that we check for all exsiting contacts before hand.
For file, we ensure that we don't attempt to write 2 files with the same
id more than once
We were using the stripped version of label/folder names for the id
hash, e.g. [Gmail]/Drafts would be Drafts.
However, we can't do this because it might collide with other names. e.g. if the
user created a Drafts label, it would end up colliding with [Gmail]/Drafts
Minor lint fix
Summary:
This commit also fixes snippets for HTML-only messages to strip out HTML
tags, and makes us preserve whitespace for plaintext emails by
displaying them in <pre class="nylas-plaintext"> tags, and makes us log
messages that fail to parse at all to a tempdir.
The only issue I found with using <pre> tags for plaintext email was
that some lines may trigger scrolling, so there is an associated commit
(D3484) that changes the CSS for <pre class="nylas-plaintext"> to wrap
lines.
In the future, we can add regression tests to this test suite whenever
we fix parsing bugs.
Test Plan: unit tests included
Reviewers: bengotow
Reviewed By: bengotow
Differential Revision: https://phab.nylas.com/D3483
When running syncback requests, if a task is meant to change the UID of
a message (e.g. move it to a new folder), that task should be run after
other tasks that don't affect the UID. Otherwise, when trying to run the
other tasks, they would reference a UID that is no longer valid.
This commit will make sure that we run any tasks that will change message uids last,
/and/ make sure that we don't run more than 1 task that will affect the uids of
the same messages in a row (i.e. without running a sync loop in between)
This commit will correctly keep track of folder and label ids when
creating them from N1.
Previously, when we sent the request to create a folder or label to our api,
we would immediately get back a serverId because it was created optimistically
in the back end— given that K2 is strictly non-optimistic, we won’t have a serverId
until some undetermined time in the future, and we need to somehow reference
the object that /was/ optimistically created in N1 to update the ui when
we do get the server id.
Since we can deterministically generate ids for folders and labels,
we "guess" what its going to be, and include it in the props of the syncback request
returned to N1. This is the simplest solution to get thing working
correctly right now, but we’ll need to revisit this in the future for
other types of objects (drafts, contacts, events), and revisit how we
will manage optimistic updates in N1 when we merge the 2 codebases
with K2 (given that K2 was designed to be non-optimisitc).
Summary:
Fixes T7283. We weren't creating deltas for contacts because we were inserting contacts using `UPSERT`, which requires us to add another sequelize hook. Unfortunately, support for this hook is only available on sequelize 4.x.
I didn't want to upgrade our sequelize version right now and cause of bunch of mysterious failures, so I've simply changed the `UPSERT` to be either an `INSERT`or `UPDATE`. We didn't really need it anyway.
Test Plan: Checked that N1 was receiving contact deltas.
Reviewers: bengotow
Reviewed By: bengotow
Maniphest Tasks: T7283
Differential Revision: https://phab.nylas.com/D3481
Also renames JSONType() -> buildJSONColumnOptions() and
JSONARRAYType() -> buildJSONARRAYColumnOptions() to prevent passing
those return values in as just the type value instead of the entire
options object.
Since message IDs are now static but there's no good way to generate
static thread IDs while syncing an account from newest message first,
we give threads the ID of any message on that thread and, when setting
metadata, look up the local thread ID by first going through the
message table.
- Don’t need functions in delta.js which must be called to return promsies. Fun of promsies is that you don’t need to care when they’re built to attach a .then.
- Make boundary between route handler and delta stream builder more explicit, don’t do query parsing in helpers, always reply from handler.
- Remove pushJSON extension to outputStream which never actually received JSON.
- Remove `takeUntil` - disposing of the downstream observable should dispose of all the merged/upstream observables
- Rename inflate => stringify since the returned value is a string not an object.
- Remove support for delta streams with no cursors. Don’t think this was supposed to be a feature.
- Add accountId to Transaction models
- Make database hooks shared in isomorphic core
Summary:
T7253 has two related parts:
1. Stripping the '[Gmail]/' prefix from any canonical Gmail folders
2. Handling nested IMAP folders.
This diff fixes 1. Changes for 2. are forthcoming.
Test Plan: Tested manually. Checked that the part was stripped from the N1 folder list.
Reviewers: juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3475
Moving to between gmail folders (all, spam, trash) or moving to inbox,
involves changing labels /and/ folders, simultaneously. For this I added
a task to perform both operations, and apply labels first before
attempting to move the folder
This also ensures that we can use the accountId as part of the ids for
other models, like labels and folders
Before this commit, labels and folders with the same name for different
accounts would have the same id, screwing things up in n1
Summary:
This diff adds persistent unique ids for messages and contacts. For messages, we just take a hash of the headers. For contacts, we hash the contact's email address.
This diff bundles a couple of tiny fixes too, like always trying to restart an account's sync, even after an exception.
Note that since there's no reliable way to have persistent ids for threads, we'll have to change our code to use message ids instead. Alas, that's a story for another (massive) diff.
Test Plan: Tested manually.
Reviewers: bengotow
Reviewed By: bengotow
Differential Revision: https://phab.nylas.com/D3468
This will only generate 1 delta per new message instead of 2
Adds a MessageFactory to create messages from imapmessage objects, and
unifies and cleans up that logic
Also modified the message-processor to be more suited for local sync,
made emphasis that it's only for processing new messages, and got rid of
the while "pipeline" concept
- TODO message-processor specs are broken
- Remove unused exported PromiseUtils from cloud-core, import instead
from nylas-core where needed
- Remove unused MessageTypes from cloud-core
- Move MessageTypes from nylas-core to local-sync. It's completely
isolated to local-sync
