Commit graph

741 commits

Author SHA1 Message Date
Juan Tejada 0c34238862 🎨 rm bad comment 2017-02-03 11:30:15 -08:00
Christine Spang 8b9f89ab14 [local-api] 🎨 Remove unused message and file GET routes
Summary:
This code is dead and has confused me grepping around the codebase before.

If for some unexpected reason we need these routes back in the future, we
can always extract them from version control. For now the routes we aren't
using are a distraction.

Test Plan: been using Nylas Mail with this local patch all week

Reviewers: evan, halla, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3833
2017-02-03 10:42:22 -08:00
Evan Morikawa 32b955ad36 [local-private] update to use hosted Sentry 2017-02-02 17:21:32 -08:00
Juan Tejada 574f171fbd [local-sync] Properly set the sent label for Gmail accounts
Summary:
We can't try to set or remove the sent label on gmail accounts because
the operation will silently fail and cause the threads to later bounce
back.

This occurred when trying to delete or archive a thread that contained a
sent message, and we incorrectly tried to overwrite or remove all of the
labels on messages, without regard for sent.

This was causing https://github.com/nylas/nylas-mail/issues/2706 and
sending and archiving to immediately bounce back.

Addresses T7757

Test Plan: unit tests

Reviewers: halla, evan, spang

Reviewed By: halla, evan, spang

Differential Revision: https://phab.nylas.com/D3829
2017-02-02 17:20:38 -08:00
Juan Tejada 7b63912955 [local-sync] Correctly retry syncback tasks
Summary:
Previously, we attempted to immeditely retry syncback tasks when a
RetryableError was encountered. However, if the error was an
IMAPConnection error, we would keep retrying with a broken connection,
which would keep failing.

The correct way to retry is to wait for the next sync loop, since at the
beginning of each loop we ensure that we are correctly connected to
imap.

To achieve this this commit simply marks a failed task as NEW if it
encoutners a RetryableError and we haven't retried too many times. To
keep track of the number of retries, we save a new field in the `props`
field of the request.

Test Plan: manual

Reviewers: evan, halla, mark, spang

Reviewed By: spang

Subscribers: khamidou

Differential Revision: https://phab.nylas.com/D3831
2017-02-02 17:19:13 -08:00
Karim Hamidou ae32666609 [cloud-api] Base infrastructure for running cloud workers
Summary:
I've decided to break my snooze patches in multiple parts to make it easier to review. This diff contains all the code related to running workers in the cloud. Cloud workers all inherit from the `Worker` class which defines a bunch of useful things like error handling.

What's left to do:
- spawn workers based on the plugin type
- add monitoring (I'm going to add a simple HTTP endpoint for that)
- writing a migration for the local sync db and the prod metadata db.

Test Plan: Tested manually.

Reviewers: halla, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3814
2017-02-02 16:14:59 -08:00
Juan Tejada cb9faea6f0 [local-sync] Small restructuring of local-sync specs
Summary: Make them map the same directory structure of `src`

Test Plan: unit

Reviewers: evan, halla, spang

Reviewed By: halla, spang

Differential Revision: https://phab.nylas.com/D3826
2017-02-02 14:48:26 -08:00
Christine Spang c6f371aa0f [local-sync] Serialize category sync progress to edgehill rather than syncState
Summary:
syncState on folders may contain arbitrarily long arrays of UIDs
(particularly, failedUIDs). If we serialize this JSON column to
edgehill.db, we can end up serializing very large objects when
persisting the local task queue. When the queue contains many tasks,
this can balloon the JSON blob to megabytes, causing the main window and
the worker window to become unresponsive.

The UI doesn't need to know about IMAP bookkeeping internals, so
serialize the sync progress instead of the sync state. This has the
advantages that (1) we don't need to worry about future keys added
to the syncState being large and (2) when we add Exchange support
we already have an abstraction for sync progress.

Test Plan: manual

Reviewers: juan, mark, halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D3817
2017-02-01 07:15:33 -08:00
Evan Morikawa efba50bd9f [local-sync] properly cleanup broken messages 2017-02-01 06:57:30 -08:00
Evan Morikawa 04ff46eb04 [local-sync] fix cleanup of sent messages 2017-02-01 06:24:03 -08:00
Evan Morikawa 47f1f440d7 [local-sync] fixed issue where single digit date would create dupes 2017-01-31 18:00:42 -08:00
Juan Tejada 4769d4d476 [local-sync] Save error when clearing tasks INPROGRESS 2017-01-31 15:59:56 -08:00
Juan Tejada 7029653c00 [local-sync] Retry syncback tasks that throw retryable errors
Summary:
This will prevent us from showing error messages to the user when we
can automatically recover from the error

Test Plan: manual-- throw error from syncback task, check expected results

Reviewers: evan, mark, spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D3812
2017-01-31 10:54:18 -08:00
Juan Tejada 71959b44fb [local-sync] Account for additional IMAP retryable errors
Summary:
There are 2 types of IMAP errors that need to be treated as retryable. See code
comments as to why.

Test Plan: manual

Reviewers: khamidou, evan, spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D3811
2017-01-31 10:53:43 -08:00
Juan Tejada 6aa5ad509a 🎨 remove outdated comments 2017-01-30 22:28:38 -08:00
Juan Tejada 9abcf7e8d8 [local-sync] Report permanent sync errors to sentry 2017-01-30 21:59:03 -08:00
Evan Morikawa e2d201c4b7 [*] update nylas/N1 to nylas/nylas-mail GitHub link 2017-01-30 11:50:15 -08:00
Juan Tejada 469cbad22b [local-sync] Fix compilation of FetchNewMessagesInFolder operation
Summary: This was broken because babel could not compile the `super` keyword

Test Plan: manual

Reviewers: evan, mark, halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D3804
2017-01-27 13:57:19 -08:00
Juan Tejada b4973589f0 [local-sync] Sync new uid in sent folder when moving to msg to sent
Summary:
EnsureMessageInSentFolder also needs to sync the sent folder to fetch
the uid of the newly moved message (like `MoveThreadToFolder` does)

Test Plan: manual

Reviewers: halla, mark, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3803
2017-01-27 12:50:19 -08:00
Juan Tejada e29fe2ee9b [local-sync] When fetching /new/ messages, make sure we have fetchedmax
Summary:
In `FetchNewMessagesInFolder`, sometimes we haven't synced anything in the folder
we are trying to fetch new messages in. Previously this would just throw
an error, now we properly check if we have a fetchedmax, and if not just
run a normal fetch.

Also, when the target folder box was already open, we were not fetching the /latest/ box status to check the latest uidnext value, so we would skip fetching new messages when in fact there were new messages to fetch

(This can happen for example when moving a sent message to the Sent
folder before we've started syncing the Sent folder)

Test Plan: manual

Reviewers: halla, mark, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3802
2017-01-27 12:49:18 -08:00
Juan Tejada a0356ca76f [cloud-*] Add more detailed logs to auth endpoints
Summary: see title

Test Plan: manual, deploy to staging, check that it works

Reviewers: evan, spang, tomasz, khamidou

Reviewed By: tomasz

Differential Revision: https://phab.nylas.com/D3800
2017-01-27 11:48:06 -08:00
Juan Tejada 575ff4a73e [local-sync] Ensure messages always have a uid after syncback tasks
Summary:
Syncback tasks that move messages to a different folder used to leave
those messages without a uid because it was unknown at the moment, and
it would only be discovered later on in the sync loop.
This had the unwanted effect of not allowing you to perform more than 1
syncback action on the same thread back to back (e.g. undoing an archive, or
sending and archiving immediately)

This commit makes it so that `SetThreadFolderAndLabels` and
`MoveThreadToFolder` runs a new sync task to fetch the new uids for the
moved messages.

We do this via a new sync task, `FetchNewMessagesInFolder` which /only/
fetches new messages in a folder (no fetching old messages or attribute
updates). This is a first step to cleaning up the gigantic
`FetchMessagesInFolder` task into smaller parts-- but that will come in
a separate diff. For now we want to fix the immediate problem.

See D3788 and D3789 for more details

Test Plan:
manually move threads around, undo moving threads, reply on a thread
and immediately archive

Reviewers: khamidou, mark, spang, halla, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3798
2017-01-27 08:28:33 -08:00
Juan Tejada 4480fd89a2 [local-sync] Correctly check presence of messages when updating thread
Summary:
updateMessagesFromThread requires `messages` to be an array only when
recompute is falsy

Test Plan: manual

Reviewers: halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D3797
2017-01-26 16:08:17 -08:00
Juan Tejada 651a199cb6 [local-sync] Correctly check uidvalitity when syncing and save when updated
Summary:
If there were any uidvalidity changes after we've completely synced a folder, we
would completely ignore them and not attempt to sync the folder.

Also, we weren't saving the latest uidvalidity from the box to the folder
syncState as soon as we recovered, we only saved it until after fetching
messages. This meant that if the operation was interrupted before updating
syncState.uidvalidity, we would always think that we were in a state of uid invalidity

Recovering from uidvalidity was also broken because we weren't resetting the
fetched ranges, and unnecessarily setting the folderId to null, which meant that
we would never restore the uids of messages we had already fetched.

Test Plan: manual

Reviewers: evan, khamidou, mark, halla

Reviewed By: mark

Differential Revision: https://phab.nylas.com/D3796
2017-01-26 16:05:18 -08:00
Juan Tejada 668be7463d [local-sync] Correctly close imap connections when downloading files
Summary: Addresses T7708

Test Plan: manual

Reviewers: evan, khamidou, mark, halla

Reviewed By: khamidou, mark, halla

Subscribers: halla

Differential Revision: https://phab.nylas.com/D3795
2017-01-26 15:27:56 -08:00
Juan Tejada ded002924a [local-sync] Make invalid imap box errors retryable
Summary:
If we attempt to operate on a box that is no longer open, we should make
the error retryable so that we re-open the correct box and continue
syncing instead of showing the scary red box to users

Addresses T7680

Test Plan: manual

Reviewers: evan, spang, halla, mark

Reviewed By: halla, mark

Subscribers: mark

Differential Revision: https://phab.nylas.com/D3792
2017-01-26 13:32:39 -08:00
Juan Tejada 1d96c3f4da [local-sync] Ensure syncback actions have uids available before running
Summary:
Given that we perform syncback actions optimistically, i.e. we save the changes to the database before they are synced, some messages might not have an IMAP UID if they haven't been synced in the loop.

When we encounter messages without uid, we could just skip them, but this would mean that we end up moving only a subset of the messages we intended to move, or not move any at all, while thinking that we /did/ move everything, which might cause the thread to bounceback later on when we sync the messages that were missing.

The permanent fix is for syncback actions to not succeed until their changes have been synced, which is coming up in a separate diff

Test Plan: manual

Reviewers: evan, spang, khamidou, mark

Reviewed By: mark

Subscribers: mark

Differential Revision: https://phab.nylas.com/D3789
2017-01-26 13:31:53 -08:00
Juan Tejada f98b9449bf [local-sync] Fix transaction syntax 2017-01-26 09:19:17 -08:00
Juan Tejada 2391f58dfc Add comment 2017-01-26 00:06:20 -08:00
Juan Tejada 073310f49f [local-sync] Make sure folder move syncback actions clear folderImapUID 2017-01-26 00:06:20 -08:00
Christine Spang 6ddabff389 [local-sync] Avoid skipping All Mail updates if no new messages during Gmail inbox sync
Summary:
In JavaScript, null <= null is truthy. So if you set fetchedmax but not
fetchedmin and minUID, and then highestmodseq doesn't increment, we will
incorrectly skip updates on All Mail after the first batch of Gmail
inbox UID downloads, only making further progress on sync if the user
receives a new mail.

This patch tightens isSyncComplete() to require that all variables used
in the math comparisons are set.

Test Plan: manual 😢

Reviewers: evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3787
2017-01-25 16:31:20 -08:00
Mark Hahnenberg 7ad9e36cec [local-sync] Set isBatteryCharging on startup
Summary:
Previously we just assumed that we were in the "charging" state and wouldn't
update that until the state changed. This would cause us to throttle even if
the app was opened while plugged in. Now we don't do that.

Test Plan: Run locally, verify that we no longer throttle

Reviewers: spang, evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3782
2017-01-25 14:03:29 -08:00
Juan Tejada c097a86169 [local-sync] Fix isDraft check (check array exists)
Summary: see title

Test Plan: manual

Reviewers: spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D3781
2017-01-25 13:53:03 -08:00
Juan Tejada b33c55443b [local-sync] Fix attribute updates
Summary:
Previously, when updating message attributes during folder sync, we
would fetch local db messages only by folderImapUID. This was incorrect,
because messages in different folders could have the same uid, so our
query would return messages from other folders, causing us to
incorrectly update their attributes.

This commit ensures we fetch local messages in the current folder

Test Plan: manual

Reviewers: evan, halla, mark, spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D3777
2017-01-25 12:25:34 -08:00
Juan Tejada 470a47abfe [local-sync] Make sure thread is saved before updating folders/labels
We now correctly `await` for the `setFolders` and `setLabels`
operations, and make sure that the thread is always saved.

Apparently the `thread.id` check wasn't working as expected and we were skipping a
thread save, which only became apparent when we awaited for `setFolders`
and `setLabels`. Now, we always save the thread (like we did before this
method got updated)
2017-01-25 10:50:10 -08:00
Christine Spang 3f61e3da6a [local-sync] Sync isDraft correctly on Gmail
Summary:
When I tested if we could use labels in place of the flag, I had the
Drafts folder selected, not All Mail, and the label did not show up.
Turns out that if you have All Mail selected, all draft messages will be
correctly tagged with the Draft label.

With this patch, we should correctly sync isDraft on all supported
providers and suppress these messages from the app until if or when we
decide to implement draft sync.

Thanks Brandon Long from the Gmail team for the help in getting this right.

Test Plan: manual

Reviewers: juan, evan

Reviewed By: juan, evan

Differential Revision: https://phab.nylas.com/D3775
2017-01-25 10:36:46 -08:00
Juan Tejada cdf43ba804 [local-sync] Outbox shouldn't be included in localized names for sent
Summary:
Otherwise accounts with `Sent` and an `Outbox` will both get the `sent`
role, causing inconsistencies in sync

Addresses T7682

Test Plan: manual

Reviewers: khamidou, spang

Reviewed By: khamidou, spang

Differential Revision: https://phab.nylas.com/D3773
2017-01-25 09:59:36 -08:00
Christine Spang bc0b411be7 [local-sync] Remove duplicate db calls for looking up existing messages
Summary:
Now that message processing is serialized, this is unnecessary. I
was debugging something else and watched the code step through the
exact same db query twice in a row.

Test Plan: manual

Reviewers: evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3764
2017-01-25 09:53:48 -08:00
Christine Spang fba9ac0147 [local-sync] Still sync new mail during Gmail inbox UID prioritization
Summary: Fixes T7620

Test Plan: manual

Reviewers: mark, juan

Reviewed By: juan

Maniphest Tasks: T7620

Differential Revision: https://phab.nylas.com/D3765
2017-01-25 09:49:16 -08:00
Christine Spang 933ebdfebb [iso-core] Split cert validation error handling off socket error handling
Summary:
Now that we don't do strict validation of certificates for non-major IMAP
providers this shouldn't come up as much, but when it does we're gonna
want a better error message to help support out.

I am not 100% sure there aren't other socket errors that should be fatal,
but this was the one I could figure out by test authing against a server
with a self-signed cert and grepping around the node socket source code.

Test Plan: manual

Reviewers: evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3774
2017-01-25 09:45:20 -08:00
Evan Morikawa ddcd097c9b [local-sync] remove /account API endpoint
Summary:
It turns out we don't ever use the /account API endpoint.

The GET /accounts endpoint was designed to query all accounts connected to
a system for old K2, but we don't use that anymore. The environment
variable protecting the endpoint isn't set anywhere.

We used to `GET /account` from the N1 AccountStore to attempt to refresh
the health of the accounts. The accompianing diff to this one makes that
obsolete. We never need to query for the account health since the sync
loop pushes it to us through `Actions.updateAccount`.

We also never `DELETE /account` because our local-sync runs a function
called `ensureK2Consistency` that compares its DB against the Nylas Mail
`AccountStore`.

This file has always been a huge source of confusion to me and a massive
red herring for anyone trying to understand how the account system work.
The accompyaning diff has more comments explaining the existing system

Depends on https://phab.nylas.com/D3770

Test Plan:
Manually boot N1. Ensure existing account works. Add a new account. Remove
an account. Open the developer tools and check that all the tabs still
work.

Lots of grepping through the code base.

Reviewers: halla, mark, juan, khamidou

Reviewed By: juan, khamidou

Differential Revision: https://phab.nylas.com/D3769
2017-01-25 10:10:10 -05:00
Juan Tejada 0b89947c13 [local-sync] Make EnsureMessageInSentFolder save db changes after imap success
Summary:
This will solve T7579 when saving messages to the sent folder. I
attempted to clean up the references code but decided it was better left for a
new diff, so added a bunch of TODO's in this diff

Test Plan: manual

Reviewers: halla, spang, evan

Reviewed By: spang, evan

Differential Revision: https://phab.nylas.com/D3766
2017-01-25 00:29:30 -08:00
Christine Spang ca5c676c18 [iso-core] 🎨 Use a more descriptive variable name in _resolveIMAPSettings() 2017-01-24 19:22:14 -08:00
Christine Spang 6f25c1de34 [iso-core] Allow self-signed certificates in IMAP connections to non-major providers
Summary:
Unfortunately, many IMAP hosts outside the major ones do not have
certificates issued by a certificate authority, and it is very confusing
to folks to have their account auth not work. This patch relaxes our
certificate requirements for IMAP hosts outside the major providers.

It's cool that node 6 has secure TLS settings by default!

Fixes: T7673

Test Plan: manual

Reviewers: mark, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3771
2017-01-24 18:52:30 -08:00
Juan Tejada b0f2068b08 [*] report sending errors to sentry 2017-01-24 13:13:05 -08:00
Juan Tejada 6811a6f37e [*] Increment IMAP auth timeout
- More than 50% of auth errors we're seeing still seem to be geniune
timeouts. Increment the timeout to account for this.
2017-01-24 11:35:46 -08:00
Juan Tejada 8b3944d235 [local-sync] Make category syncback tasks save changes after imap success
Summary: This will solve T7579 for actions involving categories (folders/labels)

Test Plan: manual

Reviewers: khamidou, halla, mark, evan, spang

Reviewed By: evan, spang

Differential Revision: https://phab.nylas.com/D3753
2017-01-24 07:30:43 -08:00
Juan Tejada e909c8dfc7 [local-sync] Make thread syncback tasks save changes after imap success
Summary: This will solve T7579 for thread operations

Test Plan: manual

Reviewers: evan, mark, khamidou, halla, spang

Reviewed By: halla, spang

Differential Revision: https://phab.nylas.com/D3755
2017-01-24 07:29:25 -08:00
Christine Spang 359e32282f [local-sync] Sync draft flag from provider to K2 & exclude drafts from Edgehill
Summary:
We've been syncing drafts messages but not the drafts flag in K2, making
them appear in Edgehill as regular old messages.

This commit makes K2 sync the drafts flag, and also correctly label
folders called "Drafts" with the 'drafts' role.

Because 2-way syncing of drafts is very complex and error-prone since
you need to add new drafts and delete the old ones on every update, and
we reaally don't want to do things like create multiplying draft copies
or accidentally lose a draft someone started composing elsewhere, we
simply exclude messages marked as drafts from being serialized to
Edgehill through the delta stream for now. This removes the confusing
behaviour and also sets a better stage for completing drafts sync later.

Eventually we will also want to add functionality to allow users to
select their drafts folder, but for now this code does the right thing
in many more cases.

While investigating this behaviour, I also discovered a bug we've never
seen before where Gmail isn't applying the \Draft flag to draft
messages, no matter which folder we fetch them from. :-/ This is very
unfortunate and there's no way for us to work around it other than to
fetch messages in the Drafts folder and manually apply the flag locally,
since "drafts" is not a label in Gmail, only another IMAP folder. Brandon
Long from the Gmail team says that this is because they've had
problems with clients which sync drafts, so the Gmail web client and
mobile apps do not set the \Draft flag on drafts. (I don't get how this
solves their problem, but okay.) Let's solve the issue on Gmail if it
comes up by user demand—should be relatively straightforward to
implement, but it adds sync work & complexity.

Fixes T7593

Test Plan: manual

Reviewers: halla, juan

Reviewed By: juan

Maniphest Tasks: T7593

Differential Revision: https://phab.nylas.com/D3749
2017-01-23 20:58:32 -08:00
Christine Spang 902ce10881 [local-sync] Don't be so strict about updating threads from messages
This is how we _add_ messages to a thread in the first place, so we cannot
place the constraint that the thread must contain the message already.
2017-01-23 20:54:01 -08:00
Mark Hahnenberg ac6ad38296 [local-sync] Throttle message processing when on battery
Summary:
We don't want to run message processing full tilt when a user isn't plugged in.
This diff adds some detection logic that causes message processing to be
throttled/unthrottled when a user unplugs/plugs in their computer.

Test Plan: Run locally unplugged and plugged in, verify that CPU use goes up/down

Reviewers: evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3759
2017-01-23 18:31:35 -08:00
Halla Moore 9f5631b1fe [local-sync]: Update threads when they have messages removed
Summary:
See title

Depends on D3744

Test Plan: tested locally

Reviewers: spang, evan, juan

Reviewed By: spang, evan, juan

Differential Revision: https://phab.nylas.com/D3745
2017-01-23 17:04:39 -08:00
Christine Spang 0db62579bd [local-sync] Keep only one HTML part in multipart/alternative structures
Summary:
I thought it was gonna be OK that we kept all HTML parts in a multipart/alternative
MIME structure because the world is a sane place and nobody would ever put more
than one HTML part in a multipart/alternative structure.

I was wrong.

We have found extraterrestrial life^W^WI mean emails which contain duplicate,
exactly the same MIME parts within a multipart/alternative MIME structure: two
text/plain parts and two text/html parts. This is likely due to a broken MIME
implementation, or perhaps a bug in someone's email script. So, we should
only keep one text/html MIME part if there are multiple.

Test Plan:
manual for now---added this to my mail parsing regression test list
for implementation once we unify the DBs and have a roughly stable code
structure

Reviewers: halla, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3750
2017-01-20 17:53:57 -08:00
Halla Moore 4d10abcc2c [local-sync] Be more permissive with content transfer encodings (CTEs)
Summary:
Rather than having a strict model where we don't decode the message if we
don't specifically recognize the CTE, treat any CTEs we don't recognize as
having no encoding. There are several CTE strings that could mean this (e.g.
7bit, 7BITS, 8-bit, binary, NONE, utf8), and we don't want to check for them
all. Additionally, if there is a CTE we don't support, the user will likely
see rendering issues and contact support. This will allow us to obtain more
concrete information about these messages.

Test Plan: manual

Reviewers: spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D3748
2017-01-20 13:54:53 -08:00
Juan Tejada 0a12608971 [local-sync] Never show red box on retryable errors
Summary:
Given that we were marking the account as errored if we've encountered
enough RetryableErrors, we would show the red box to the user when in
fact the problem was the user was offline, causing confusion

If the user is offline, we will constantly get RetryableErrors in the
sync loop, and we can't mark the account as errored in that case.

Test Plan: manual

Reviewers: evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3752
2017-01-20 10:16:28 -08:00
Juan Tejada 5025dbe080 🎨 Error message 2017-01-19 23:33:22 -08:00
Christine Spang fb8fc9f9e8 [local-sync] typo fix 2017-01-19 19:27:50 -08:00
Karim Hamidou 9b40df95b4 [mini-feat] Display the IMAP commands we're sending when NYLAS_DEBUG is set.
Makes debugging a bit easier.
2017-01-19 16:59:27 -08:00
Juan Tejada 76ce0d6e2f [local-sync]: Token error handler fix 2017-01-19 11:25:31 -08:00
Juan Tejada 83b57be19f [local-sync] Properly handle errors when refreshing tokens
Summary:
This commit ensures that we handle transient errors correctly when refreshing
tokens

Test Plan: manual

Reviewers: khamidou, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3740
2017-01-19 11:16:27 -08:00
Juan Tejada f09bc902a0 [local-sync] Add a few improvements to error management
Summary:
In the sync worker:
- Move the backoff logic inside `scheduleNextSync`, where all logic to schedule the next sync loop now lives
- If we've retried a RetryableError a bunch of times, show the error to the user, otherwise the user might think the app is not working for no reason
- Clean up logging

In the message processor:
- Report message processing errors to sentry!

Sync Process Manager:
- Listen to new `Actions.debugSync` to show the Activity Window and open dev tools

Test Plan: manual

Reviewers: khamidou, evan

Reviewed By: khamidou, evan

Differential Revision: https://phab.nylas.com/D3736
2017-01-19 10:52:21 -08:00
Juan Tejada b4ef3f47a7 [local-sync] Batch imap operations in syncback tasks for faster performance
Summary:
This commit makes it so our syncback tasks send as few imap commands as possible by passing a set of UIDs whenever possible. Previously, we would send 1 command per message, with a single UID, which was very wasteful given that we can pass a set of UIDs. This is especially helpful for operating on threads with a large number of messages.

Syncback actions will now group all messages in a thread by the folder they belong to, and issue a single operation on the folder box. When removing all labels from a thread (setting labels to []), we need to issue a command of the form `box.delLabels(uids, labels)`, so we also group messages by set their set of labels to issue as few commands as possible.

This commit only batches imap commands, but we can still batch syncback actions themselves, which can be implemented in a separate patch.

Test Plan: manual

Reviewers: evan, mark, spang

Reviewed By: spang

Subscribers: halla, mg

Differential Revision: https://phab.nylas.com/D3719
2017-01-19 09:53:34 -08:00
Evan Morikawa a124b6c100 [local-sync] protection from reset action being double-fired
Summary: Double-firing protection since the DatabaseStore can now fire this

Test Plan: manual

Reviewers: juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3730
2017-01-18 17:30:30 -08:00
Evan Morikawa 02cf6fbf6e [isomorphic-core] have IMAP auth retry
Summary: IMAP auth retries

Test Plan: manual

Reviewers: khamidou, juan

Reviewed By: khamidou, juan

Differential Revision: https://phab.nylas.com/D3735
2017-01-18 17:29:58 -08:00
Juan Tejada 4e86d26478 [*] Increment default timeout connection
Summary:
We are getting to many imap timeout connection errors because the
authTimeout was just 5 secs

Test Plan: manual

Reviewers: khamidou, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3733
2017-01-18 14:05:27 -08:00
Juan Tejada 05fd276ec7 [cloud-api] Improve auth error logging 2017-01-18 13:36:50 -08:00
Juan Tejada 7c99e9c55f 🎨 2017-01-18 10:35:25 -08:00
Juan Tejada e1f1b22ee2 [local-sync] Ensure all imap operations are wrapped with timeout
Summary:
Specifically, these imap connections have been known to hang when you close your laptop or go offline/online, even though we are passing a `socketTimeout` to node-imap. When they hang, everything freezes because the promise waiting for the result never resolves.

`_createConnectionPromise` wraps the operations with a timeout we implemented ourselves, and correctly rejects on timeout.

This commit wraps other imap operations that were missing. (I notices because I encountered the hanging on one of these operations)

Test Plan: manual

Reviewers: evan, mark, spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D3720
2017-01-18 10:24:26 -08:00
Christine Spang 944b677d3e [local-sync] Set threading headers when uploading message to Sent folder
Summary:
We weren't, which meant that us sending with multi-send or generic IMAP
broke threading. :(

Test Plan: manual

Reviewers: juan, evan

Reviewed By: juan, evan

Differential Revision: https://phab.nylas.com/D3718
2017-01-16 20:17:44 -08:00
Juan Tejada 107fcbf355 [local-sync] Fix attribute sync for non Gmail accounts
Summary:
Before this commit, if folder sync was complete, and the account didn't support CONDSTORE (e.g. Office365, Yahoo), we would only check for attribute updates every 10 minutes.

This commit makes it so we always check for attribute updates if the server doesn't support CONDSTORE

So for example, when marking a thread as read, we would perform the optimistic update in N1, queue the syncback task which would succeed, but the thread in k2s db would never get updated and become stale, with an unreadCount > 0. If we emitted a delta for that thread during the window of time where we ignored attribute updates, it would be set as unread again in N1, even though all of its messages were read.

This still doesn't guarantee that it wont happen (we could still get a delta for the thread before we actually fetch the attribute updates from IMAP), but before this commit it was sure to happen. This should be properly fixed with the sync scheduler refactor

Test Plan: manual

Reviewers: evan, mark, spang

Reviewed By: mark, spang

Differential Revision: https://phab.nylas.com/D3714
2017-01-16 19:09:18 -08:00
Juan Tejada f34269d20f [local-sync] Prioritize syncing archive for non Gmail accounts!
Summary:
Previously, we were not pripritizing archive sync when getting folders to sync, causing it to be synced almost last. I believe this was causing the issues regarding archived items coming back, because we would optimistically archive in N1, but the changes wouldn't be reflected in K2's database until we synced the archive, causing the data to become out of sync. If for whatever reason we got a delta for any of those messages before the archive was synced, they would pop back in the inbox because in k2, they were still in the inbox. This was exacerbated by the fact that all syncback tasks would interrupt the loop, so we would reach the archive until very late, making this scenario way more likely.

This still wont guarantee that it wont happen, because we dont do /any/ optimistic updates in K2, so we could still get deltas before we actually sync the folder, but makes the scenario way less likely. This should be properly fixed with the sync scheduler refactor

Test Plan: manual

Reviewers: spang, evan, mark

Reviewed By: mark

Differential Revision: https://phab.nylas.com/D3716
2017-01-16 17:46:24 -08:00
Juan Tejada 3cbd7e0e88 [local-sync] Correctly interrup! (don't wait for interruption) 2017-01-16 17:33:08 -08:00
Juan Tejada 87c5dd49b2 [local-sync] Surface send network errors
Summary: see title

Test Plan: manual

Reviewers: evan, halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D3708
2017-01-16 12:30:56 -08:00
Christine Spang 5cdd82fc62 [local-sync] On Gmail initial sync, prioritize up to 1k inbox UIDs
Summary:
All we do is use the SEARCH X-GM-RAW IMAP extension to find the UIDs
to prioritize at the beginning of initial sync, and download these UIDs
until there are none left. Then we continue downloading All Mail as
usual.

Because of the way we batch via ranges, the most expedient way to
implement this means that all prioritized emails will end up being
downloaded twice (the second time we'll detect that the message exists
and do nothing).

This seems like a worthwhile tradeoff for quick appearance of the
messages in a user's inbox.

Test Plan: manual

Reviewers: evan, juan

Reviewed By: evan, juan

Differential Revision: https://phab.nylas.com/D3706
2017-01-16 10:47:14 -08:00
Evan Morikawa 9986406c9a [local-private] use new action name 2017-01-16 10:30:41 -08:00
Christine Spang bbae3c2155 [local-sync] throw proper error if invalid range passed to FETCH 2017-01-16 06:29:55 -08:00
Juan Tejada f358cc0b14 [local-sync] Fix yahoo sending
Summary: Was missing from in envelope when sending custom message

Test Plan: manual, tested sending from Gmail, Office365, Yahoo, all work

Reviewers: evan, halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D3700
2017-01-15 17:29:00 -08:00
Halla Moore 3e193be099 [local-sync, iso-core] Don't treat inline images as attachments
Summary:
Don't show the attachment icon on threads that only have inline
images. We do this by assuming that inline images have a contentID,
and regular attachments do not. Also updates the way we send
attachments in order to adhere to this standard.

Test Plan: tested manually

Reviewers: spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D3696
2017-01-15 17:08:16 -08:00
Juan Tejada 79a8aa9319 [local-sync] 🎨 renaming 2017-01-15 17:04:01 -08:00
Juan Tejada c22703bd6e [local-sync] Fix imap box status check
Summary:
When syncing folders, we check if the folder needs syncing by checking if it has any new messages via the STATUS command (STATUS returns uidnext, highestmodseq among others, and is cheaper than SELECT)

However, we can't issue a STATUS on a box that is already selected. Previously, if the box was already selected, we would just return it, but this was incorrect because we wouldn't get the latest box values (e.g. uidnext), causing us to think that there were no updates available, and skip syncing folders that actually needed to be synced.

Now, if the box is already selected when getting the status, we have to re select it to refresh the latest values

Test Plan: manual

Reviewers: evan, khamidou, spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D3697
2017-01-15 16:45:16 -08:00
Halla Moore 450f6fde13 [local-sync] Don't run interrupts within the sync-worker serially
Run them at the same time instead, using Promise.all()
2017-01-15 15:53:43 -08:00
Evan Morikawa fa6aec3cee [local-sync] fix lowerbound error where uid set could be zero 2017-01-15 15:35:58 -08:00
Evan Morikawa a7bd1d66b7 [local-sync] decode quoted-printable encoded attachments
Summary:
Needed to stream process quoted-printable attachments
Fixes T7530

Test Plan: manual

Reviewers: juan, spang

Reviewed By: spang

Maniphest Tasks: T7530

Differential Revision: https://phab.nylas.com/D3690
2017-01-15 15:12:18 -08:00
Juan Tejada e2b317d09d [local-sync] Make all syncback tasks interrupt sync so they run fast
Summary:
This commit also lowers the batch size of messages to fetch on folder sync down to 30. This is in order to prevent sync from getting stuck if we queue too many syncback tasks-- given that we only update the range of fetched uids after we've actually fetched and processed messages, if the batch size is too big and we interrupt too often, we might end up never advancing the range and re fetching the same messages over and over.
This also makes the sync loop run faster through all folders in general.

Depends on D3689 to make sure that the batch size actually reflects a message count, i.e. to ensure that we are making /visible/ progress.

Test Plan: manual

Reviewers: spang, khamidou, evan

Reviewed By: evan

Maniphest Tasks: T7477

Differential Revision: https://phab.nylas.com/D3692
2017-01-15 15:07:49 -08:00
Halla Moore 6312c3a06b [iso-core] Use SUPPORTED_PROVIDERS in Account.smtpConfig()
Summary:
Consolidating provider checks to use the same source of truth.
Fixes send issues with some provider types.

Test Plan: tested locally

Reviewers: tomasz

Reviewed By: tomasz

Differential Revision: https://phab.nylas.com/D3694
2017-01-15 14:37:38 -08:00
Christine Spang d91992cdb8 [local-sync] Fetch at least the batch size of messages on each sync iteration
Summary:
Because we optimistically fetch UIDs by expanding a range without looking
at the actual UIDs in the inbox and the actual space of UIDs with messages
attached may be sparse due to message moves, we need to track how many
messages we actually download during a range expansion and continue
expanding the range if we haven't downloaded enough messages.

If we reach a large gap where we download no messages at all during a batch, we
pause and check the actual UID list for the folder for the next UID to
download, as otherwise we may spin indefinitely fetching UIDs that don't exist.
(Example: my "Deleted Items" folder had about 300k worth of empty UIDs between
a very small UID and a very large UID. With the new system, this registers as a
completed sync within a single iteration as soon as sync hits the gap.)

Test Plan: manual

Reviewers: juan, evan

Reviewed By: juan, evan

Differential Revision: https://phab.nylas.com/D3689
2017-01-15 14:28:24 -08:00
Karim Hamidou 078b015202 Fix unit test path. 2017-01-15 14:09:05 -08:00
Karim Hamidou aadc322ef4 [local-sync] Back off exponentially when getting a sync error
Summary:
This patch changes the sync worker to back off exponentially when there is an issue syncing an account. This has two goals:
- first, it's a bit dangerous to retry immediately. We don't want hundreds of thousands of machines trying to refresh tokens unsuccessfully because our service is struggling.
- second, it's nicer on the CPU to wait a bit between retries.

Currently, we sleep for at most 2 minutes, with some random jitter added.

Test Plan: Tested manually, stared at the code a long time.

Reviewers: evan, juan

Reviewed By: evan, juan

Differential Revision: https://phab.nylas.com/D3684
2017-01-15 11:57:47 -08:00
Halla Moore 77ad25af24 [local-sync] Stop sync worker before deleting account database
Summary:
Various errors are thrown when the sync worker tries accessing
a database that we've already deleted, so make sure the sync
worker has been stopped before we remove the database. This diff
involves modifying `Interruptible` so that `interrupt()` returns
a promise that resolves once the interrupt has been completed.

Addresses T7472

Test Plan: manual

Reviewers: evan, juan

Reviewed By: evan, juan

Differential Revision: https://phab.nylas.com/D3679
2017-01-15 10:33:12 -08:00
Christine Spang f06ba78d8a [local-sync] Update comment on _processExistingMessage 2017-01-15 10:06:52 -08:00
Christine Spang 9e426419c5 [local-sync] Correctly add references when processing existing messages 2017-01-15 10:04:29 -08:00
Evan Morikawa 9381b51746 [local-sync] fix fetch unsynced messages logic looking for min 2017-01-14 18:36:51 -08:00
Juan Tejada 50f00e5174 [local-sync] Fix order of execution in fetch-messages-in-folder 2017-01-14 17:43:42 -08:00
Halla Moore c5453ca21b [iso-core] Let auth methods accept all supported providers
Summary: Treat any that aren't gmail or office365 as standard imap

Test Plan: manual

Reviewers: juan, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3686
2017-01-14 17:35:09 -08:00
Evan Morikawa ae981af646 [local-sync] fix performance of attribute changes
Summary:
On MG's machine this function is EXTREMELY non performant and causes
things like archive to lock up when the console is running here for some
reason. Not entirely sure exactly what's causing it, but there were some
simple DB cleanups that will make it faster for large queries.

There's likely other things involved since the sequelize DB being locked
up shouldn't affect the peformLocal of the edgehill db for things like
archive. Still looking into that

Test Plan: manual

Reviewers: juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3683
2017-01-14 17:27:15 -08:00
Juan Tejada 814581f3d0 [local-sync] 🎨 2017-01-14 17:23:43 -08:00
Juan Tejada fc433dd6c6 [local-sync] Await instead of yielding when fetching the very first batch of messages in folder sync 2017-01-14 16:32:14 -08:00
Juan Tejada f4f78f4449 [local-sync] Minor syntax fix 2017-01-14 16:22:53 -08:00
Juan Tejada 7b2e27b87b [local-sync] Skip unecessary folder syncs
Summary:
Before trying to sync a folder, check if we actually need to do so. This will prevent us from doing unnecessary work that slows down the sync loop (like performing SELECT commands)

We will perform a folder sync if any of the following are true
- The folder hasn't been completely synced
- There are new messages (using imap STATUS command)
- There are attribute changes indicated via highestmodseq (using imap STATUS command)
- If server doesn't support highestmodseq, it has passed enough time since we last ran an attribute scan on the folder.

Addresses T7513

Test Plan: manual

Reviewers: evan, halla, spang

Reviewed By: halla, spang

Differential Revision: https://phab.nylas.com/D3675
2017-01-14 15:56:17 -08:00
Christine Spang ff9c2fd57c [local-sync] grr phab lost this update 2017-01-14 14:57:22 -08:00