Commit graph

5816 commits

Author SHA1 Message Date
Halla Moore
379edfe0b8 [*] MessageFactory -> MessageUtils, SendUtils -> ModelUtils
Summary: Rename some util files

Test Plan: manual

Reviewers: evan, mark, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4364
2017-04-11 10:49:30 -07:00
Halla Moore
f82c72aa88 [client-sync] Don't report stuck sync processes to Sentry
Summary: Just to mixpanel. Addresses T8095

Test Plan: Meh

Reviewers: mark, spang, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4403
2017-04-11 10:45:09 -07:00
Juan Tejada
f6f8f16e25 [client-app] Update changelog 2017-04-11 10:37:13 -07:00
Nylas Coffee Machine
a412580cad bump(version): 2.0.9 2017-04-11 08:12:13 -07:00
Halla Moore
15c3e328fc [client-sync] Poll for gmail attribute changes from _mailListenerIMAPConn
Summary:
Normally, the imap connection emits an 'update' event when there are any
flag changes. Gmail, however, only emits the 'update' event when there
are new or removed messages. If there's a fairly large batch size being
synced in the sync loop, we don't want to wait until the end of that to
check if the highestmodseq has changed. To get around this, this diff
updates the new mail listener connection to periodically poll for the
highestmodseq and trigger the same update handler when changes are
detected.

Test Plan: manual

Reviewers: evan, spang, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4397
2017-04-10 15:46:02 -07:00
Halla Moore
c6db6c3ae2 [client-app] Re-add imap to the onboarding accounts page
Summary:
Put the IMAP option back on the provider selection page within
onboarding. Also remove the 'hidden' property on account types since I
don't think we'll be using it for any other providers, and remove the
'Add Custom IMAP Account' from the application menu.

Addresses T8097

Test Plan: Manual

Reviewers: spang, mark, juan

Reviewed By: mark, juan

Differential Revision: https://phab.nylas.com/D4402
2017-04-10 15:44:26 -07:00
Juan Tejada
07d576d5f4 [client-app] Fix delta streaming connection retries
Summary:
Previously, every time our NylasLongConnection changed its status to
`connected`, we would reset our backoff scheduler, assuming that
every time our status changed to `connected` it meant that we would not
receive any errors.

However, if our backend is down or overloaded (or returning 401s like in
our recent outage), we would do the following:

1. Establish connection, change status to `connected`
2. Reset backoff scheduler
3. Receive error
4. Close connection
5. Schedule next retry using backoff scheduler. Delay will be 1s or less, because scheduler was reset in 2.
6. Retry after delay,
7. Repeat from 1., ad infinitum

This caused us to consistantly hammer our servers and render our
exponential backoff useless.

This commit makes it so we conly reset the backoff scheduler when we
actually receive deltas, and increments the max backoff to 10 minutes

Test Plan:
run against local n1Cloud, manually return 401s from the api, verify that we
didn't backoff at all before this diff, verify that we backoff correctly after
this diff

Reviewers: mark, spang, halla

Reviewed By: mark, spang, halla

Differential Revision: https://phab.nylas.com/D4401
2017-04-10 15:08:48 -07:00
Halla Moore
e8957b787e [isomorphic-core] Handle weird MIME edge case with @ symbol
Summary:
Something weird happens with the mime building when the participant name
has an @ symbol in it (e.g. a name and email of hello@gmail.com turns
into 'hello@ <gmail.com hello@gmail.com>'), so replace it with
whitespace.

Test Plan: manual

Reviewers: evan, mark, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4363
2017-04-10 12:40:22 -07:00
Mark Hahnenberg
be4b0d773c [client-app] Add benchmark mode
Summary:
There are some settings that apply to dev and prod modes that we don't want
to use while benchmarking. E.g. the folder where we store messages,
whether we generate long stack traces in our bluebird promises, etc.
This diff adds a benchmark mode so that we can change these settings to
something that works better for benchmarking.

Test Plan: Run locally, verify it works

Reviewers: evan, juan, spang

Reviewed By: juan, spang

Differential Revision: https://phab.nylas.com/D4374
2017-04-07 15:33:57 -07:00
Mark Hahnenberg
c808438ee9 [client-sync] Transaction-ify processMessage
Summary:
We do lots of writes while processing a single message. To reduce the
write churn from lots of mini-transactions, this diff threads one
overarching transaction to everything in processMessage.

Test Plan: Run locally

Reviewers: spang, evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4394
2017-04-07 15:09:25 -07:00
Mark Hahnenberg
b323e38f7c [client-sync] Increase sqlite page_size and cache_size
Summary:
The defaults are a little low. edgehill.db already does this. There's no
obvious effect on smaller dbs, but we should at least make the two
consistent.

Test Plan: Run locally

Reviewers: spang, evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4395
2017-04-07 13:08:13 -07:00
Juan Tejada
3befaf2d6e [client-app] Update changelog 2017-04-07 12:14:51 -07:00
Nylas Coffee Machine
a30e63c91a bump(version): 2.0.8 2017-04-07 11:42:30 -07:00
Juan Tejada
763010326e [client-app] Ensure SyncbackMetadataTask dependency to prevent version conflicts
Summary:
Previously, SyncbackMetadataTasks for the same plugin did not depend on
each other, so they could try to modify the same piece of metadata
concurrently, which would likely produce version conflict errors.

This was happening with send reminders because we were queuing 2 of
these tasks back to back upn send success. Adding this dependency fixes
the version conflict errors when setting the metadata

Test Plan: manual

Reviewers: evan, halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D4393
2017-04-07 11:41:21 -07:00
Juan Tejada
17ade153f8 bump(version): 2.0.7 2017-04-06 19:32:30 -07:00
Karim Hamidou
01318370d4 [cloud-workers] [send-later] Fix send later token expiration
Summary:
Fixes T8076. So, the Gmail OAuth API returns us an expiration time in milliseconds, but we convert it to seconds everywhere, except in the cloud-api. This caused us to never refresh tokens because we'd be comparing seconds to milliseconds.

Fix this by converting the `expiry_date` from our credentials to seconds.

Test Plan: Tested manually. Will do more testing tomorrow, after my credentials expire.

Reviewers: evan, juan

Reviewed By: evan, juan

Maniphest Tasks: T8076

Differential Revision: https://phab.nylas.com/D4390
2017-04-06 18:57:16 -07:00
Evan Morikawa
236c71ea6a [iso-core, cloud-*] catch JSON parse errors in DB columns
Summary:
MySQL would insert bad JSON due to emoji errors. This would cause JSON
parse errors and prevent us from ever reading objects from the DB. We now
catch and return null instead with an error which will at least let the
query complete to let us isolate and deal with the malformed obeject (like
destroy it)

Test Plan: manual

Reviewers: spang, halla, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4389
2017-04-06 18:22:29 -07:00
Christine Spang
d09312c27f [cloud-*] Switch MySQL charset to utf8mb4
Summary:
We need this in order to be able to store 4-byte emoji and other
extended unicode characters in the database.

Test Plan: deploy to staging - need the change that sets the connection charset on staging :(

Reviewers: khamidou, juan, evan

Reviewed By: juan, evan

Differential Revision: https://phab.nylas.com/D4387
2017-04-06 16:25:09 -07:00
Nylas Coffee Machine
3074ee2912 bump(version): 2.0.6 2017-04-06 16:12:23 -07:00
Juan Tejada
10d199f389 [dev] Ensure daily script grabs current version after pulling latest changes
Summary: see title

Test Plan: manual

Reviewers: halla, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D4378
2017-04-06 14:34:31 -07:00
Juan Tejada
5880f2ace7 [client-app] Always fetch and update identity regardless of env
Summary:
This was annoying because we wouldn't fetch the identity when running `local`
env which produced all sorts of errors

Test Plan: manual

Reviewers: evan, spang, halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D4380
2017-04-06 13:28:09 -07:00
Nylas Coffee Machine
50c0329591 bump(version): 2.0.5 2017-04-06 11:56:10 -07:00
Evan Morikawa
2ee67d2a7a [iso-core, cloud-*] move generate XOAuth2 & fix first auth usage
Summary:
We were not generating the XOAuth2 token on the first go when connecting
accounts. This would cause the first send later to fail. Since this
happens in ISO core, we needed to move the function out of cloud-core and
import from other places that use it

Test Plan: Manual

Reviewers: khamidou, juan, halla, spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D4375
2017-04-06 11:53:39 -07:00
Juan Tejada
8633058488 [isomorphic-core] Ensure IMAPConnPool uses updated account credentials
Summary:
Previously, we would initialize an `AccountPool` with an account
instance and reference the account as an instance variable. Then,
every time we created a connection, we used the account we had a
reference to obtain the credentials.

However, if the credentials on the account changed, e.g. when we refresh
the access token for gmail accounts, the connection pool would never
realize that the account had new credentials and always try to create
the connection with the old credentials, causing the sync worker to
enter an error loop of `Invalid credentials` errors

To fix this, we could reload the account every time we try to create a
connection, but it seems that the fresh credentials should just be
passed every time we create connections without the pool having to be
aware of potential changes under the rug.

This commit makes it so the pool no longer holds a reference to the
account, but rather it is passed in every time we check out a connection

Test Plan: manual. how did this even work before? :(

Reviewers: evan, halla, spang, mark

Reviewed By: mark

Differential Revision: https://phab.nylas.com/D4370
2017-04-06 10:32:42 -07:00
Nylas Coffee Machine
bb03d3c78e bump(version): 2.0.4 2017-04-05 17:54:09 -07:00
Nylas Coffee Machine
68b5550d54 bump(version): 2.0.3 2017-04-05 17:51:11 -07:00
Christine Spang
cfe2971c2e [*] Revamp SSL options (including user-facing)
Summary:
Previously, the generic IMAP auth screen presented one security option to
users: "Require SSL". This was ambiguous and difficult to translate into
the correct security options behind the scenes, causing confusion and problems
connecting some accounts.

This patch does the following:
* Separates security settings for IMAP and SMTP, as these different protocols
  may also require different SSL/TLS settings

* Reworks the generic IMAP auth page to allow specifying security settings
  with higher fidelity. We looked at various different email apps and decided
  that the best solution to this problem was to allow more detailed
  specification of security settings and to ease the burden of more options
  by having sane defaults that work correctly in the majority of cases.
  This new screen allows users to pick from "SSL / TLS", "STARTTLS", or "none"
  for the security settings for a protocol, and also to instruct us that
  they're OK with us using known insecure SSL settings to connect to their
  server by checking a checkbox.

  We default to port 993 / SSL/TLS for IMAP and port 587 / STARTTLS for SMTP.
  These are the most common settings for providers these days and will work
  for most folks.

* Significantly tightens our default security. Now that we can allow folks to
  opt-in to bad security, by default we should protect folks as best we can.

* Removes some now-unnecessary jank like specifying the SSLv3 "cipher"
  in some custom SMTP configs. I don't think this was actually necessary
  as SSLv3 is a protocol and not a valid cipher, but these custom
  configs may have been necessary because of how the ssl_required flag was
  linked between IMAP and SMTP before (and thus to specify different
  settings for SMTP you'd have to override the SMTP config).

* Removes hard-coding of Gmail & Office365 settings in several
  locations. (This was a major headache while working on the patch.)

This depends on version 2.0.1 of imap-provider-settings, which has major
breaking changes from version 1.0. See commit for more info:
9851054f91

Among other things, I did a serious audit of the settings in this file and
"upgraded" a few servers which weren't using the SSL-enabled ports for their
provider to the secure ones. Hurray for nmap and openssl.

Test Plan: manual

Reviewers: evan, mark, juan, halla

Reviewed By: juan, halla

Differential Revision: https://phab.nylas.com/D4316
2017-04-05 17:49:43 -07:00
Halla Moore
b5a19dd3e3 bump(version): 2.0.2 2017-04-05 16:04:42 -07:00
Evan Morikawa
4904a9ae85 [client-app] Updates to feature limiting
This is a squash of a bunch of commits releated to feature limiting.
Includes making the upgrade path seamless by accessing billing through a
WebView and making sure that the feature is enabled after upgrading, or
canceled if the upgrade path is stopped by the user in any way.

Included diffs
---------------
Differential Revision: https://phab.nylas.com/D4078
Differential Revision: https://phab.nylas.com/D4136
Differential Revision: https://phab.nylas.com/D4137
Differential Revision: https://phab.nylas.com/D4143
Differential Revision: https://phab.nylas.com/D4147
Differential Revision: https://phab.nylas.com/D4171
Differential Revision: https://phab.nylas.com/D4173
2017-04-05 16:02:53 -07:00
Mark Hahnenberg
965946724b [client-app] Some final benchmark fixes
Summary: See title

Test Plan: Run on the coffee machine

Reviewers: juan, evan, spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D4368
2017-04-05 15:59:17 -07:00
Halla Moore
d9a9d66f5f [client-sync] Fix syncWorker arg
Summary: It needs to be wrapped in an object.

Test Plan: manual

Reviewers: evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4367
2017-04-05 15:44:13 -07:00
Juan Tejada
fc7856b4d0 [client-sync] Fix runtime error when getting sync activity for account
Summary:
Make sure SyncActivity.getLastSyncActivityForAccount returns a sensible
default to prevent runtime errors when the SyncProcessManager uses it

Test Plan: manual

Reviewers: evan, halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D4366
2017-04-05 14:10:01 -07:00
Nylas Coffee Machine
3653a66be2 bump(version): 2.0.1 2017-04-05 11:21:56 -07:00
Juan Tejada
f708e3af0b [iso-core] IMAPConnectionPool now correctly disposes connections onDone
Summary:
Previously, calling the IMAPConnectionPool `onDone` callback when checking out a connection would just return the connection to the pool without re-establishing it the next time we checked it out of the pool.

When the `onDone` callback was called in an error scenario, this had the unintended consequence of returning a bad connection to the pool, and then re-using this bad connection. The only scenario when the connection was re-established was if `onConnected` threw an error, but if the connection was kept open outside the scope of onConnected, this logic would never run.

To make things simpler and more consistent, `onDone` will always destroy all connections and connections will always be re-established from scratch every time they are checked out of the pool. The pool acts more as a limit to the number of connections per account, than an actual shared pool of connections.

Test Plan: manual

Reviewers: evan, spang, halla, mark

Reviewed By: halla, mark

Differential Revision: https://phab.nylas.com/D4360
2017-04-05 11:10:41 -07:00
Juan Tejada
5269c11ee0 [client-sync] Prevent IMAP connection leaking in sync worker
Summary:
When the sync worker got stuck inside `performSync`, we would never
release the connection back to the connection pool. We would then try to
start a new sync worker, and it would claim a new connection. If this
happened enough times, we would eventually exhaust the whole pool and
keep the sync worker blocked forever.

Test Plan: manual

Reviewers: evan, spang, halla, mark

Reviewed By: halla, mark

Differential Revision: https://phab.nylas.com/D4359
2017-04-05 11:08:20 -07:00
Evan Morikawa
78a15700c8 [cloud-*] remove old SignalFX reporter & add docs
Summary:
This removes the old SignalFX reporter that we weren't using
Adding documentation around logging

Test Plan: manual

Reviewers: juan, halla, spang

Reviewed By: juan, halla, spang

Differential Revision: https://phab.nylas.com/D4341
2017-04-05 10:51:38 -07:00
Evan Morikawa
08a697e4c9 [cloud-*] log errors according to bunyan specs
Summary:
According to bunyan, https://github.com/trentm/node-bunyan, errors are
treated special. They either need to be the first argument of a log, or
they need to be attached to the special `err` (NOT `error`) key of the
logger object.

Test Plan: manual

Reviewers: halla, juan, spang

Reviewed By: halla, juan, spang

Differential Revision: https://phab.nylas.com/D4337
2017-04-05 10:47:56 -07:00
Juan Tejada
a3072f6b04 [client-sync] 🎨 logging 2017-04-05 09:22:00 -07:00
Juan Tejada
d75a5ee243 [client-app] Add better logging when restarting stuck sync worker
Summary: see title

Test Plan: manual

Reviewers: halla, mark, spang, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D4358
2017-04-04 21:37:55 -07:00
Juan Tejada
f8d84a76e2 [iso-core] Remove unused vars from IMAPConnectionPool
Summary: see title

Test Plan: manual

Reviewers: mark, halla, spang, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D4357
2017-04-04 21:34:20 -07:00
Juan Tejada
2949ecffd5 [client-sync] 🎨 sync worker
Summary: Just renaming some stuff

Test Plan: manual

Reviewers: mark, spang, evan, halla

Reviewed By: spang, evan, halla

Differential Revision: https://phab.nylas.com/D4342
2017-04-04 17:53:13 -07:00
Halla Moore
5c8915f144 [client-app] Fix the unapply transformation for overlaid components
Summary:
The unapply transformation was incorrectly passing the props parameter
to buildAnchorTag() as a string, even though it expects an object.
buildAnchorTag() goes on to stringify this parameter, which causes extra
surrounding quotations and escape slashes in front of all the other
quotations. The unapply transform is applied on several re-renders,
which causes the number of escape characters to unboundedly increase.
Firstly, this was causing inline images to not appear properly in the
draft, and secondly, the large number of escape characters was making
the draft body large enough to make the app unresponsive.

This generally wasn't an issue because the unapply transformation is
only applied when there has been an apply transformation, and this is
triggered by the SyncbackDraftTask. This task was previously only queued
upon send, in which case the user never re-opens the draft. Now,
enabling send-later on a draft will queue the task, and these issues
appear if the user re-opens the draft for editing.

This diff makes it so that the unapply transform passes in the props as
an object, like the buildAnchorTag function expects.

Test Plan: manual

Reviewers: evan, juan

Reviewed By: evan, juan

Differential Revision: https://phab.nylas.com/D4348
2017-04-04 16:43:05 -07:00
Halla Moore
2b39af5aba [client-app] Allow the packageMigrationManager to re-enable packages
Summary:
Remove packages from the disabled package list when they have been
migrated to be enabled by default.

Also, the last migration would not have worked properly anyways because
the daily channel was already on 1.0.56, and we check for greater-than
rather than equal-to. Bump that version to match the next update.

Test Plan: manual

Reviewers: juan, evan

Reviewed By: juan, evan

Differential Revision: https://phab.nylas.com/D4335
2017-04-04 16:36:39 -07:00
Juan Tejada
7d99567855 [client-app] Make sure to fetch latest update url every time we check for updates
Summary:
Previously, is you signed out of your NylasID, you would not pick up new
updates because we never updated the autoupdater url to use the new id.

Test Plan: manual

Reviewers: mark, halla, evan

Reviewed By: halla, evan

Differential Revision: https://phab.nylas.com/D4346
2017-04-04 16:30:52 -07:00
Juan Tejada
d1b81b6afe [client-app] Prevent from making any requests when NylasID isn't present
Summary:
Sometimes, when logging out of your NylasID and restarting the app, we
would continue making some requests from the worker window that required a
NylasID. This would make the app enter a restart loop and become
completely unresponsive because when we made a request without a
NylasID, we would force the user to log out and restart the app, and
then we would again make the requests without the id, ad infinitum.

To fix this, we make sure we have a NylasID before making any requests
that require it

Test Plan: manual

Reviewers: halla, evan

Reviewed By: halla, evan

Differential Revision: https://phab.nylas.com/D4344
2017-04-04 16:26:32 -07:00
Christine Spang
2095f0907e [scripts] Batch upload each row
Makes the script like 4x faster.
2017-04-04 15:24:01 -07:00
Mark Hahnenberg
00e1df8491 [client-app] Add script to run benchmarks once per day at specified time
Summary: See title

Test Plan: Run locally

Reviewers: juan, evan, spang

Reviewed By: spang

Differential Revision: https://phab.nylas.com/D4345
2017-04-04 15:07:08 -07:00
Christine Spang
bd933f1bc0 [scripts] Add script to upload benchmark data to Google Sheets
Test Plan: ran locally (need client secret for this)

Reviewers: mark

Reviewed By: mark

Differential Revision: https://phab.nylas.com/D4343
2017-04-04 15:04:03 -07:00
Mark Hahnenberg
4eef086e3e [client-app] Fix benchmark script
Summary:
We weren't changing into the directory of the git repository, so running
the script from anywhere but inside the git repo didn't work. We also
weren't pulling before checking for the latest commit. Now we do both of
these things.

Test Plan: Run locally

Reviewers: spang, juan, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D4339
2017-04-04 13:28:46 -07:00
Juan Tejada
a461490e41 [client-sync] Ensure we use refreshed access token for mail listener conn
Summary:
Before this commit, we would just establish a single mail listener imap
connection and never check if we needed to re-connect it due to an
expired access token. Even though we correctly refreshed the access
token at the beggining of each sync loop (hidden under
`ensureSMTPConnection`), we would never re-establish the mail listener
connection with the new access token.

This would cause the app to enter an `Invalid Credentials` error loop in
the sync loop, preventing from syncing any mail at all (T8064)

Test Plan: manual

Reviewers: spang, halla, mark, evan

Reviewed By: mark, evan

Differential Revision: https://phab.nylas.com/D4338
2017-04-04 13:27:43 -07:00