Summary:
Previously, SyncbackMetadataTasks for the same plugin did not depend on
each other, so they could try to modify the same piece of metadata
concurrently, which would likely produce version conflict errors.
This was happening with send reminders because we were queuing 2 of
these tasks back to back upn send success. Adding this dependency fixes
the version conflict errors when setting the metadata
Test Plan: manual
Reviewers: evan, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4393
Summary:
This was annoying because we wouldn't fetch the identity when running `local`
env which produced all sorts of errors
Test Plan: manual
Reviewers: evan, spang, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4380
Summary:
Previously, the generic IMAP auth screen presented one security option to
users: "Require SSL". This was ambiguous and difficult to translate into
the correct security options behind the scenes, causing confusion and problems
connecting some accounts.
This patch does the following:
* Separates security settings for IMAP and SMTP, as these different protocols
may also require different SSL/TLS settings
* Reworks the generic IMAP auth page to allow specifying security settings
with higher fidelity. We looked at various different email apps and decided
that the best solution to this problem was to allow more detailed
specification of security settings and to ease the burden of more options
by having sane defaults that work correctly in the majority of cases.
This new screen allows users to pick from "SSL / TLS", "STARTTLS", or "none"
for the security settings for a protocol, and also to instruct us that
they're OK with us using known insecure SSL settings to connect to their
server by checking a checkbox.
We default to port 993 / SSL/TLS for IMAP and port 587 / STARTTLS for SMTP.
These are the most common settings for providers these days and will work
for most folks.
* Significantly tightens our default security. Now that we can allow folks to
opt-in to bad security, by default we should protect folks as best we can.
* Removes some now-unnecessary jank like specifying the SSLv3 "cipher"
in some custom SMTP configs. I don't think this was actually necessary
as SSLv3 is a protocol and not a valid cipher, but these custom
configs may have been necessary because of how the ssl_required flag was
linked between IMAP and SMTP before (and thus to specify different
settings for SMTP you'd have to override the SMTP config).
* Removes hard-coding of Gmail & Office365 settings in several
locations. (This was a major headache while working on the patch.)
This depends on version 2.0.1 of imap-provider-settings, which has major
breaking changes from version 1.0. See commit for more info:
9851054f91
Among other things, I did a serious audit of the settings in this file and
"upgraded" a few servers which weren't using the SSL-enabled ports for their
provider to the secure ones. Hurray for nmap and openssl.
Test Plan: manual
Reviewers: evan, mark, juan, halla
Reviewed By: juan, halla
Differential Revision: https://phab.nylas.com/D4316
Summary:
Make sure SyncActivity.getLastSyncActivityForAccount returns a sensible
default to prevent runtime errors when the SyncProcessManager uses it
Test Plan: manual
Reviewers: evan, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4366
Summary:
The unapply transformation was incorrectly passing the props parameter
to buildAnchorTag() as a string, even though it expects an object.
buildAnchorTag() goes on to stringify this parameter, which causes extra
surrounding quotations and escape slashes in front of all the other
quotations. The unapply transform is applied on several re-renders,
which causes the number of escape characters to unboundedly increase.
Firstly, this was causing inline images to not appear properly in the
draft, and secondly, the large number of escape characters was making
the draft body large enough to make the app unresponsive.
This generally wasn't an issue because the unapply transformation is
only applied when there has been an apply transformation, and this is
triggered by the SyncbackDraftTask. This task was previously only queued
upon send, in which case the user never re-opens the draft. Now,
enabling send-later on a draft will queue the task, and these issues
appear if the user re-opens the draft for editing.
This diff makes it so that the unapply transform passes in the props as
an object, like the buildAnchorTag function expects.
Test Plan: manual
Reviewers: evan, juan
Reviewed By: evan, juan
Differential Revision: https://phab.nylas.com/D4348
Summary:
Remove packages from the disabled package list when they have been
migrated to be enabled by default.
Also, the last migration would not have worked properly anyways because
the daily channel was already on 1.0.56, and we check for greater-than
rather than equal-to. Bump that version to match the next update.
Test Plan: manual
Reviewers: juan, evan
Reviewed By: juan, evan
Differential Revision: https://phab.nylas.com/D4335
Summary:
Previously, is you signed out of your NylasID, you would not pick up new
updates because we never updated the autoupdater url to use the new id.
Test Plan: manual
Reviewers: mark, halla, evan
Reviewed By: halla, evan
Differential Revision: https://phab.nylas.com/D4346
Summary:
Sometimes, when logging out of your NylasID and restarting the app, we
would continue making some requests from the worker window that required a
NylasID. This would make the app enter a restart loop and become
completely unresponsive because when we made a request without a
NylasID, we would force the user to log out and restart the app, and
then we would again make the requests without the id, ad infinitum.
To fix this, we make sure we have a NylasID before making any requests
that require it
Test Plan: manual
Reviewers: halla, evan
Reviewed By: halla, evan
Differential Revision: https://phab.nylas.com/D4344
Summary:
Previously we would always search all mail. Now, if the user has focused
a particular folder we will limit our search to that folder. The inbox
is an exception--it will always search all mail unless the user
explicitly uses an "in:" clause.
Test Plan: Run locally, verify that searching folders returns the correct results.
Reviewers: evan, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4328
Summary:
Before this patch, the IdentityStore would initialize in our empty hot
window. However, hot windows don't receive any `action-bridge-message`s,
which include DB updates. Since the hot window loads first, it was with a
stale verison of the Identity. The main window fetches a fresh identity,
but that fresh update failed to get to new composers because the hot
window wasn't listening to changes to the DB.
This makes it such that the IdentityStore properly boots up when the
window props change.
Test Plan: manual
Reviewers: halla, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4327
Summary:
This adds feature limit modals (graphics pending) for reminders and send
later
Test Plan: manual
Reviewers: juan, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4322
Summary: This shaves off ~150ms trying to issue this query
Test Plan: manual
Reviewers: evan, spang, mark
Reviewed By: mark
Differential Revision: https://phab.nylas.com/D4317
Summary:
Bodies that already exist on K2 shouldn't be overridden.
See the comment for how this caused open/link tracking to fail in certain
cases when a lot of messages were being sent
Test Plan: manual
Reviewers: halla, mark, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4319
Summary: It was really slow, and we no longer need it.
Test Plan: Run locally, verify that we no longer do that super slow query
Reviewers: evan, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4312
Summary:
Removes some VERY long running `ANLAYZE` queries. Was taking up to 50
seconds on my 9GB database on every boot
https://sqlite.org/pragma.html#pragma_optimize
Test Plan:
I tested to make sure the app still quits quickly. It does. The SQLite
docs also say this should be fast.
Reviewers: halla, spang, mark, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4314
Summary:
This improves window open times by about 70ms
We would spend a very long time in the backend browser process building a
new hot window when we didn't need to do that until much later
Test Plan: manual
Reviewers: halla, mark, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4311
Summary:
This prevents us from hitting the endpoint before the database for that
account is initialized and it's actually syncing.
Before this commit, we would throw errors upon adding an account
Test Plan: manually add, remove accounts, run benchmark script
Reviewers: mark, halla, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4305
Summary:
Creates a new `DatabaseStore.write` method alongside
`DatabaseStore.inTransaction` that doesn't always wrap trivial queries in
a transaction. SQLite does this automatically.
Test Plan: manual
Reviewers: spang, mark, halla, juan
Reviewed By: halla, juan
Differential Revision: https://phab.nylas.com/D4303
Summary: Renamed in prep for next few diffs
Test Plan: manual
Reviewers: juan, mark, spang, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4302
Summary:
Use the wonderful `debug` library instead of all our random flags.
You can now do things like: `DEBUG="app:*,sync:*" npm start` to print out
everything.
Test Plan: manual
Reviewers: mark, halla, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4297
Summary: See title
Test Plan: Run locally, verify that attachments are returned
Reviewers: spang, evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4293
Summary:
Every persistModel would trigger a large number of downstream updates.
These weren't necessary for thread indexing and causing a lot of
unnecessary DB thrashing. This adds a `silent` flag to `persistModel` and
its ilk that just does the write.
significantly improve performance, and also contribute to T8046
Test Plan: manual
Reviewers: spang, mark, halla, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4295
Summary:
This adds better logging to the DB
You can use `ENABLE_SEQUELIZE_DEBUG_LOGGING=true` and
`ENABLE_RXDB_DEBUG_LOGGING=true` to spit out the raw queries of both DBs.
Test Plan: manual
Reviewers: mark, halla, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4294
Summary:
We would mark modified threads for reindexing, the thread search indexer
would reindex them, which would trigger a notification that the thread
had been modified, so we would mark the thread for reindexing, ...
Now we keep track in memory of which threads we've marked for reindexing
so we avoid re-marking them when the update notification arrives later.
Test Plan:
Run locally, verify same threads aren't getting continuously
reindexed and that the in-memory set doesn't grow unboundedly
Reviewers: spang, juan, evan
Reviewed By: juan, evan
Differential Revision: https://phab.nylas.com/D4289
Summary:
This commit makes a few changes to how we log database queries:
- We log queries in 2 different places now:
- When `DatabaseStore._executeLocally` takes longer than 100ms. This might be due to the actual db operation taking long, or it might be due to our retry logic, or potentially other factors.
- When the actual raw database operation takes more than 100ms.
- When raw database queries take more than 100ms, we log out the query plan if it is a `SELECT` query and we are in dev mode.
- If `DEBUG_TO_LOG` is true, we always log all queries (including background queries)
Test Plan: manual
Reviewers: mark, halla, evan
Reviewed By: halla, evan
Differential Revision: https://phab.nylas.com/D4288
Summary:
- Rename `databaseAgent` usage to `background`, in the code and in the logs, which I
believe is less confusing. Often times found myself asking what was this agent thing in
the logs.
- Make messaging more explicit
Test Plan: manual
Reviewers: evan, mark, halla
Reviewed By: mark, halla
Differential Revision: https://phab.nylas.com/D4285
Summary:
This commit makes it so that when we retry database queries after
encountering a db locked error, we back off for at most 500ms, instead
of the current max of 3s. This meant that a single query could
potentially take 3+ seconds and block the transaction queue for that long.
Test Plan: manual
Reviewers: evan, mark, halla
Reviewed By: mark, halla
Differential Revision: https://phab.nylas.com/D4283
Summary:
After introducing retry logic, we would always defer db queries to the
next tick even if there was no retry delay present.
This would cause almost all db queries take more than 100ms even if the
actual db operation was really fast. This affected app performance
negatively
Test Plan: manual
Reviewers: evan, halla, mark
Reviewed By: halla, mark
Differential Revision: https://phab.nylas.com/D4282
Summary:
The problem actually was that we weren't applying the transform for Gmail
messages sent to 1 recipient
Test Plan: manual
Reviewers: halla, juan
Reviewed By: halla, juan
Differential Revision: https://phab.nylas.com/D4291
Summary:
We weren't stripping link and open items from messages we sent properly.
While we stripped it from the first outgoing message, when the sync loop
came back around it would have them back in. This adds it in the message
processor to extra ensure that ANYTHING that comes from us (or one of our
aliases) gets the open/link tracking stripped
Test Plan: New tests
Reviewers: juan, halla
Reviewed By: juan, halla
Differential Revision: https://phab.nylas.com/D4287
Summary:
Prior to this diff, the "in:" search query syntax didn't work for IMAP.
This diff implements "in:" by changing the IMAP search backend to take
folder context into account and emit the appropriate queries for each
folder. Queries that include "in:foo" will replace the corresponding AST
nodes with 'ALL' or 'NOT ALL' depending on whether or not the current folder
is "foo". We also now filter which folders we search based on which
folders are referenced in the query.
Test Plan: Run locally, verify that in: works quickly
Reviewers: evan, juan, spang
Reviewed By: juan, spang
Differential Revision: https://phab.nylas.com/D4284
This commit is the same as 7d409f5.
This commit was accidentally reverted in 95ec679, because I (juan) was being
dumb. So I'm re-committing here.
Summary:
We weren't doing that, so we would have accurate search index info for
threads when they were, e.g., moved to another folder.
Test Plan: Run locally
Reviewers: spang, evan, juan
Reviewed By: evan, juan
Differential Revision: https://phab.nylas.com/D4260
Summary:
By default, the 'local'/'development' environment assumes that all services
are running locally. This makes sense.
However, if you're not making changes to auxiliary services such as
billing.nylas.com and hence do not have them running locally, firing up
Nylas Mail with the 'local' env will not work. It's a bit of a pain to
get a local billing service running locally, especially if you're not
making any changes to that service. To ease development, change the
default billing URL during development to staging, and allow manually
overriding the billing server URL to point to localhost in the case
that you're making changes to the billing service also.
To override the billing URL for local development, run like this:
npm run cloud -- --env billing_local
BILLING_URL=http://billing.lvh.me:5555 npm start
This is both a problem that I've run into during development as well as
(my hunch) why Karim accidentally landed a global override of the
billing server URL.
Test Plan: manual
Reviewers: evan, khamidou, halla, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4202
Summary:
This prevented the /health endpoint from being available in the
local-api
Test Plan: manual
Reviewers: mark, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4280
Summary:
This patch will prevent users from being able to connect accounts which sync
mail but fail to send.
This commit includes a couple pieces:
* Adds a call to nodemailer's `verify()` function in the /auth endpoint
* Adds Error object conversion for SMTP errors. Since we don't implement our
own connection object or connection pool for SMTP, we simply wrap the couple
places we call functions from nodemailer that connect to SMTP, namely
SendmailClient's _send() and the new verify() call in /auth.
* Moves RetryableError to the 'errors' module since it's now a base class for
retryable IMAP //and// SMTP errors.
* Moves the main `smtpConfig()` logic which used to live on the Account model
into AuthHelpers so it can be shared between the Account model and the verify
code.
* Converts a few different places to use `import` syntax instead of
`require` syntax for module imports. Apologies for not splitting this out
into a separate diff—would have been a fair amount of work and looks not too
difficult to skim over in the context of the rest of the patch.
* Fixing a bug in a previous commit where erroring sends would crash because of
using `this._transporter.options` instead of `this._transporter.transporter.options`
Test Plan: manual
Reviewers: evan, halla, juan
Reviewed By: halla, juan
Differential Revision: https://phab.nylas.com/D4200
Summary:
This diff will give us insight into how people use Nylas Mail on their
computers with respect to how mobile they are (i.e. whether they use NM
on battery or plugged in).
Test Plan: Run locally
Reviewers: evan, spang, juan
Reviewed By: juan
Subscribers: gleb
Differential Revision: https://phab.nylas.com/D4267
Summary:
Prior to Nylas Mail, the Nylas Cloud API provided an endpoint that
returned rankings for contacts which it computed based on how frequently
and how recently a user sent mail to a recipient. This diff reimplements
that functionality in Nylas Mail. This should improve contact
auto-complete when composing emails to frequently contacted recipients.
Test Plan: Run locally, verify that frequent contacts are suggested earlier
Reviewers: spang, evan, juan
Reviewed By: evan, juan
Maniphest Tasks: T7948
Differential Revision: https://phab.nylas.com/D4253
Summary: See title. Part of T7681.
Test Plan: ran the specs
Reviewers: evan, mark, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4264
Summary:
Periodically ping client-sync's /health endpoint and store the latest
sync activity. If we get an ECONNREFUSED error, the worker window is
unavailable. Report the last known activity and restart the worker
window. Part of T7681.
Test Plan: manual, specs
Reviewers: evan, mark, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4263
Summary:
We still don't know exactly what scenarios cause us to get 'database is
locked' errors, but generically this means that we have multiple
processes trying to access the database at the same time. In an attempt
to handle this gracefully, this diff makes it so that we retry the
queries in these cases. Theoretically, the database should free up once
the other process is done using it, and the erroring process just needs
to wait its turn. We still throw an error after 5 retries, so if there's
a larger issue, we'll still be able to tell in Sentry.
Addresses T7992
Test Plan:
I opened a transaction in the worker window and then tried to
do the same in the main window. If I didn't release the transaction in
the worker window, the main window eventually errored. If I did release
the transaction, the main window continued creating its own transaction.
Reviewers: mark, juan, evan
Reviewed By: juan, evan
Differential Revision: https://phab.nylas.com/D4254
Summary:
We weren't doing that, so we would have accurate search index info for
threads when they were, e.g., moved to another folder.
Test Plan: Run locally
Reviewers: spang, evan, juan
Reviewed By: evan, juan
Differential Revision: https://phab.nylas.com/D4260
Summary:
Before this commit, it was impossible to remove inline images via the x
button
Test Plan: manual
Reviewers: mark, halla
Reviewed By: mark, halla
Differential Revision: https://phab.nylas.com/D4257
Summary:
I had a bit of downtime this morning so I decided to look into how to store Electron crash data. Electron, like Chromium, stores crash data in an arcane file format named minidump.
There's a bunch of services you can use to store files formatted in this format, like Mozilla's Socorro or zcbenz's own mini-breakpad-server but they're all pretty hard to install. Luckily, it turns out there's a Ruby gem to process minidump files and send them to Sentry! Seeing that, I whipped up a quick sinatra service and hosted it on Heroku – you can take a look at the source code here if you're curious: https://github.com/khamidou/electron-breakpad-sentry
This diff adds the client-side code we need to start sending crash reports.
Test Plan: Tested manually.
Reviewers: juan, evan
Reviewed By: juan, evan
Maniphest Tasks: T7926
Differential Revision: https://phab.nylas.com/D4259
Summary:
We'd like to get an idea of how long the client app is having to wait
when requesting a file. Waiting can cause things like inline image
attachments to appear very slow to load.
Test Plan: Run locally
Reviewers: evan, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4255
Summary:
The original name seems like it's initiating the download, when really
it's just returning the data of an already in-progress/completed
download.
Test Plan: Manual, specs
Reviewers: evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4251
Summary:
We need to download the files and then treat them as uploads. Rather
than using an actual Upload object, which would require another data
transfer, we create an object with all the necessary Upload-like
properties and point it to the downloaded file.
Addresses part of T7960
Test Plan: Manual, some specs
Reviewers: evan, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4249
Summary: Using `await` instead of `advanceClock()` fixes all the things!
Test Plan: Ran the spec file
Reviewers: evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4248
Summary:
This fixes T7995.
Previously, attempting to log out from your NylasID would just
automatically sign you back in with the same NylasID because the webview
session was preserved.
Now, instead of relaunching the windows, we restart the app to clear the
webview session
Test Plan: manual
Reviewers: halla, evan
Reviewed By: evan
Maniphest Tasks: T7995
Differential Revision: https://phab.nylas.com/D4224
Summary:
Previously, the logic for deciding wether to report an APIError lived
inside NylasAPIRequest and depended on an array of ignorable
statusCodes. However, when handling APIErrors, NylasAPIRequest would
convert special error codes (for offline errors) into a statusCode of 0,
which would automatically be ignored.
Given that the delta streaming connection didn't convert status codes to
0, it wouldn't actually ignore the error and end up reporting it,
flooding sentry.
This commit makes it so that that logic lives inside APIError, and
anyone who handles APIErrors can check wether to report them or not,
including the delta streaming connection
Test Plan: manual :(
Reviewers: spang, evan, halla
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4235
Summary:
Currently, when we auth an account for the first time in Nylas Mail (or
we blow away the database), the app is going to request transactions
since cursor `null` from the /delta/streaming endpoint and from the local-sync
delta observable, instead of requesting transactions since cursor `0`
This is due to a subtle bug with the use of default values when
destructuring an object. Our coded did the following:
```
const {cursor = 0} = this._state
```
Which at a glance seems correct. However, this will only work as
expected if `this._state` has the following shape:
```
{cursor: undefined}
```
And unfortunately, our `this._state` looked like this when authing an
account for the first time:
```
{cursor: null}
```
Which would make `cursor === null` instead of `0`.
This is because when using default values, null is considered an
intentional argument/value, as opposed to not passing any argument/value
(which will mean that the argument is undefined).
This was a regression introduced in d60a23c and 8bc2ec5
Test Plan: manual, will add regression test in upcoming diff
Reviewers: evan, spang, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4243
Summary:
Previously, after creating a new folder, the UI would indicate that the new
folder had children, even though it didn't. This was caused by duplicate models
in our `MutableQueryResultSet` for the user's categories. Basically, we would
sync the server version of the folder before the `SyncbackTask` for the new
folder returned its `serverId`. Without the `serverId`, the synced version of
the folder couldn't yet be tied to the optimistic folder, so a second row was
created in the database. This second row is removed when the `syncbackTask`
does return the `serverId`, because we persist the optimistic folder with a
`REPLACE INTO` query. (This deletes other rows with the same id.) However,
since this was done inside a `persist` change with the `serverId` and no
`unpersist` was ever recorded for the `clientId`, our `MutableQueryResultSet`
never removed the `clientId` model.
To address this, this diff adds a check in `updateModel` to see if the
`serverId` is being added. If it is, and both the `serverId` and `clientId`
exist in the `_ids` list, we remove the `clientId`.
The children indicator does still briefly show up while there are still two
separate rows for that folder in the database. If we want to get rid of this
completely, we would have to ensure that we do not sync the folder before the
`syncbackTask` returns the `serverId`. However, this would probably be pretty
involved, and for not much gain. This fix is much simpler and reduces most of
the issue.
Test Plan: manual
Reviewers: juan, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4228
Summary:
The ignore list was very old. It included several dozen MB of docs_src and
other crap in our builds
Test Plan: manual
Reviewers: halla, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4213
Summary: We need to upload the nupkg for the Windows autoupdater to work
Test Plan: manual
Reviewers: juan, spang, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4219
Summary:
This is going to be a diff way shorter than the previous one! Basically, it adds a new endpoint, `/blobs` to our API to store send later attachments. When a user schedules a draft to be sent, we send all attachments to this endpoint. Separately, we store the rest of the message as metadata.
When it's time to send the message, we fetch the attachments from S3, fetch the metadata and merge them together to get a message we can send.
Test Plan: Tested manually. Will make a final QA pass before landing.
Reviewers: juan, halla, evan
Reviewed By: halla, evan
Differential Revision: https://phab.nylas.com/D4196
Summary:
On windows running `git rev-parse --short HEAD` does in fact now give you
9 characters instead of 7 like it does on Mac. This will ensure that
builds get uploaded to the same folder and help ensure we don't post a
version that doesn't exist on the release page.
Test Plan: Manual
Reviewers: juan, halla, spang
Reviewed By: spang
Differential Revision: https://phab.nylas.com/D4217
Summary:
Use electron's `powerMonitor` module to detect when the computer resumes
from sleep, and restart the sync loop when that happens in order to
sync the inbox immediately, in case we received any new mail events
while the computer was asleep
Test Plan: manual
Reviewers: evan, spang, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4216
Summary:
This commit makes it so `resetEmailCache` works as expected, i.e. it
removes all databases, without forcing the user to re sign-in to their
accounts or NylasID
Previously, this method removed the database without removing the
accounts, left users in an un-authed state that was hard to recover
from. This was fixed in D4212 which makes sure that when we get a new
identity, sync and deltas are restarted
However, resetEmailCache would still force you to log in to yoru NylasID
because it was deleted from the database. However, if we reuse the
command `application:relaunch-to-initial-windows` instead of manually
deleting the database, we can relaunc the app while preserving the users
NylasID session, so they don't have to sign back in manually.
Test Plan: manual
Reviewers: evan, spang, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4215
Summary:
This prevents the app from entering a restart loop when there's no
identity.
Specifically, when a user logs out of their identity, or when she resets the
email cache, or any other scenario that leaves the app without an
identity but with accounts added, the sync loop (and deltas) will start
without an Identity.
This will cause NylasAPIRequest to throw an error that
forces the user to close the app. When the app restarts, sync will start
again without an identity, and the user will be forced to close the app
again, and so on and so forth for the rest of eternity
Relevant error:
https://github.com/nylas/nylas-mail-all/blob/master/packages/client-app/src/flux/nylas-api-request.es6#L165-L174
Additionaly, this makes sure that after resetting the email cache, the sync
process starts when the identity becomes available
This solves T7989
Test Plan: manual
Reviewers: evan, spang, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4212
Summary:
build-client, aka packaging, aka bundling is now separate from uploading.
This is both to compartmentalize our tasks a bit more and so we can add a
non-grunt windows task inbetween packaging and uploading.
No more heavily-overloaded PUBLISH_BUILD flag.
Added SIGN_BUILD flag instead.
No more TRAVIS and TRAVIS_PULL_REQUEST flag.
Test Plan: Manual
Reviewers: halla, juan, spang
Reviewed By: juan, spang
Differential Revision: https://phab.nylas.com/D4208
Summary:
This removes specs from the build production version.
One of the major issues with the windows build was that one of the package
specs included a .node compiled file that the codesigner attempte to sign,
but couldn't find for some reason.
In testing, removing the specs from the build prod version fixed this.
While we could just remove the one offending test, I think we should just
remove the whole suite from the build version. That'd save in the number
of files we have to ship, save download time, save build time, and have
less chance for some symlinked issue failling a build in the future.
Test Plan: Manual
Reviewers: khamidou, halla, juan, spang
Reviewed By: juan, spang
Differential Revision: https://phab.nylas.com/D4206
Summary:
It always drives me crazy when folks use "setup" (no space) as a verb,
because the verb version has a space in it.
Also remove the word "Setup" from the displayName for custom IMAP accounts
because it doesn't make sense for how we use this field, e.g.
a8f36b88d9/packages/client-app/internal_packages/onboarding/lib/page-account-onboarding-success.jsx (L27)
will display "Sucessfully connected to IMAP / SMTP Setup!" with the old
wording when you successfully auth a custom IMAP account.
Test Plan: manual
Reviewers: evan, juan, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4199
Summary:
Dunno if I'm the only one having this problem, but I spent at least an
hour over the last two days wondering why hapi just sits there and hangs
in the Joi validation step when I copy-paste calls to /auth from the
developer bar to my terminal. It turns out that if you fail to send the
correct Content-Type when sending a JSON payload, Joi just hangs
forever. (WTF!)
Test Plan: manual
Reviewers: halla, evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4204
Summary:
Previously, n1 cloud auth errors were stuffed into an account's
syncState. The problem with that is that the sync loop manages that
state, and as long as the sync loop is running it will set that state to
running. However, it might be the case that the sync loop is running but
we can't connect to n1Cloud, so even though we would set the `syncState`
to `n1_cloud_auth_failed`, the sync loop would just set it back to
'running', and the user wouldn't see the error notification indicating
that it can't connect to n1Cloud
This commit makes it so we keep tracj of the auth failure state for
n1Cloud in a separate field, and makes sure that the error notification
component shows that error.
Test Plan: manual
Reviewers: mark, spang, evan, halla
Reviewed By: evan, halla
Differential Revision: https://phab.nylas.com/D4172
Summary:
On rare occasions, the api error would not have a body, and we would try
to access the message prop on the body, which would in turn throw an
error.
This commit prevents this error, and adds better messaging to the sentry
report
Test Plan: manual
Reviewers: mark, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4194
Summary:
This commit adds a new perf metric for inline composer times.
Additionally, it consolidates the timer logic for all other types of draft
creation (mailto links, dropping a file in the app dock icon, etc).
Test Plan: manual
Reviewers: halla, evan, mark
Reviewed By: evan, mark
Differential Revision: https://phab.nylas.com/D4186
Summary:
Before this commit, we would call .close() onError and when the
connection closed. We also closed the connection onError, even though
NylasLongConnection had also closed it, so we ended up calling close a
bunch of times. This would cause us to set a bunch of timeouts to retry
unecessarilly.
This commit makes it so we ensure there's only one retry timeout and
consolidates the logic that calls .close() so we don't call it so many
times unnecessarily
Test Plan: manual
Reviewers: spang, mark, evan
Reviewed By: mark, evan
Differential Revision: https://phab.nylas.com/D4166
Summary:
This commit consolidates the `DeltaConnectionStatusStore` and the
`DeltaConnectionStore` which kept track of very similar state and made
sense to be the same store (as per feedback in D4118#77647)
Given that this state needs to be available app-wide for plugins to
query the status of delta connections, `internal_packages/deltas` was
removed (given that it only activated that store), in favor of having the
unified store inside `src/flux/stores` and available via `nylas-exports`
The `deltas` package also contained some contacts-ranking code, which is
no longer in use until we restore that fetaure, so I created a
`internal_packages/contact-rankings` which contains this unused code for
now.
Test Plan:
manually open, close, end delta connections, verify that I'm getting
correct results. unit tests to come
Reviewers: halla, spang, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4145
Summary:
This DRY's up code that use these status codes
Depends on D4130
Test Plan: manual
Reviewers: spang, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4140
Summary:
When reporting error inside `IdentityStore._fetchIdentity`, we would
pass null as the `extra` parameter for `reportError`. This would cause
us to try to assign something to null and throw an error while reporting
the error.
(Even though `extra` has a default value of `{}` (see
declaration of `NylasEnv.reportError`), passing null will be interpreted
as a valid arg and it wont get assigned the default value)
Thankfully, the second error /is/ reported to sentry: https://sentry.io/nylas/nylas-mail/issues/230589717/
Test Plan: manual
Reviewers: mark, halla, spang, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4150
Summary:
This is not a bug. Maaaybe we want to report this to MixPanel for individual
user debugging (how do we do that?), but it doesn't make sense to report this
error to Sentry.
Example:
https://sentry.io/nylas/nylas-mail/issues/230530818/
Test Plan: yolo
Reviewers: mark, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4130
Summary:
If you have the console open and are looking at the Source panel or
something, this will constantly bring you back to the Console panel. Won't
do that anymore with this chec
Test Plan: manual
Reviewers: spang, mark, halla, juan
Reviewed By: mark, halla, juan
Differential Revision: https://phab.nylas.com/D4162
Summary:
Errors weren't getting reported because of a circular reference causing
JSON.stringify to fail.
We can't add the whole API `response` object onto the error object,
because sometimes (like when the delta connection errors with a 401), the
`response` object is circular. We only use it to the extract the status
code anyway and it's a huge object. This diff excludes them from the
APIError objects.
We also have code to recover just in case JSON.stringify errors for some
other weird reason. It'll still attempt to report the error prefixed with
`Recovered Error`
Finally, we used to spit only the `error.stack` to the console. This meant
what we saw in the console didn't properly include the error messages
being incredibly not useful.
It's better to console.error the whole error object since that'll more
nicely display on the inspector console
Test Plan: manual
Reviewers: juan, spang, mark, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4154
Summary:
Previously if we got back a huge number of unknown UIDs from IMAP search
we would try to sync all of them at once. This could lead to hanging the sync
loop trying to download tons of messages. This diff limits the UIDs we're
willing to sync per task to 500 and splits each task up into chunks of 25
messages so that we don't try to download all of them at once. If we need
to sync more than 500 uids then at the end of the syncback task it will
queue another task to run the next time the sync loop rolls back around.
Test Plan:
Run locally, verify that we gracefully handle various situations
including cancelling during the syncback task, cancelling between syncback
tasks, huge numbers of results, etc
Reviewers: evan, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4142
Summary:
This commit bumps the version of sequelize to point to the latest
version in our fork. This version bump also includes latest updates from
upstream sequelize since the last time we bumped the version.
Notably, it includes a critical fix that will prevent the sync
loop from getting stuck on rare occasions.
Specifically, sequelize's `Query.prototype.run` could in some cases return a
Promise that would /never/ resolve or reject, effectively halting the
execution of any code that was waiting for that promise. This was due to
a lack of error handling inside a the query's `afterExecute` function; if an
error was thrown there, the enclosing Promise would never reject, and
the error would just remain uncaught. An example of such an error that
could cause this scenario is: T7742 - https://sentry.io/nylas/nylas-mail/issues/230016155/
Sync getting halted in this way can produce a variety of user facing
bugs like not being able to send or it taking an absurd amount of time
(hours), tasks never finishing or taking an absurd amount of time to
complete, new mail not arriving, among others
The commit that fixes this is: 0deeda9e1a
The full set of changes introduced by this version bump are: https://github.com/nylas/sequelize/compare/nylas-3.30.0...nylas-3.40.0
**Note:**
It is important to note that sequelize might still halt the sync loop if
there are other places in the code that can return Promises that never
resolve or reject under certain circumstances. I can consistently reproduce this
scenario when an error is thrown inside `afterExecute`, and I've seen it
happen in the wild, but I've also ran into this type of sync loop halting (db
promises never resolve) without any sequelize errors being thrown, which
suggests that there are other places in the sequelize code that might end
up returning a Promise that will halt sync.
Unfortunately, we don't have a good way to detect and report when this
happens yet, but we are adding one in upcoming diffs in order have data
on how many people are running into this, and/or if this patch completely
fixes the issue. Otherwise, we'd need to audit sequelize's code.
Should resolve T7837 and T7767
Test Plan: manual
Reviewers: spang, mark, halla, khamidou, evan
Reviewed By: evan
Subscribers: mg, tomasz
Maniphest Tasks: T7767, T7837
Differential Revision: https://phab.nylas.com/D4152
Summary:
This commit splits apart the `AccountDeltaConnection` class, which was
in charge of listening to both cloud /and/ local deltas by way of an
artificial interface, `DeltaStreamingInMemoryConnection`.
Splitting this into 2 modules with separate responsibilities will hopefully
make this code easier to reason about and reduce some cruft and unnecessary
indirection.
Specifically, this commit makes it so:
- `DeltaConnectionManager` is only in charge of starting and ending `DeltaStreamingConnection`s, which are solely in charge of listening to deltas from the cloud api
- `LocalSyncDeltaEmitter` no longer unnecessarily emits events for the `deltas` package to listen to but rather directly processes and saves those deltas from the K2 db to edgehill.db
- `LocalSyncDeltaEmitter` is also in charge of keeping track of the latest received cursor, under its own JSONBlob key in edgehill.db. This migrates localSync cursors saved under the old key.
- `LocalSyncDeltaEmitter` is now instantiated and managed from within the `SyncProcessManager` as opposed to the `SyncWorker`. Apart from removing extra state from the `SyncWorker`, this removes dependencies on the client-app environment from the sync-worker.
- `DeltaStreamingInMemoryConnection` and `AccountDeltaConnection` are now gone
(Sorry for the big diff! This one was a little hard to split up without landing something broken)
Depends on D4121
Test Plan: manual + unit tests planned in upcoming diff
Reviewers: halla, mark, evan, spang
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4122
Summary:
This is to make it available to all plugins in the app
Depends on D4120
Test Plan: manual
Reviewers: halla, mark, spang, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4121
Summary:
This commit completely refactors `DeltaStreamingConnection`, notably
introducing the following changes:
- Right now, `AccountDeltaConnection` establishes both delta connections to the cloud api and to the `client-sync` database (K2). This class is meant to disapper in favor of splitting into two different classes meant for syncing with the n1Cloud api and the local database. Specifically, `DeltaStreamingConnection`'s only responsibility is now to connect to the n1Cloud API and establish an http streaming connection for metadata deltas, etc. This class no longer unecessarily inherits from `NylasLongConnection`, which removes a lot of unecessary callback indirection.
- The statuses of the n1Cloud delta streaming connections are now stored in as JSONBlobs in edgehill.db under new keys. This commit ensures that the data is correctly migrated from the old key (`NylasSyncWorker:<account_id>`).
- The `DeltaStreamingConnection` now correctly retries when closed or errors. This logic previously existed, but was removed during the K2 transition: https://github.com/nylas/nylas-mail/blob/n1-pro/internal_packages/worker-sync/lib/nylas-sync-worker.coffee#L67
- Delta connection retries now backoff using the `ExponentialBackoffScheduler`
- Attempt to restore the delta connection whenever the app comes back online
Depends on D4119
Test Plan: manual + planned unit tests in upcoming diff
Reviewers: halla, mark, evan, spang
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4120
Summary:
For legacy reasons, NylasLongConnection did not take the `onResults`
callback as an option, but rather it was a method you had to call on the
connection, which took the callback, and which you had to call after
instantiating the connection. This was an annoying and clunky interface.
This commit makes it so NylasLongConnection takes an `onResults` option
as one would expect it to, and updates any references to the old
interface
Depends on D4118
Test Plan: manual
Reviewers: halla, mark, spang, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4119
Summary:
Turns out, NylasSyncStatusStore doesn't need to concern itself with the
status of delta connections.
This is legacy cruft from the days when this store used to provide the
online status of the app based on the delta connections. Now we have an
OnlineStatusStore, so we don't need this at all
Instead of stuffing that state in that store, this commit adds a small
DeltaConnectionStatusStore to provide an easy way to listen to changes
on the status of delta connections. The only part of the app concerned
with that state is the DeveloperBarStore in order to render the green
circles for delta connections in the worker window.
Depends on D4117
Test Plan: manual
Reviewers: mark, halla, spang, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4118
