Summary:
- Remove nylas-long-connection.coffee duplicate code from sync worker package
and replace with delta-streaming-connection which uses NylasAPI.longConnection
- Replacement for XHR will come in another diff -- still need to clean up and
fix some edge cases
Test Plan:
- TODO! Will add test coverage for NylasLongConnection and
DeltaStreamingConnection
Reviewers: evan, bengotow
Reviewed By: evan, bengotow
Differential Revision: https://phab.nylas.com/D2875
Summary:
This diff is designed to dramatically speed up new window load time for
all window types and reduce memory consumption of our hot windows.
Before this diff, windows loaded in ~3 seconds. They now boot in a couple
hundred milliseconds without requiring to keep hot windows around for
each and every type of popout window we want to load quickly.
One of the largest bottlenecks was the `require`ing and initializing of
everything in `NylasExports`.
I changed `NylasExports` to be entirely lazily-loaded. Drafts and tasks
now register their constructors with a `StoreRegistry` and the
`TaskRegistry`. This lets us explicitly choose a time to activate these
stores in the window initalization instead of whenever nylas-exports
happens to be required first.
Before, NylasExports was required first when components were first
rendering. This made initial render extremely slow and made the proposed
time picker popout slow.
By moving require into the very initial window boot, we can create a new
scheme of hot windows that are "half loaded". All of the expensive
require-ing and store initialization is done. All we need to do is
activate the packages for just the one window.
This means that the hot window scheme needs to fundamentally change from
have fully pre-loaded windows, to having half-loaded empty hot windows
that can get their window props overridden again.
This led to a major refactor of the WindowManager to support this new
window scheme.
Along the way the API of WindowManager was significantly simplifed.
Instead of a bunch of special-cased windows, there are now consistent
interfaces to get and `ensure` windows are created and displayed. This
DRYed up a lot of repeated logic around showing or creating core windows.
This also allowed the consolidation of the core window configurations into
one place for much easier reasoning about what's getting booted up.
When a hot window goes "live" and gets populated, we simply change the
`windowType`. This now re-triggers the loading of all of the packages for
the window. All of the loading time is now just for the packages that
window requires since core Nylas is there thanks to the hot window
mechanism.
Unfortunately loading all of the packages for the composer was still
unnaceptably slow. The major issue was that all of the composer plugins
were taking a long time to process and initialize. The solution was to
have the main composer load first, then trigger another window load
settings change to change the `windowType` that loads in all of the
plugins.
Another major bottleneck was the `RetinaImg` name lookup on disk. This
requires traversing the entire static folder synchronously on boot. This
is now done once when the main window loads and saved in a cache in the
browser process. Any secondary windows simply ask the backend for this
cache and save the filesystem access time.
The Paper Doc below is the current set of manual tests I'm doing to make
sure no window interactions (there are a lot of them!) regressed.
Test Plan: https://paper.dropbox.com/doc/Window-Refactor-UYsgvjgdXgVlTw8nXTr9h
Reviewers: juan, bengotow
Reviewed By: bengotow
Differential Revision: https://phab.nylas.com/D2916
- Make the retry interval go 2 sec, 3.4s, 6 sec...
- Only show the connection status bar if the interval is > 5 seconds, in case the error was temporary.
- Do not show sync errors in the sidebar. The only available action is "Try Again", and we try again on our own. The error is frustrating and the user can't do anything about it anyway.
Summary:
This diff includes several updates:
- Update sqlite version to use FTS5
- Adds new methods to DatabaseStore to create and update Search Indexes
- Currently indexing subject, participants, and thread message bodies or snippets if the body is not available
- Update initial sync to fetch 5000 message bodies per account
- Adds a new SearchMatcher
- Add new thread-search-index package to run in the worker window to
init and keep thread search index up to date
- Converts Thread to ES6
TODO:
- Remove/Update suggestions dropdown
- Add tests
Test Plan: - TODO
Reviewers: evan, bengotow
Reviewed By: evan, bengotow
Differential Revision: https://phab.nylas.com/D2826
Summary:
The TaskQueue does it's own throttling and has it's own processQueue retry timeout, no need for longPollConnected
Remove dead code (OfflineError)
Rename long connection state to status so we don't ask for `state.state`
Remove long poll actions related to online/offline in favor of exposing connection state through NylasSyncStatusStore
Consoliate notifications and account-error-heaer into a single package and organize files into sidebar vs. header.
Update the DeveloperBarStore to query the sync status store for long poll statuses
Test Plan: All existing tests pass
Reviewers: juan, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D2835
Summary:
- Was not properly updating the references to snoozed categories when
accounts were added or removed
- Update whenCategoriesReady to make sure we listen until category syncing has concluded (Move inside CategoryStore)
- #1676, #1658
Test Plan: - TODO
Reviewers: evan, drew, bengotow
Reviewed By: bengotow
Differential Revision: https://phab.nylas.com/D2723
Summary:
Snooze should wait for categories on all accounts
Fix authPlugin to rembmer `plugin+accountId`, not pluginId, add specs
categories() returned [], categories(acctId) returned {}
dry up sync worker, fetch metadata before anything else
Test Plan: Run tests
Reviewers: drew, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D2693
Summary:
Changes the delta code to handle new deltas on the Account object,
which are triggered by changes in sync state indicating various backend
issues. Saves the sync state in a new field on the Account object, which
is persisited in `config.cson`.
Includes several UI changes to display more information when an account has
backend sync issues. Adds better messages and new actions the user can take
based on the type of sync issue.
Additionally, fixes bug in action bridge that was preventing multi-arg global
actions from working.
Test Plan:
Manual, by testing different sync state values and triggering deltas from the
backend
Reviewers: juan, evan, bengotow
Reviewed By: evan, bengotow
Subscribers: khamidou
Differential Revision: https://phab.nylas.com/D2696
Summary:
Previously we were storing sync cursors in config.cson. They were by
far the most frequently updated piece of data in config. To make things worse,
all the writing was happening in the worker window - the main window was just
seeing the changes on disk and reloading.
I believe there's an edge case which causes the main window to read the config
file when it's mid-write and empty. This causes the accounts array to become
empty in the AccountStore and lots of downstream issues. It's also then possible
for the main window to write a config change of it's own and empty the file
permanently.
Test Plan: A few new tests to make sure this is backwards compatible.
Reviewers: juan, evan
Reviewed By: juan, evan
Differential Revision: https://phab.nylas.com/D2642
Summary:
- Adds a class ModelWithMetadata which models can now extend from
- Instances of this class can query metadata for a plugin via
`obj.metadataForPluginId(pluginId)`
- To observe changes on metadata it is sufficient to observe database changes on
the model. e.g.:
`DatabaseStore.findAll(Thread,
[Thread.attributes.pluginMetadata.contains(pluginId)])`
- To set metadata a new action has been created: Actions.setMetadata
- Adds a helper observable in nylas-observables to query for models with
metadata
- Merges CreateModelTask and UpdateModelTask into SyncbackModelTask
- Update SendDraftTask ans SynbackDraftTask to handle metadata changes
Test Plan: - Unit tests
Reviewers: drew, evan, bengotow
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D2575
Summary:
1. **Generic CUD Tasks**: There is now a generic `CreateModelTask`,
`UpdateModelTask`, and `DestroyModelTask`. These can either be used as-is
or trivially overridden to easily update simple objects. Hopefully all of
the boilerplate rollback, error handling, and undo logic won't have to be
re-duplicated on every task. There are also tests for these tasks. We use
them to perform mutating actions on `Metadata` objects.
1. **Failing on Promise Rejects**: Turns out that if a Promise rejected
due to an error or `Promise.reject` we were ignoring it and letting tests
pass. Now, tests will Fail if any unhandled promise rejects. This
uncovered a variety of errors throughout the test suite that had to be
fixed. The most significant one was during the `theme-manager` tests when
all packages (and their stores with async DB requests) was loaded. Long
after the `theme-manager` specs finished, those DB requests were
(somtimes) silently failing.
1. **Globally stub `DatabaseStore._query`**: All tests shouldn't actually
make queries on the database. Furthremore, the `inTransaction` block
doesn't resolve at all unless `_query` is stubbed. Instead of manually
remembering to do this in every test that touches the DB, it's now mocked
in `spec_helper`. This broke a handful of tests that needed to be manually
fixed.
1. **ESLint Fixes**: Some minor fixes to the linter config to prevent
yelling about minor ES6 things and ensuring we have the correct parser.
Test Plan: new tests
Reviewers: bengotow, juan, drew
Differential Revision: https://phab.nylas.com/D2419
Remove cloudState and N1-Send-Later
Summary:
- WIP: Need to fix tests and some errors!
- Refactors Category class to hold information about its type
- Refactors CategoryStore to rely on observables instead of local caches
- Adds and updates Observables and helpers
- Refactors ContactStore to hold entire cache of contacts instead of per
current account
- Same for ContactRankingStore and other stores
- Refactors method names for AccountStore + some helpers
- Updates MailViewFilter to hold an account
- Adds basic Unified filter
- Replaces AccountStore.current calls with either:
- The account of the currently focused MailViewFilter
- The account associated with a thread, message, file, etc...
- A parameter to be passed in
- Arbitrarily, the first account in the AccountsStore
Test Plan: - Unit tests
Reviewers: evan, bengotow
Differential Revision: https://phab.nylas.com/D2423
Summary:
Originally, this was going to be a totally independent package, but
I wasn't able to isolate the functionality and get it tied in to
the delta-stream consumption. Here's how it currently works:
- The preferences package has a new tab which allows you to edit
mail filters. Filters are saved in a new core store, and a new
stock component (ScenarioEditor) renders the editor. The editor
takes a set of templates that define a value space, and outputs
a valid set of values.
- A new MailFilterProcessor takes messages and creates tasks to
apply the actions from the MailFiltersStore.
- The worker-sync package now uses the MailFilterProcessor to
apply filters /before/ it calls didPassivelyReceiveNewModels,
so filtrs are applied before any notifications are created.
- A new task, ReprocessMailFiltersTask allows you to run filters
on all of your existing mail. It leverages the existing TaskQueue
architecture to: a) resume where it left off if you quit midway,
b) be queryable (for status) from all windows and c) cancelable.
The TaskQueue is a bit strange because it runs performLocal and
performRemote very differently, and I had to use `performRemote`.
(todo refactor soon.)
This diff also changes the EditableList a bit to behave like a
controlled component and render focused / unfocused states.
Test Plan: Run tests, only for actual filter processing atm.
Reviewers: juan, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D2379
Summary:
Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction.
You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff:
- Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/
the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers.
- Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross.
- Many parts of the code ran extensive logic inside a promise chained within `atomically`:
BAD:
```
DatabaseStore.atomically =>
DatabaseStore.persistModel(draft) =>
GoMakeANetworkRequestThatReturnsAPromise
```
OVERWHELMINGLY BETTER:
```
DatabaseStore.inTransaction (t) =>
t.persistModel(draft)
.then =>
GoMakeANetworkRequestThatReturnsAPromise
```
Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-)
Other changes:
- This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large.
- This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster.
Test Plan: Run tests!
Reviewers: juan, evan
Reviewed By: juan, evan
Differential Revision: https://phab.nylas.com/D2353
Summary:
Add concept of "final" to Query, clean up internals
Tiny bug fixes
RxJs Observables!
WIP
Test Plan: Run tests
Reviewers: evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D2319
Summary:
Fixes bug where contact ranking was not being fetched, and refactors the refreshing
of contact ranks. Moves periodic refreshing of the database-stored ranks to the sync
workers so it occurs in the background, once per account. Refactors JSON cache code
accordingly.
Test Plan: manual
Reviewers: evan, bengotow
Reviewed By: bengotow
Differential Revision: https://phab.nylas.com/D2137
Summary:
Previously, when an error was encountered during initial mailbox sync we just started it
over after a retry delay. Recent API uptime issues mean that this was happening often and lots of
people were seeing sync retry many times. This is bad because the app is less performant while
it's syncing mail, and also generates unnecessary load as the app re-fetches threads it already has.
In this diff, there are new specs and functionality in nylas-sync-worker to start fetching
where we left off. This is typically going to be OK because the default sort ordering of the
threads endpoint is newest->oldest, so if new items have arrived since we started fetching
and page boundaries have changed, we'll get duplicate data rather than missing data. Connceting
to the streaming API as soon as we start the sync also ensures that we roll up any changes to
data we've already paginated over.
Test Plan: Run tests
Reviewers: drew, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D2132
Summary:
fixes T3725
changes:
- show the sync mail activity only if number of deltas received is greater than 10
- make the sync mail activity notification persist for longer, since large deltas (>1k) come in intervals of about 25 seconds, which was greater than our previous timeout interval of 15 seconds
- change to clearer copy
- minor style changes to match new copy
Test Plan: tested manually
Reviewers: bengotow
Maniphest Tasks: T3725
Differential Revision: https://phab.nylas.com/D2051
Summary:
fixes T3563
todo
[x] check with @bengotow and @evan -- does this solution make sense to you?
[x] decide on the actual visual cue to the user -- @bengotow you suggested a loader or spinner, but i was thinking that actual copy should be used so it doesn't look like an unhelpful spinner which goes indefinitely. what do you think of the copy here?
[ ] implement spinner similar to sending a draft
[ ] write tests
[ ] verify that we're showing progress in a helpful way
problem
- new users would see nothing in edgehill after they log in, with no indication on why they can't see any of their mail
- in actuality, our backend is actively pulling all their mail to serve it to edgehill
- new users think edgehill is broken because they're unaware of the work that our backend was doing
solution
- when the backend sends deltas to edgehill, we show a message in the activity bar saying that we're trying to get their mail
- we'll show the sync progress
Test Plan: still need to add tests. all previous tests still green.
Reviewers: bengotow, evan
Subscribers: sdw, bengotow, evan
Maniphest Tasks: T3563
Differential Revision: https://phab.nylas.com/D2032
Summary:
This started when I noticed that drafts weren't dissapearing from the
draft list after send. This was a pretty big bug because if you ever
clicked on one again and tried to re-send it would throw a 400 error
saying the draft id doesn't exist.
This uncovered a few fundamental issues with the DB.
First of all, the reason the draft list wasn't updating was because the DB
trigger that happened when we got in a new message, was being ignored
since the diff contained no drafts (it's now a message).
The bigger issue was that if you had a draft with only a clientId, gave it
a serverId, and tried to call "save", the REPLACE INTO method would not
update the old object, but rather create a second duplicate. This is
because the `id` field was being used as the PRIMARY KEY, and in this
case, that `id` field changed! The fix was to change the PRIMARY KEY to be
the `cilent_id` instead of the `id` and use that as the REPLACE INTO
index.
We still need the `id` field; however, because all of our reads depend on
that field usually being the serverId
Fixes T3507
Test Plan: See new and updated tests
Reviewers: dillon, bengotow
Reviewed By: bengotow
Maniphest Tasks: T3507
Differential Revision: https://phab.nylas.com/D1992
Summary:
Move sync workers and Edgehill token checks to work window
Move the task queue and database setup to the work window
Move ContactStore background refresh to work window
Store the task queue in the database
WIP
The TaskQueue now puts tasks in the database instead of in a file, which also means it can be observed
Move all delta sync and initial sync to a package, make NylasSyncStore which exposes read-only sync state
DraftStore no longer reads task status. Once you set the "sending" bit on a draft, it never gets unset. But that's fine actually.
If your package lists windowTypes, you *only* get loaded in those windowTypes. If you specify no windowTypes, you get loaded in the root window.
This means that onboarding, worker-ui, worker-sync, etc. no longer get loaded into the main window
ActivitySidebar has a special little store that observes the task queue since it's no longer in the window
Move "toggle component regions" / "toggle react remote" to the Developer menu
Move sync worker specs, update draft store specs to not rely on TaskQueue at all
Test Plan: Run existing tests, all pass
Reviewers: dillon, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D1936
