Summary:
This replaces the API delta stream with a direct in-memory one
Depends on D3548
Test Plan: manual
Reviewers: halla, jackie
Reviewed By: halla, jackie
Subscribers: juan
Differential Revision: https://phab.nylas.com/D3549
Summary:
We were returning an array of arrays, when really we wanted an object
of arrays keyed by model type. This also involved removing an implementation
of mapObject that was overwriting underscore's implementation of it.
Test Plan: local, having others try it.
Reviewers: juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3509
Summary:
This commit fixes processing deltas for `ProviderSyncbackRequests` and
resolution of the `SyncbackTaskAPIRequest`
Specifically:
- Fix and update field definitions for `ProviderSyncbackRequest`
- Correctly parse the error attached to any `ProviderSyncbackRequest`
- Correctly process deltas for `ProviderSyncbackRequests` inside the
delta-processor
- Fix any tasks that used `SyncbackTaskAPIRequest`, and update send to use those
Depends on D3510
Test Plan: Todo/Manual
Reviewers: jackie, halla, evan
Reviewed By: halla, evan
Differential Revision: https://phab.nylas.com/D3511
Summary:
Addresses T7275
Previously, we kept track of the sync status of each API model, and the progress
we'd made syncing all available models (e.g. all threads, messages, events, etc)
Given K2's set up, we are now keeping track of sync status per folder, i.e. what
percent of the folder's messages we've synced. This status is now reported from K2
to N1 via folder object deltas, and this commit rewrites the
NylasSyncStatusStore (in ES6) to reflect that.
The new Store keeps the sync state per account, which is the merged state of
per folder sync state, and delta connections state.
We also got rid of `CategoryStore.whenCategoriesReady` in favor of
`whenCategoryListSynced`, which is derived from the fact hat as long as we've
started syncing one folder, we've already synced the entire list of of folders/labels.
There are a couple of TODOs to be addressed in upcoming diffs:
- T7329 Restore the sidebar component to show sync progress, which was previously removed
- T7330 Figure out how to report sync progress per label, specifically, we are interested in knowing how much of the inbox we've synced, which is a label in Gmail. (This might be a non-issue if we sync the inbox very fast, first)
Depends on D3514
Test Plan: Manual
Reviewers: mark, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3515
More spec fixes
replace process.nextTick with setTimeout(fn, 0) for specs
Also added an unspy in the afterEach
Temporarily disable specs
fix(spec): start fixing specs
Summary:
This is the WIP fix to our spec runner.
Several tests have been completely commented out that will require
substantially more work to fix. These have been added to our sprint
backlog.
Other tests have been fixed to update to new APIs or to deal with genuine
bugs that were introduced without our knowing!
The most common non-trivial change relates to observing the `NylasAPI` and
`NylasAPIRequest`. We used to observe the arguments to `makeRequest`.
Unfortunately `NylasAPIRequest.run` is argumentless. Instead you can do:
`NylasAPIRequest.prototype.run.mostRecentCall.object.options` to get the
`options` passed into the object. the `.object` property grabs the context
of the spy when it was last called.
Fixing these tests uncovered several concerning issues with our test
runner. I spent a while tracking down why our participant-text-field-spec
was failling every so often. I chose that spec because it was the first
spec to likely fail, thereby requiring looking at the least number of
preceding files. I tried binary searching, turning on and off, several
files beforehand only to realize that the failure rate was not determined
by a particular preceding test, but rather the existing and quantity of
preceding tests, AND the number of console.log statements I had. There is
some processor-dependent race condition going on that needs further
investigation.
I also discovered an issue with the file-download-spec. We were getting
errors about it accessing a file, which was very suspicious given the code
stubs out all fs access. This was caused due to a spec that called an
async function outside ot a `waitsForPromise` block or a `waitsFor` block.
The test completed, the spies were cleaned up, but the downstream async
chain was still running. By the time the async chain finished the runner
was already working on the next spec and the spies had been restored
(causing the real fs access to run).
Juan had an idea to kill the specs once one fails to prevent cascading
failures. I'll implement this in the next diff update
Test Plan: npm test
Reviewers: juan, halla, jackie
Differential Revision: https://phab.nylas.com/D3501
Disable other specs
Disable more broken specs
All specs turned off till passing state
Use async-safe versions of spec functions
Add async test spec
Remove unused package code
Remove canary spec
This commit will correctly keep track of folder and label ids when
creating them from N1.
Previously, when we sent the request to create a folder or label to our api,
we would immediately get back a serverId because it was created optimistically
in the back end— given that K2 is strictly non-optimistic, we won’t have a serverId
until some undetermined time in the future, and we need to somehow reference
the object that /was/ optimistically created in N1 to update the ui when
we do get the server id.
Since we can deterministically generate ids for folders and labels,
we "guess" what its going to be, and include it in the props of the syncback request
returned to N1. This is the simplest solution to get thing working
correctly right now, but we’ll need to revisit this in the future for
other types of objects (drafts, contacts, events), and revisit how we
will manage optimistic updates in N1 when we merge the 2 codebases
with K2 (given that K2 was designed to be non-optimisitc).
Since message IDs are now static but there's no good way to generate
static thread IDs while syncing an account from newest message first,
we give threads the ID of any message on that thread and, when setting
metadata, look up the local thread ID by first going through the
message table.
Also changed the DeltaProcessor so it doesn’t query for models before sending out `Actions.didPassivelyReceiveCreateDeltas`, and renames it to be more clear it’s about deltas.
This reverts commit ee5609bdb0.
Updates to nylas sync worker to support multiple cursors
Convert NylasSyncWorker to es6
Convert NylasSyncWorkerPool to es6
Extract into deltaProcessor
Update names to NylasSyncWorker state
Working on spec fixes
More spec fixes
Delta stream refactor fixes
- Threads: N1 always asks for the 'expanded' view, so K2 just always returns
the expanded view. N1 no longer needs to specify this.
- Messages: N1 only needed a 'count' view to display the initial sync progress
We've removed this progress bar and the corresponding request.
Summary:
Fixes a test that was broken due to my unfamiliarity with
CoffeeScript :-/
Test Plan: Tests
Reviewers: bengotow
Reviewed By: bengotow
Subscribers: juan
Differential Revision: https://phab.nylas.com/D3302
Summary:
This should help us avoid the thundering herd problem if we have some
kind of API outage affecting a wide number of clients.
Test Plan: Tests
Reviewers: bengotow
Reviewed By: bengotow
Subscribers: juan
Differential Revision: https://phab.nylas.com/D3297
NylasLongConnection ends the connection when the 'end' event is emitted
by the `request` object. When this happens, the global connection buffer is cleared.
Also, the global buffer holds the data we've received from the connection, and
whenever we receive new data, we accumulate it in the buffer and call a processBuffer function
which is throttled to 400ms.
Given that the buffer is global state, and processing occurs
asynchronously with a delay of up to 400ms, if the 'end' event on the connection is
fired before we actually get to process the buffer, we would clear it and show no results.
This scenario currently only affected search because if we accidentally
threw away some data when streaming deltas, we will get that data again
when we reopen the delta streaming connection.
- Completely remove nylas-long-connection.coffee file and consolidate all logic into
nylas-long-connection.es6 and worker-sync/delta-streaming-connection.es6:
- Separate it into DeltaStreamingConnection which is specific to the worker-sync
package for handling deltas
- NylasLongConnection is a "reusable" persistent connection to our API
which is also used by the search package
- This is a basically a fixed revision of https://phab.nylas.com/D2875,
see difff for more info
- Makes it so delta streaming does not retry so aggressively on 403s, which
happens whenever a user’s trial has expired
When APIRoot is using https, the port will forced to be 443 even when APIRoot
indicates another port. This is because of the line options.port = 443
overwriting the correct options.port extracted by url.parse. This is fixed by
removing the problematic line. The default port is still 443 if you look into
https.js. In case we want a different default port in the future, we can do
options.port = options.port || 443.
