Commit graph

592 commits

Author SHA1 Message Date
Christine Spang
9752eea9f7 [local-sync] Set thread IDs to the ID of a message in the thread
Since message IDs are now static but there's no good way to generate
static thread IDs while syncing an account from newest message first,
we give threads the ID of any message on that thread and, when setting
metadata, look up the local thread ID by first going through the
message table.
2016-12-02 14:13:47 -08:00
Evan Morikawa
d9f3edad0f [isomorphic-core] ignore folder deltas due to just syncState version issues 2016-12-02 16:25:42 -05:00
Evan Morikawa
b06566d8bd [local-sync] make syncbackRequest objects N1-ready 2016-12-02 16:14:42 -05:00
Juan Tejada
b36f61812d [local-sync]: Don't create or sync \NonExistent or \NoSelect folders 2016-12-02 12:14:09 -08:00
Juan Tejada
f1491fa4cc fix observable disposable 2016-12-02 12:09:35 -08:00
Juan Tejada
5ac3fe9ea8 [local-sync] Fix starred role on folders 2016-12-02 12:07:21 -08:00
Evan Morikawa
40791028d5 [cloud-api] Update cloud-api readme & remove unused files
[cloud-api] move Dockerfile and pm2 files into cloud-api

[*] additions to .gitignore

[cloud-workers]: move pm2 file into cloud-workers

updates to readme

Revert "[cloud-workers]: move pm2 file into cloud-workers"

This reverts commit 952292124b6e8dfa304c5fbd945c0b00f427e154.

Revert "[cloud-api] move Dockerfile and pm2 files into cloud-api"

This reverts commit aa92f691bf991848a51c00a2d01a5848f20a35a2.

Update deploy readme
2016-12-02 13:28:05 -05:00
Ben Gotow
5ff7a92197 [*] fix(deltas): 0 as first cursor, deltas from redis are already JSON 2016-12-01 18:54:26 -08:00
Ben Gotow
7712269402 [*] fix(deltas): Cloud-API not filtering deltas at all, refactor a few things
- Don’t need functions in delta.js which must be called to return promsies. Fun of promsies is that you don’t need to care when they’re built to attach a .then.

- Make boundary between route handler and delta stream builder more explicit, don’t do query parsing in helpers, always reply from handler.

- Remove pushJSON extension to outputStream which never actually received JSON.

- Remove `takeUntil` - disposing of the downstream observable should dispose of all the merged/upstream observables

- Rename inflate => stringify since the returned value is a string not an object.

- Remove support for delta streams with no cursors. Don’t think this was supposed to be a feature.

- Add accountId to Transaction models

- Make database hooks shared in isomorphic core
2016-12-01 18:41:46 -08:00
Karim Hamidou
15cfe2cec0 Strip the '[Gmail]/' prefix from folder names.
Summary:
T7253 has two related parts:
1. Stripping the '[Gmail]/' prefix from any canonical Gmail folders
2. Handling nested IMAP folders.

This diff fixes 1. Changes for 2. are forthcoming.

Test Plan: Tested manually. Checked that the part was stripped from the N1 folder list.

Reviewers: juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3475
2016-12-01 16:49:09 -08:00
Juan Tejada
d650be5429 [local-sync] Refactor tasks and /actually/ fix moving between Gmail folders
Moving to between gmail folders (all, spam, trash) or moving to inbox,
involves changing labels /and/ folders, simultaneously. For this I added
a task to perform both operations, and apply labels first before
attempting to move the folder
2016-12-01 16:37:49 -08:00
Ben Gotow
4b4ab726e2 [💄] fix eslint issues before they get overwhelming 2016-12-01 15:37:48 -08:00
Karim Hamidou
aa49f85980 Fix bug where we wouldn't create an id for file parts. 2016-12-01 15:30:01 -08:00
Juan Tejada
2d932dd090 [*] Generate persistent id for Accounts based on email+settings
This also ensures that we can use the accountId as part of the ids for
other models, like labels and folders

Before this commit, labels and folders with the same name for different
accounts would have the same id, screwing things up in n1
2016-12-01 14:51:33 -08:00
Ben Gotow
47d8614ed7 [local-sync] Ship first sync metrics to honeycomb 2016-12-01 14:23:52 -08:00
Juan Tejada
ce27b0db2a [local-sync] Fix setlabels task
Make sure we /remove/ the labels when we get an empty set of label ids
2016-12-01 14:03:19 -08:00
Evan Morikawa
aba77ca637 [cloud-api] add npm start for pm2 launch 2016-12-01 16:33:05 -05:00
Juan Tejada
b43be69cd3 [local-sync] Give better name to thread folders task 2016-12-01 13:30:47 -08:00
Juan Tejada
8939364dc3 [local-sync]: Add distinct ids to folders and labels 2016-12-01 11:57:29 -08:00
Ben Gotow
209122c346 [cloud-api] Fix more broken package.jsons 2016-12-01 11:39:25 -08:00
Ben Gotow
0aac73b6d2 [cloud-api] New honeycomb ingestion endpoint 2016-12-01 11:21:28 -08:00
Karim Hamidou
47e0683cac First step to getting persistent id in K2
Summary:
This diff adds persistent unique ids for messages and contacts. For messages, we just take a hash of the headers. For contacts, we hash the contact's email address.

This diff bundles a couple of tiny fixes too, like always trying to restart an account's sync, even after an exception.

Note that since there's no reliable way to have persistent ids for threads, we'll have to change our code to use message ids instead. Alas, that's a story for another (massive) diff.

Test Plan: Tested manually.

Reviewers: bengotow

Reviewed By: bengotow

Differential Revision: https://phab.nylas.com/D3468
2016-12-01 11:15:13 -08:00
Ben Gotow
3d79f9b8be [local-sync] Run message processor for one message at a time
This avoids issues that arise when we process two messages on the same thread concurrently!
2016-11-30 17:26:23 -08:00
Ben Gotow
94722f27ab [local-sync] Fix “Add INBOX label” issue 2016-11-30 16:26:53 -08:00
Ben Gotow
d8703b90c4 [local-sync] Fix deletion of labels 2016-11-30 16:26:53 -08:00
Ben Gotow
bf9db2738c [local-sync] Fix ingestion of flags and labels 2016-11-30 16:26:53 -08:00
Ben Gotow
935d8cf141 [local-sync] Fix transaction log to only ignore syncState 2016-11-30 16:25:28 -08:00
Ben Gotow
2476cae297 [local-sync] Sync a bit faster for testing 2016-11-30 16:25:28 -08:00
Juan Tejada
1470565bcf [local-sync] Fix observable for deltas
We were missing a return :'(
2016-11-30 16:12:35 -08:00
Ben Gotow
1e01878e5a fix(sync): Fix attribute updates applying to threads 2016-11-30 13:55:49 -08:00
Evan Morikawa
a36b1e1f28 [*] add delta endpoints and DRY deltaStreamBuilder
[cloud-core] add objectType to Metadata

[*] binding fixes to delta stream
2016-11-30 13:41:09 -05:00
Juan Tejada
a37b90a209 [local-sync] Remove unecessary db queries in detect-thread 2016-11-30 10:17:59 -08:00
Ben Gotow
ece519553f fix(sync): Issue with CONDSTORE args not passed to fetch 2016-11-29 18:02:25 -08:00
Ben Gotow
82f505b659 refactor(sync-worker): Convert IMAP operations to async/await 2016-11-29 17:04:01 -08:00
Ben Gotow
fb9d770cc6 refactor(sync-worker) async/await for Karim 2016-11-29 16:20:18 -08:00
Ben Gotow
0c79ebb86a [local-sync] Synchronize K2 accounts with N1 accounts 2016-11-29 16:07:33 -08:00
Karim Hamidou
0879b6390e Make db reset synchronous. 2016-11-29 15:13:42 -08:00
Juan Tejada
20be7cc513 [local-sync] properly save folders when detecting thread 2016-11-29 15:04:45 -08:00
Karim Hamidou
ac818f30fe Add a shortcut to restart the sync for an account from scratch. 2016-11-29 14:59:37 -08:00
Ben Gotow
48fff4f4e1 fix(dashboard): Remove sync policy 2016-11-29 14:58:52 -08:00
Juan Tejada
b459076415 [local-sync] minor fix 2016-11-29 12:52:36 -08:00
Juan Tejada
b7b0808689 [local-sync]: Load all db models 2016-11-29 12:38:49 -08:00
Juan Tejada
2fe5f73401 [iso-core] Fix dependencyin model 2016-11-29 11:56:00 -08:00
Juan Tejada
a9be3bb3ed [local-sync]: Refactor message ingestion to write new messages only once
This will only generate 1 delta per new message instead of 2
Adds a MessageFactory to create messages from imapmessage objects, and
unifies and cleans up that logic
Also modified the message-processor to be more suited for local sync,
made emphasis that it's only for processing new messages, and got rid of
the while "pipeline" concept

- TODO message-processor specs are broken
2016-11-29 11:24:27 -08:00
Evan Morikawa
fc24ffea63 [isomorphic-core]: move Transaction model to isomorphic-core 2016-11-29 13:57:19 -05:00
Ben Gotow
f804c53522 [dashboard] Put dashboard into work window 2016-11-28 18:02:42 -08:00
Halla Moore
10f62f6b5a [local-sync, iso-core] feat(send): Support basic sending for gmail accounts 2016-11-28 14:46:31 -08:00
Evan Morikawa
f08e03803c Readme updates 2016-11-28 14:13:23 -08:00
Ben Gotow
217ba1e020 [sync]: Sync again immediately if more emails remain 2016-11-28 12:00:57 -08:00
Juan Tejada
f9b6e63958 [local-sync] Support parsing 8bit encoded messages 2016-11-28 11:41:16 -08:00