Mailspring/internal_packages/worker-sync/lib/nylas-sync-worker-pool.coffee

131 lines
4.3 KiB
CoffeeScript
Raw Normal View History

_ = require 'underscore'
{NylasAPI,
Actions,
AccountStore,
DatabaseStore,
feat(mail-rules): Per-account mail rules filter incoming, existing mail Summary: Originally, this was going to be a totally independent package, but I wasn't able to isolate the functionality and get it tied in to the delta-stream consumption. Here's how it currently works: - The preferences package has a new tab which allows you to edit mail filters. Filters are saved in a new core store, and a new stock component (ScenarioEditor) renders the editor. The editor takes a set of templates that define a value space, and outputs a valid set of values. - A new MailFilterProcessor takes messages and creates tasks to apply the actions from the MailFiltersStore. - The worker-sync package now uses the MailFilterProcessor to apply filters /before/ it calls didPassivelyReceiveNewModels, so filtrs are applied before any notifications are created. - A new task, ReprocessMailFiltersTask allows you to run filters on all of your existing mail. It leverages the existing TaskQueue architecture to: a) resume where it left off if you quit midway, b) be queryable (for status) from all windows and c) cancelable. The TaskQueue is a bit strange because it runs performLocal and performRemote very differently, and I had to use `performRemote`. (todo refactor soon.) This diff also changes the EditableList a bit to behave like a controlled component and render focused / unfocused states. Test Plan: Run tests, only for actual filter processing atm. Reviewers: juan, evan Reviewed By: evan Differential Revision: https://phab.nylas.com/D2379
2015-12-23 15:19:32 +08:00
MailRulesProcessor,
DatabaseObjectRegistry} = require 'nylas-exports'
NylasLongConnection = require './nylas-long-connection'
NylasSyncWorker = require './nylas-sync-worker'
class NylasSyncWorkerPool
constructor: ->
@_workers = []
feat(mail-rules): Per-account mail rules filter incoming, existing mail Summary: Originally, this was going to be a totally independent package, but I wasn't able to isolate the functionality and get it tied in to the delta-stream consumption. Here's how it currently works: - The preferences package has a new tab which allows you to edit mail filters. Filters are saved in a new core store, and a new stock component (ScenarioEditor) renders the editor. The editor takes a set of templates that define a value space, and outputs a valid set of values. - A new MailFilterProcessor takes messages and creates tasks to apply the actions from the MailFiltersStore. - The worker-sync package now uses the MailFilterProcessor to apply filters /before/ it calls didPassivelyReceiveNewModels, so filtrs are applied before any notifications are created. - A new task, ReprocessMailFiltersTask allows you to run filters on all of your existing mail. It leverages the existing TaskQueue architecture to: a) resume where it left off if you quit midway, b) be queryable (for status) from all windows and c) cancelable. The TaskQueue is a bit strange because it runs performLocal and performRemote very differently, and I had to use `performRemote`. (todo refactor soon.) This diff also changes the EditableList a bit to behave like a controlled component and render focused / unfocused states. Test Plan: Run tests, only for actual filter processing atm. Reviewers: juan, evan Reviewed By: evan Differential Revision: https://phab.nylas.com/D2379
2015-12-23 15:19:32 +08:00
AccountStore.listen(@_onAccountsChanged, @)
@_onAccountsChanged()
_onAccountsChanged: ->
refactor(env): new NylasEnv global Converted all references of global atom to NylasEnv Temporary rename atom.io find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.io/temporaryAtomIoReplacement/g' atom.config to NylasEnv.config find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.config/NylasEnv.config/g' atom.packages -> NylasEnv.packages atom.commands -> NylasEnv.commands atom.getLoadSettings find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.commands/NylasEnv.commands/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getLoadSettings/NylasEnv.getLoadSettings/g' More common atom methods find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.styles/NylasEnv.styles/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.emitError/NylasEnv.emitError/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.inSpecMode/NylasEnv.inSpecMode/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.inDevMode/NylasEnv.inDevMode/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getWindowType/NylasEnv.getWindowType/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.displayWindow/NylasEnv.displayWindow/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.quit/NylasEnv.quit/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.close/NylasEnv.close/g' More atom method changes find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.keymaps/NylasEnv.keymaps/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.hide/NylasEnv.hide/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getCurrentWindow/NylasEnv.getCurrentWindow/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.menu/NylasEnv.menu/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getConfigDirPath/NylasEnv.getConfigDirPath/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.isMainWindow/NylasEnv.isMainWindow/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.finishUnload/NylasEnv.finishUnload/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.isWorkWindow/NylasEnv.isWorkWindow/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.showSaveDialog/NylasEnv.showSaveDialog/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.append/NylasEnv.append/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.confirm/NylasEnv.confirm/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.clipboard/NylasEnv.clipboard/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getVersion/NylasEnv.getVersion/g' More atom renaming Rename atom methods More atom methods Fix grunt config variable Change atom.cmd to N1.cmd Rename atom.coffee and atom.js to nylas-env.coffee nylas-env.js Fix atom global reference in specs manually Fix atom requires Change engine from atom to nylas got rid of global/nylas-env rename to nylas-win-bootup Fix onWindowPropsChanged to onWindowPropsReceived fix nylas-workspace atom-text-editor to nylas-theme-wrap atom-text-editor -> nylas-theme-wrap Replacing atom keyword AtomWindow -> NylasWindow Replace Atom -> N1 Rename atom items nylas.asar -> atom.asar Remove more atom references Remove 6to5 references Remove license exception for atom
2015-11-12 02:25:11 +08:00
return if NylasEnv.inSpecMode()
accounts = AccountStore.accounts()
workers = _.map(accounts, @workerForAccount)
# Stop the workers that are not in the new workers list.
# These accounts are no longer in our database, so we shouldn't
# be listening.
old = _.without(@_workers, workers...)
worker.cleanup() for worker in old
@_workers = workers
workers: =>
@_workers
workerForAccount: (account) =>
worker = _.find @_workers, (c) -> c.account().id is account.id
return worker if worker
worker = new NylasSyncWorker(NylasAPI, account)
connection = worker.connection()
connection.onStateChange (state) ->
Actions.longPollStateChanged({accountId: account.id, state: state})
if state == NylasLongConnection.State.Connected
## TODO use OfflineStatusStore
Actions.longPollConnected()
else
## TODO use OfflineStatusStore
Actions.longPollOffline()
connection.onDeltas (deltas) =>
@_handleDeltas(deltas)
@_workers.push(worker)
worker.start()
worker
_cleanupAccountWorkers: ->
for worker in @_workers
worker.cleanup()
@_workers = []
_handleDeltas: (deltas) ->
Actions.longPollReceivedRawDeltas(deltas)
Actions.longPollReceivedRawDeltasPing(deltas.length)
# Create a (non-enumerable) reference from the attributes which we carry forward
# back to their original deltas. This allows us to mark the deltas that the
# app ignores later in the process.
deltas.forEach (delta) ->
if delta.attributes
Object.defineProperty(delta.attributes, '_delta', { get: -> delta })
{create, modify, destroy} = @_clusterDeltas(deltas)
# Apply all the deltas to create objects. Gets promises for handling
# each type of model in the `create` hash, waits for them all to resolve.
create[type] = NylasAPI._handleModelResponse(_.values(dict)) for type, dict of create
Promise.props(create).then (created) =>
# Apply all the deltas to modify objects. Gets promises for handling
# each type of model in the `modify` hash, waits for them all to resolve.
modify[type] = NylasAPI._handleModelResponse(_.values(dict)) for type, dict of modify
Promise.props(modify).then (modified) =>
# Now that we've persisted creates/updates, fire an action
# that allows other parts of the app to update based on new models
# (notifications)
if _.flatten(_.values(created)).length > 0
feat(mail-rules): Per-account mail rules filter incoming, existing mail Summary: Originally, this was going to be a totally independent package, but I wasn't able to isolate the functionality and get it tied in to the delta-stream consumption. Here's how it currently works: - The preferences package has a new tab which allows you to edit mail filters. Filters are saved in a new core store, and a new stock component (ScenarioEditor) renders the editor. The editor takes a set of templates that define a value space, and outputs a valid set of values. - A new MailFilterProcessor takes messages and creates tasks to apply the actions from the MailFiltersStore. - The worker-sync package now uses the MailFilterProcessor to apply filters /before/ it calls didPassivelyReceiveNewModels, so filtrs are applied before any notifications are created. - A new task, ReprocessMailFiltersTask allows you to run filters on all of your existing mail. It leverages the existing TaskQueue architecture to: a) resume where it left off if you quit midway, b) be queryable (for status) from all windows and c) cancelable. The TaskQueue is a bit strange because it runs performLocal and performRemote very differently, and I had to use `performRemote`. (todo refactor soon.) This diff also changes the EditableList a bit to behave like a controlled component and render focused / unfocused states. Test Plan: Run tests, only for actual filter processing atm. Reviewers: juan, evan Reviewed By: evan Differential Revision: https://phab.nylas.com/D2379
2015-12-23 15:19:32 +08:00
MailRulesProcessor.processMessages(created['message'] ? []).finally =>
Actions.didPassivelyReceiveNewModels(created)
# Apply all of the deletions
destroyPromises = destroy.map(@_handleDeltaDeletion)
Promise.settle(destroyPromises).then =>
Actions.longPollProcessedDeltas()
_clusterDeltas: (deltas) ->
# Group deltas by object type so we can mutate the cache efficiently.
# NOTE: This code must not just accumulate creates, modifies and destroys
# but also de-dupe them. We cannot call "persistModels(itemA, itemA, itemB)"
# or it will throw an exception - use the last received copy of each model
# we see.
create = {}
modify = {}
destroy = []
for delta in deltas
if delta.event is 'create'
create[delta.object] ||= {}
create[delta.object][delta.attributes.id] = delta.attributes
else if delta.event is 'modify'
modify[delta.object] ||= {}
modify[delta.object][delta.attributes.id] = delta.attributes
else if delta.event is 'delete'
destroy.push(delta)
{create, modify, destroy}
_handleDeltaDeletion: (delta) =>
klass = NylasAPI._apiObjectToClassMap[delta.object]
return unless klass
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
DatabaseStore.inTransaction (t) =>
t.find(klass, delta.id).then (model) ->
return Promise.resolve() unless model
return t.unpersistModel(model)
feat(mail-rules): Per-account mail rules filter incoming, existing mail Summary: Originally, this was going to be a totally independent package, but I wasn't able to isolate the functionality and get it tied in to the delta-stream consumption. Here's how it currently works: - The preferences package has a new tab which allows you to edit mail filters. Filters are saved in a new core store, and a new stock component (ScenarioEditor) renders the editor. The editor takes a set of templates that define a value space, and outputs a valid set of values. - A new MailFilterProcessor takes messages and creates tasks to apply the actions from the MailFiltersStore. - The worker-sync package now uses the MailFilterProcessor to apply filters /before/ it calls didPassivelyReceiveNewModels, so filtrs are applied before any notifications are created. - A new task, ReprocessMailFiltersTask allows you to run filters on all of your existing mail. It leverages the existing TaskQueue architecture to: a) resume where it left off if you quit midway, b) be queryable (for status) from all windows and c) cancelable. The TaskQueue is a bit strange because it runs performLocal and performRemote very differently, and I had to use `performRemote`. (todo refactor soon.) This diff also changes the EditableList a bit to behave like a controlled component and render focused / unfocused states. Test Plan: Run tests, only for actual filter processing atm. Reviewers: juan, evan Reviewed By: evan Differential Revision: https://phab.nylas.com/D2379
2015-12-23 15:19:32 +08:00
module.exports = NylasSyncWorkerPool