Mailspring/internal_packages/worker-sync/lib/nylas-sync-worker-pool.coffee

157 lines
5.4 KiB
CoffeeScript
Raw Normal View History

_ = require 'underscore'
{NylasAPI,
Actions,
AccountStore,
DatabaseStore,
feat(win): faster popout windows Summary: This diff is designed to dramatically speed up new window load time for all window types and reduce memory consumption of our hot windows. Before this diff, windows loaded in ~3 seconds. They now boot in a couple hundred milliseconds without requiring to keep hot windows around for each and every type of popout window we want to load quickly. One of the largest bottlenecks was the `require`ing and initializing of everything in `NylasExports`. I changed `NylasExports` to be entirely lazily-loaded. Drafts and tasks now register their constructors with a `StoreRegistry` and the `TaskRegistry`. This lets us explicitly choose a time to activate these stores in the window initalization instead of whenever nylas-exports happens to be required first. Before, NylasExports was required first when components were first rendering. This made initial render extremely slow and made the proposed time picker popout slow. By moving require into the very initial window boot, we can create a new scheme of hot windows that are "half loaded". All of the expensive require-ing and store initialization is done. All we need to do is activate the packages for just the one window. This means that the hot window scheme needs to fundamentally change from have fully pre-loaded windows, to having half-loaded empty hot windows that can get their window props overridden again. This led to a major refactor of the WindowManager to support this new window scheme. Along the way the API of WindowManager was significantly simplifed. Instead of a bunch of special-cased windows, there are now consistent interfaces to get and `ensure` windows are created and displayed. This DRYed up a lot of repeated logic around showing or creating core windows. This also allowed the consolidation of the core window configurations into one place for much easier reasoning about what's getting booted up. When a hot window goes "live" and gets populated, we simply change the `windowType`. This now re-triggers the loading of all of the packages for the window. All of the loading time is now just for the packages that window requires since core Nylas is there thanks to the hot window mechanism. Unfortunately loading all of the packages for the composer was still unnaceptably slow. The major issue was that all of the composer plugins were taking a long time to process and initialize. The solution was to have the main composer load first, then trigger another window load settings change to change the `windowType` that loads in all of the plugins. Another major bottleneck was the `RetinaImg` name lookup on disk. This requires traversing the entire static folder synchronously on boot. This is now done once when the main window loads and saved in a cache in the browser process. Any secondary windows simply ask the backend for this cache and save the filesystem access time. The Paper Doc below is the current set of manual tests I'm doing to make sure no window interactions (there are a lot of them!) regressed. Test Plan: https://paper.dropbox.com/doc/Window-Refactor-UYsgvjgdXgVlTw8nXTr9h Reviewers: juan, bengotow Reviewed By: bengotow Differential Revision: https://phab.nylas.com/D2916
2016-04-23 04:30:42 +08:00
MailRulesProcessor} = require 'nylas-exports'
NylasSyncWorker = require './nylas-sync-worker'
class NylasSyncWorkerPool
constructor: ->
@_workers = []
feat(mail-rules): Per-account mail rules filter incoming, existing mail Summary: Originally, this was going to be a totally independent package, but I wasn't able to isolate the functionality and get it tied in to the delta-stream consumption. Here's how it currently works: - The preferences package has a new tab which allows you to edit mail filters. Filters are saved in a new core store, and a new stock component (ScenarioEditor) renders the editor. The editor takes a set of templates that define a value space, and outputs a valid set of values. - A new MailFilterProcessor takes messages and creates tasks to apply the actions from the MailFiltersStore. - The worker-sync package now uses the MailFilterProcessor to apply filters /before/ it calls didPassivelyReceiveNewModels, so filtrs are applied before any notifications are created. - A new task, ReprocessMailFiltersTask allows you to run filters on all of your existing mail. It leverages the existing TaskQueue architecture to: a) resume where it left off if you quit midway, b) be queryable (for status) from all windows and c) cancelable. The TaskQueue is a bit strange because it runs performLocal and performRemote very differently, and I had to use `performRemote`. (todo refactor soon.) This diff also changes the EditableList a bit to behave like a controlled component and render focused / unfocused states. Test Plan: Run tests, only for actual filter processing atm. Reviewers: juan, evan Reviewed By: evan Differential Revision: https://phab.nylas.com/D2379
2015-12-23 15:19:32 +08:00
AccountStore.listen(@_onAccountsChanged, @)
@_onAccountsChanged()
_onAccountsChanged: ->
refactor(env): new NylasEnv global Converted all references of global atom to NylasEnv Temporary rename atom.io find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.io/temporaryAtomIoReplacement/g' atom.config to NylasEnv.config find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.config/NylasEnv.config/g' atom.packages -> NylasEnv.packages atom.commands -> NylasEnv.commands atom.getLoadSettings find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.commands/NylasEnv.commands/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getLoadSettings/NylasEnv.getLoadSettings/g' More common atom methods find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.styles/NylasEnv.styles/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.emitError/NylasEnv.emitError/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.inSpecMode/NylasEnv.inSpecMode/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.inDevMode/NylasEnv.inDevMode/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getWindowType/NylasEnv.getWindowType/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.displayWindow/NylasEnv.displayWindow/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.quit/NylasEnv.quit/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.close/NylasEnv.close/g' More atom method changes find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.keymaps/NylasEnv.keymaps/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.hide/NylasEnv.hide/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getCurrentWindow/NylasEnv.getCurrentWindow/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.menu/NylasEnv.menu/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getConfigDirPath/NylasEnv.getConfigDirPath/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.isMainWindow/NylasEnv.isMainWindow/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.finishUnload/NylasEnv.finishUnload/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.isWorkWindow/NylasEnv.isWorkWindow/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.showSaveDialog/NylasEnv.showSaveDialog/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.append/NylasEnv.append/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.confirm/NylasEnv.confirm/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.clipboard/NylasEnv.clipboard/g' find -E . -regex ".*\.(coffee|cjsx|js|md|cmd|es6)" -print0 | xargs -0 sed -i "" 's/atom.getVersion/NylasEnv.getVersion/g' More atom renaming Rename atom methods More atom methods Fix grunt config variable Change atom.cmd to N1.cmd Rename atom.coffee and atom.js to nylas-env.coffee nylas-env.js Fix atom global reference in specs manually Fix atom requires Change engine from atom to nylas got rid of global/nylas-env rename to nylas-win-bootup Fix onWindowPropsChanged to onWindowPropsReceived fix nylas-workspace atom-text-editor to nylas-theme-wrap atom-text-editor -> nylas-theme-wrap Replacing atom keyword AtomWindow -> NylasWindow Replace Atom -> N1 Rename atom items nylas.asar -> atom.asar Remove more atom references Remove 6to5 references Remove license exception for atom
2015-11-12 02:25:11 +08:00
return if NylasEnv.inSpecMode()
accounts = AccountStore.accounts()
workers = _.map(accounts, @workerForAccount)
# Stop the workers that are not in the new workers list.
# These accounts are no longer in our database, so we shouldn't
# be listening.
old = _.without(@_workers, workers...)
worker.cleanup() for worker in old
@_workers = workers
workers: =>
@_workers
workerForAccount: (account) =>
worker = _.find @_workers, (c) -> c.account().id is account.id
return worker if worker
worker = new NylasSyncWorker(NylasAPI, account)
connection = worker.connection()
connection.onDeltas (deltas) =>
@_handleDeltas(deltas)
@_workers.push(worker)
worker.start()
worker
_cleanupAccountWorkers: ->
for worker in @_workers
worker.cleanup()
@_workers = []
_handleDeltas: (deltas) ->
Actions.longPollReceivedRawDeltas(deltas)
Actions.longPollReceivedRawDeltasPing(deltas.length)
# Create a (non-enumerable) reference from the attributes which we carry forward
# back to their original deltas. This allows us to mark the deltas that the
# app ignores later in the process.
deltas.forEach (delta) ->
if delta.attributes
Object.defineProperty(delta.attributes, '_delta', { get: -> delta })
{create, modify, destroy} = @_clusterDeltas(deltas)
# Remove any metadata deltas. These have to be handled at the end, since metadata
# is stored within the object that it points to (which may not exist yet)
metadata = []
for deltas in [create, modify]
if deltas['metadata']
metadata = metadata.concat(_.values(deltas['metadata']))
delete deltas['metadata']
# Remove any account deltas, which are only used to notify broken/fixed sync state
# on accounts
delete create['account']
delete destroy['account']
if modify['account']
@_handleAccountDeltas(_.values(modify['account']))
delete modify['account']
# Apply all the deltas to create objects. Gets promises for handling
# each type of model in the `create` hash, waits for them all to resolve.
create[type] = NylasAPI._handleModelResponse(_.values(dict)) for type, dict of create
Promise.props(create).then (created) =>
# Apply all the deltas to modify objects. Gets promises for handling
# each type of model in the `modify` hash, waits for them all to resolve.
modify[type] = NylasAPI._handleModelResponse(_.values(dict)) for type, dict of modify
Promise.props(modify).then (modified) =>
Promise.all(@_handleDeltaMetadata(metadata)).then =>
# Now that we've persisted creates/updates, fire an action
# that allows other parts of the app to update based on new models
# (notifications)
if _.flatten(_.values(created)).length > 0
MailRulesProcessor.processMessages(created['message'] ? []).finally =>
Actions.didPassivelyReceiveNewModels(created)
# Apply all of the deletions
destroyPromises = destroy.map(@_handleDeltaDeletion)
Promise.settle(destroyPromises).then =>
Actions.longPollProcessedDeltas()
_clusterDeltas: (deltas) ->
# Group deltas by object type so we can mutate the cache efficiently.
# NOTE: This code must not just accumulate creates, modifies and destroys
# but also de-dupe them. We cannot call "persistModels(itemA, itemA, itemB)"
# or it will throw an exception - use the last received copy of each model
# we see.
create = {}
modify = {}
destroy = []
for delta in deltas
if delta.event is 'create'
create[delta.object] ||= {}
create[delta.object][delta.attributes.id] = delta.attributes
else if delta.event is 'modify'
modify[delta.object] ||= {}
modify[delta.object][delta.attributes.id] = delta.attributes
else if delta.event is 'delete'
destroy.push(delta)
{create, modify, destroy}
_handleDeltaMetadata: (metadata) =>
metadata.map (metadatum) =>
klass = NylasAPI._apiObjectToClassMap[metadatum.object_type]
DatabaseStore.inTransaction (t) =>
t.find(klass, metadatum.object_id).then (model) ->
return Promise.resolve() unless model
model = model.applyPluginMetadata(metadatum.application_id, metadatum.value)
localMetadatum = model.metadataObjectForPluginId(metadatum.application_id)
localMetadatum.version = metadatum.version
t.persistModel(model)
_handleAccountDeltas: (deltas) =>
for delta in deltas
Actions.updateAccount(delta.account_id, {syncState: delta.sync_state})
2016-06-08 03:53:05 +08:00
if delta.sync_state isnt "running"
Actions.recordUserEvent('Account Sync Errored', {
accountId: delta.account_id
syncState: delta.sync_state
})
_handleDeltaDeletion: (delta) =>
klass = NylasAPI._apiObjectToClassMap[delta.object]
return unless klass
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
DatabaseStore.inTransaction (t) =>
t.find(klass, delta.id).then (model) ->
return Promise.resolve() unless model
return t.unpersistModel(model)
feat(mail-rules): Per-account mail rules filter incoming, existing mail Summary: Originally, this was going to be a totally independent package, but I wasn't able to isolate the functionality and get it tied in to the delta-stream consumption. Here's how it currently works: - The preferences package has a new tab which allows you to edit mail filters. Filters are saved in a new core store, and a new stock component (ScenarioEditor) renders the editor. The editor takes a set of templates that define a value space, and outputs a valid set of values. - A new MailFilterProcessor takes messages and creates tasks to apply the actions from the MailFiltersStore. - The worker-sync package now uses the MailFilterProcessor to apply filters /before/ it calls didPassivelyReceiveNewModels, so filtrs are applied before any notifications are created. - A new task, ReprocessMailFiltersTask allows you to run filters on all of your existing mail. It leverages the existing TaskQueue architecture to: a) resume where it left off if you quit midway, b) be queryable (for status) from all windows and c) cancelable. The TaskQueue is a bit strange because it runs performLocal and performRemote very differently, and I had to use `performRemote`. (todo refactor soon.) This diff also changes the EditableList a bit to behave like a controlled component and render focused / unfocused states. Test Plan: Run tests, only for actual filter processing atm. Reviewers: juan, evan Reviewed By: evan Differential Revision: https://phab.nylas.com/D2379
2015-12-23 15:19:32 +08:00
module.exports = NylasSyncWorkerPool