Mailspring/packages/client-app/internal_packages/deltas/spec/delta-processor-spec.coffee

231 lines
8.8 KiB
CoffeeScript
Raw Normal View History

_ = require 'underscore'
fs = require 'fs'
path = require 'path'
{NylasAPIHelpers,
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
Thread,
DatabaseTransaction,
Actions,
Message,
Thread} = require 'nylas-exports'
feat(mail-rules): Per-account mail rules filter incoming, existing mail Summary: Originally, this was going to be a totally independent package, but I wasn't able to isolate the functionality and get it tied in to the delta-stream consumption. Here's how it currently works: - The preferences package has a new tab which allows you to edit mail filters. Filters are saved in a new core store, and a new stock component (ScenarioEditor) renders the editor. The editor takes a set of templates that define a value space, and outputs a valid set of values. - A new MailFilterProcessor takes messages and creates tasks to apply the actions from the MailFiltersStore. - The worker-sync package now uses the MailFilterProcessor to apply filters /before/ it calls didPassivelyReceiveNewModels, so filtrs are applied before any notifications are created. - A new task, ReprocessMailFiltersTask allows you to run filters on all of your existing mail. It leverages the existing TaskQueue architecture to: a) resume where it left off if you quit midway, b) be queryable (for status) from all windows and c) cancelable. The TaskQueue is a bit strange because it runs performLocal and performRemote very differently, and I had to use `performRemote`. (todo refactor soon.) This diff also changes the EditableList a bit to behave like a controlled component and render focused / unfocused states. Test Plan: Run tests, only for actual filter processing atm. Reviewers: juan, evan Reviewed By: evan Differential Revision: https://phab.nylas.com/D2379
2015-12-23 15:19:32 +08:00
DeltaProcessor = require('../lib/delta-processor').default
fixturesPath = path.resolve(__dirname, 'fixtures')
describe "DeltaProcessor", ->
describe "handleDeltas", ->
beforeEach ->
@sampleDeltas = JSON.parse(fs.readFileSync("#{fixturesPath}/sample.json"))
@sampleClustered = JSON.parse(fs.readFileSync("#{fixturesPath}/sample-clustered.json"))
it "should immediately fire the received raw deltas event", ->
spyOn(Actions, 'longPollReceivedRawDeltas')
spyOn(DeltaProcessor, '_clusterDeltas').andReturn({create: {}, modify: {}, destroy: []})
waitsForPromise ->
2017-02-22 00:56:47 +08:00
DeltaProcessor.process(@sampleDeltas, {source: 'n1Cloud'})
runs ->
expect(Actions.longPollReceivedRawDeltas).toHaveBeenCalled()
xit "should call helper methods for all creates first, then modifications, then destroys", ->
spyOn(Actions, 'longPollProcessedDeltas')
handleDeltaDeletionPromises = []
resolveDeltaDeletionPromises = ->
fn() for fn in handleDeltaDeletionPromises
handleDeltaDeletionPromises = []
spyOn(DeltaProcessor, '_handleDestroyDelta').andCallFake ->
new Promise (resolve, reject) ->
handleDeltaDeletionPromises.push(resolve)
handleModelResponsePromises = []
resolveModelResponsePromises = ->
fn() for fn in handleModelResponsePromises
handleModelResponsePromises = []
spyOn(NylasAPIHelpers, 'handleModelResponse').andCallFake ->
new Promise (resolve, reject) ->
handleModelResponsePromises.push(resolve)
spyOn(DeltaProcessor, '_clusterDeltas').andReturn(JSON.parse(JSON.stringify(@sampleClustered)))
DeltaProcessor.process(@sampleDeltas)
createTypes = Object.keys(@sampleClustered['create'])
expect(NylasAPIHelpers.handleModelResponse.calls.length).toEqual(createTypes.length)
expect(NylasAPIHelpers.handleModelResponse.calls[0].args[0]).toEqual(_.values(@sampleClustered['create'][createTypes[0]]))
expect(DeltaProcessor._handleDestroyDelta.calls.length).toEqual(0)
NylasAPIHelpers.handleModelResponse.reset()
resolveModelResponsePromises()
advanceClock()
modifyTypes = Object.keys(@sampleClustered['modify'])
expect(NylasAPIHelpers.handleModelResponse.calls.length).toEqual(modifyTypes.length)
expect(NylasAPIHelpers.handleModelResponse.calls[0].args[0]).toEqual(_.values(@sampleClustered['modify'][modifyTypes[0]]))
expect(DeltaProcessor._handleDestroyDelta.calls.length).toEqual(0)
NylasAPIHelpers.handleModelResponse.reset()
resolveModelResponsePromises()
advanceClock()
destroyCount = @sampleClustered['destroy'].length
expect(DeltaProcessor._handleDestroyDelta.calls.length).toEqual(destroyCount)
expect(DeltaProcessor._handleDestroyDelta.calls[0].args[0]).toEqual(@sampleClustered['destroy'][0])
expect(Actions.longPollProcessedDeltas).not.toHaveBeenCalled()
resolveDeltaDeletionPromises()
advanceClock()
expect(Actions.longPollProcessedDeltas).toHaveBeenCalled()
describe "clusterDeltas", ->
beforeEach ->
@sampleDeltas = JSON.parse(fs.readFileSync("#{fixturesPath}/sample.json"))
@expectedClustered = JSON.parse(fs.readFileSync("#{fixturesPath}/sample-clustered.json"))
it "should collect create/modify events into a hash by model type", ->
{create, modify} = DeltaProcessor._clusterDeltas(@sampleDeltas)
expect(create).toEqual(@expectedClustered.create)
expect(modify).toEqual(@expectedClustered.modify)
it "should collect destroys into an array", ->
{destroy} = DeltaProcessor._clusterDeltas(@sampleDeltas)
expect(destroy).toEqual(@expectedClustered.destroy)
describe "handleDeltaDeletion", ->
beforeEach ->
@thread = new Thread(id: 'idhere')
@delta =
"cursor": "bb95ddzqtr2gpmvgrng73t6ih",
"object": "thread",
"event": "delete",
"objectId": @thread.id,
"timestamp": "2015-08-26T17:36:45.297Z"
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
spyOn(DatabaseTransaction.prototype, 'unpersistModel')
it "should resolve if the object cannot be found", ->
spyOn(DatabaseTransaction.prototype, 'find').andCallFake (klass, id) =>
return Promise.resolve(null)
waitsForPromise =>
DeltaProcessor._handleDestroyDelta(@delta)
runs =>
expect(DatabaseTransaction.prototype.find).toHaveBeenCalledWith(Thread, 'idhere')
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
expect(DatabaseTransaction.prototype.unpersistModel).not.toHaveBeenCalled()
it "should call unpersistModel if the object exists", ->
spyOn(DatabaseTransaction.prototype, 'find').andCallFake (klass, id) =>
return Promise.resolve(@thread)
waitsForPromise =>
DeltaProcessor._handleDestroyDelta(@delta)
runs =>
expect(DatabaseTransaction.prototype.find).toHaveBeenCalledWith(Thread, 'idhere')
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
expect(DatabaseTransaction.prototype.unpersistModel).toHaveBeenCalledWith(@thread)
describe "handleModelResponse", ->
2015-10-02 12:39:44 +08:00
# SEE spec/nylas-api-spec.coffee
describe "receives metadata deltas", ->
beforeEach ->
@stubDB = {}
spyOn(DatabaseTransaction.prototype, 'find').andCallFake (klass, id) =>
return @stubDB[id]
spyOn(DatabaseTransaction.prototype, 'findAll').andCallFake (klass, where) =>
ids = where.id
models = []
ids.forEach (id) =>
model = @stubDB[id]
if model
models.push(model)
return models
spyOn(DatabaseTransaction.prototype, 'persistModels').andCallFake (models) =>
models.forEach (model) =>
@stubDB[model.id] = model
return Promise.resolve()
@messageMetadataDelta =
id: 519,
event: "create",
object: "metadata",
objectId: 8876,
changedFields: ["version", "object"],
attributes:
id: 8876,
value: {link_clicks: 1},
object: "metadata",
version: 2,
plugin_id: "link-tracking",
object_id: '2887',
object_type: "message",
account_id: 2
@threadMetadataDelta =
id: 392,
event: "create",
object: "metadata",
objectId: 3845,
changedFields: ["version", "object"],
attributes:
id: 3845,
value: {shouldNotify: true},
object: "metadata",
version: 2,
plugin_id: "send-reminders",
object_id: 't:3984',
object_type: "thread"
account_id: 2,
it "saves metadata to existing Messages", ->
message = new Message({serverId: @messageMetadataDelta.attributes.object_id})
@stubDB[message.id] = message
waitsForPromise =>
DeltaProcessor.process([@messageMetadataDelta])
runs ->
message = @stubDB[message.id] # refresh reference
expect(message.pluginMetadata.length).toEqual(1)
expect(message.metadataForPluginId('link-tracking')).toEqual({link_clicks: 1})
it "saves metadata to existing Threads", ->
thread = new Thread({serverId: @threadMetadataDelta.attributes.object_id})
@stubDB[thread.id] = thread
waitsForPromise =>
DeltaProcessor.process([@threadMetadataDelta])
runs ->
thread = @stubDB[thread.id] # refresh reference
expect(thread.pluginMetadata.length).toEqual(1)
expect(thread.metadataForPluginId('send-reminders')).toEqual({shouldNotify: true})
it "knows how to reconcile different thread ids", ->
thread = new Thread({serverId: 't:1948'})
@stubDB[thread.id] = thread
message = new Message({
serverId: @threadMetadataDelta.attributes.object_id.substring(2),
threadId: thread.id
})
@stubDB[message.id] = message
waitsForPromise =>
DeltaProcessor.process([@threadMetadataDelta])
runs ->
thread = @stubDB[thread.id] # refresh reference
expect(thread.pluginMetadata.length).toEqual(1)
expect(thread.metadataForPluginId('send-reminders')).toEqual({shouldNotify: true})
it "creates ghost Messages if necessary", ->
waitsForPromise =>
DeltaProcessor.process([@messageMetadataDelta])
runs ->
message = @stubDB[@messageMetadataDelta.attributes.object_id]
expect(message).toBeDefined()
expect(message.pluginMetadata.length).toEqual(1)
expect(message.metadataForPluginId('link-tracking')).toEqual({link_clicks: 1})
it "creates ghost Threads if necessary", ->
waitsForPromise =>
DeltaProcessor.process([@threadMetadataDelta])
runs ->
thread = @stubDB[@threadMetadataDelta.attributes.object_id]
expect(thread.pluginMetadata.length).toEqual(1)
expect(thread.metadataForPluginId('send-reminders')).toEqual({shouldNotify: true})