Foundry376/Mailspring
1
1
Fork 0
mirror of https://github.com/Foundry376/Mailspring.git synced 2024-11-12 04:25:31 +08:00
Code Issues Projects Releases Packages Wiki Activity
Mailspring/spec/stores/database-transaction-spec.coffee

300 lines
13 KiB
CoffeeScript
Raw Normal View History

feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
_ = require 'underscore'
DatabaseTransaction / QueryBuilder spec fixes
2016-01-26 03:11:57 +08:00
Category = require '../../src/flux/models/category'
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
Thread = require '../../src/flux/models/thread'
TestModel = require '../fixtures/db-test-model'
ModelQuery = require '../../src/flux/models/query'
DatabaseTransaction = require '../../src/flux/stores/database-transaction'
testMatchers = {'id': 'b'}
testModelInstance = new TestModel(id: "1234")
testModelInstanceA = new TestModel(id: "AAA")
testModelInstanceB = new TestModel(id: "BBB")
describe "DatabaseTransaction", ->
beforeEach ->
@databaseMutationHooks = []
@performed = []
@database =
_query: jasmine.createSpy('database._query').andCallFake (query, values=[], options={}) =>
@performed.push({query, values})
Promise.resolve([])
accumulateAndTrigger: jasmine.createSpy('database.accumulateAndTrigger')
mutationHooks: => @databaseMutationHooks
@transaction = new DatabaseTransaction(@database)
describe "execute", ->
describe "persistModel", ->
it "should throw an exception if the model is not a subclass of Model", ->
expect(=> @transaction.persistModel({id: 'asd', subject: 'bla'})).toThrow()
it "should call through to persistModels", ->
spyOn(@transaction, 'persistModels').andReturn Promise.resolve()
@transaction.persistModel(testModelInstance)
advanceClock()
expect(@transaction.persistModels.callCount).toBe(1)
describe "persistModels", ->
it "should call accumulateAndTrigger with a change that contains the models", ->
runs =>
@transaction.execute (t) =>
t.persistModels([testModelInstanceA, testModelInstanceB])
waitsFor =>
@database.accumulateAndTrigger.callCount > 0
runs =>
change = @database.accumulateAndTrigger.mostRecentCall.args[0]
expect(change).toEqual
objectClass: TestModel.name,
objectIds: [testModelInstanceA.id, testModelInstanceB.id]
objects: [testModelInstanceA, testModelInstanceB]
type:'persist'
it "should call through to _writeModels after checking them", ->
spyOn(@transaction, '_writeModels').andReturn Promise.resolve()
@transaction.persistModels([testModelInstanceA, testModelInstanceB])
advanceClock()
expect(@transaction._writeModels.callCount).toBe(1)
it "should throw an exception if the models are not the same class,\
since it cannot be specified by the trigger payload", ->
DatabaseTransaction / QueryBuilder spec fixes
2016-01-26 03:11:57 +08:00
expect(=> @transaction.persistModels([testModelInstanceA, new Category()])).toThrow()
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
it "should throw an exception if the models are not a subclass of Model", ->
expect(=> @transaction.persistModels([{id: 'asd', subject: 'bla'}])).toThrow()
describe "mutationHooks", ->
beforeEach ->
@beforeShouldThrow = false
@beforeShouldReject = false
@hook =
beforeDatabaseChange: jasmine.createSpy('beforeDatabaseChange').andCallFake =>
throw new Error("beforeShouldThrow") if @beforeShouldThrow
new Promise (resolve, reject) =>
setTimeout =>
return resolve(new Error("beforeShouldReject")) if @beforeShouldReject
resolve("value")
, 1000
afterDatabaseChange: jasmine.createSpy('afterDatabaseChange').andCallFake =>
new Promise (resolve, reject) ->
setTimeout(( => resolve()), 1000)
@databaseMutationHooks.push(@hook)
@writeModelsResolve = null
spyOn(@transaction, '_writeModels').andCallFake =>
new Promise (resolve, reject) =>
@writeModelsResolve = resolve
it "should run pre-mutation hooks, wait to write models, and then run post-mutation hooks", ->
@transaction.persistModels([testModelInstanceA, testModelInstanceB])
advanceClock()
expect(@hook.beforeDatabaseChange).toHaveBeenCalledWith(
@transaction._query,
{
objects: [testModelInstanceA, testModelInstanceB]
objectIds: [testModelInstanceA.id, testModelInstanceB.id]
objectClass: testModelInstanceA.constructor.name
type: 'persist'
},
undefined
)
expect(@transaction._writeModels).not.toHaveBeenCalled()
advanceClock(1100)
advanceClock()
expect(@transaction._writeModels).toHaveBeenCalled()
expect(@hook.afterDatabaseChange).not.toHaveBeenCalled()
@writeModelsResolve()
advanceClock()
advanceClock()
expect(@hook.afterDatabaseChange).toHaveBeenCalledWith(
@transaction._query,
{
objects: [testModelInstanceA, testModelInstanceB]
objectIds: [testModelInstanceA.id, testModelInstanceB.id]
objectClass: testModelInstanceA.constructor.name
type: 'persist'
},
"value"
)
it "should carry on if a pre-mutation hook throws", ->
@beforeShouldThrow = true
@transaction.persistModels([testModelInstanceA, testModelInstanceB])
advanceClock(1000)
expect(@hook.beforeDatabaseChange).toHaveBeenCalled()
advanceClock()
advanceClock()
expect(@transaction._writeModels).toHaveBeenCalled()
it "should carry on if a pre-mutation hook rejects", ->
@beforeShouldReject = true
@transaction.persistModels([testModelInstanceA, testModelInstanceB])
advanceClock(1000)
expect(@hook.beforeDatabaseChange).toHaveBeenCalled()
advanceClock()
advanceClock()
expect(@transaction._writeModels).toHaveBeenCalled()
describe "unpersistModel", ->
it "should delete the model by id", ->
waitsForPromise =>
@transaction.execute =>
@transaction.unpersistModel(testModelInstance)
.then =>
expect(@performed.length).toBe(3)
expect(@performed[0].query).toBe("BEGIN IMMEDIATE TRANSACTION")
expect(@performed[1].query).toBe("DELETE FROM `TestModel` WHERE `id` = ?")
expect(@performed[1].values[0]).toBe('1234')
expect(@performed[2].query).toBe("COMMIT")
it "should call accumulateAndTrigger with a change that contains the model", ->
runs =>
@transaction.execute =>
@transaction.unpersistModel(testModelInstance)
waitsFor =>
@database.accumulateAndTrigger.callCount > 0
runs =>
change = @database.accumulateAndTrigger.mostRecentCall.args[0]
expect(change).toEqual({
objectClass: TestModel.name,
objectIds: [testModelInstance.id]
objects: [testModelInstance],
type:'unpersist'
})
describe "when the model has collection attributes", ->
it "should delete all of the elements in the join tables", ->
TestModel.configureWithCollectionAttribute()
waitsForPromise =>
@transaction.execute (t) =>
t.unpersistModel(testModelInstance)
.then =>
expect(@performed.length).toBe(4)
expect(@performed[0].query).toBe("BEGIN IMMEDIATE TRANSACTION")
fix(db): Remove `-` from table names, which was a bad idea
2016-04-05 08:44:45 +08:00
expect(@performed[2].query).toBe("DELETE FROM `TestModelCategory` WHERE `id` = ?")
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
expect(@performed[2].values[0]).toBe('1234')
expect(@performed[3].query).toBe("COMMIT")
describe "when the model has joined data attributes", ->
it "should delete the element in the joined data table", ->
TestModel.configureWithJoinedDataAttribute()
waitsForPromise =>
@transaction.execute (t) =>
t.unpersistModel(testModelInstance)
.then =>
expect(@performed.length).toBe(4)
expect(@performed[0].query).toBe("BEGIN IMMEDIATE TRANSACTION")
expect(@performed[2].query).toBe("DELETE FROM `TestModelBody` WHERE `id` = ?")
expect(@performed[2].values[0]).toBe('1234')
expect(@performed[3].query).toBe("COMMIT")
describe "_writeModels", ->
it "should compose a REPLACE INTO query to save the model", ->
TestModel.configureWithCollectionAttribute()
@transaction._writeModels([testModelInstance])
perf(db): Use subselect to improve thread list query perf (53%!) Summary: - Use a sub-select query with much better performance to display the thread list - Perform analyze on tables after launch The new query is: ``` SELECT `Thread`.`data` FROM `Thread` WHERE `Thread`.`id` IN (SELECT `id` FROM `ThreadCategory` AS `M26` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','dsnn660af0pmou2gg3nstga8a','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0) ORDER BY `Thread`.`last_message_received_timestamp` DESC; ` 0|0|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) 0|0|0|EXECUTE LIST SUBQUERY 1 1|0|0|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 1|0|0|EXECUTE LIST SUBQUERY 2 0|0|0|USE TEMP B-TREE FOR (only on 200 result items) ``` Which is twice as performant as: ``` SELECT `Thread`.`data` FROM `Thread` INNER JOIN `ThreadCategory` AS `M26` ON `M26`.`id` = `Thread`.`id` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0; 0|0|1|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 0|0|0|EXECUTE LIST SUBQUERY 1 0|1|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) ``` Test Plan: Broken! Reviewers: evan, juan Reviewed By: juan Differential Revision: https://phab.nylas.com/D2869
2016-04-12 04:29:05 +08:00
expect(@performed[0].query).toBe("REPLACE INTO `TestModel` (id,data,client_id,server_id,other) VALUES (?,?,?,?,?)")
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
it "should save the model JSON into the data column", ->
@transaction._writeModels([testModelInstance])
expect(@performed[0].values[1]).toEqual(JSON.stringify(testModelInstance))
describe "when the model defines additional queryable attributes", ->
beforeEach ->
TestModel.configureWithAllAttributes()
@m = new TestModel
id: 'local-6806434c-b0cd'
datetime: new Date()
string: 'hello world',
boolean: true,
number: 15
it "should populate additional columns defined by the attributes", ->
@transaction._writeModels([@m])
expect(@performed[0].query).toBe("REPLACE INTO `TestModel` (id,data,datetime,string-json-key,boolean,number) VALUES (?,?,?,?,?,?)")
it "should use the JSON-form values of the queryable attributes", ->
json = @m.toJSON()
@transaction._writeModels([@m])
values = @performed[0].values
expect(values[2]).toEqual(json['datetime'])
expect(values[3]).toEqual(json['string-json-key'])
expect(values[4]).toEqual(json['boolean'])
expect(values[5]).toEqual(json['number'])
describe "when the model has collection attributes", ->
beforeEach ->
TestModel.configureWithCollectionAttribute()
perf(db): Use subselect to improve thread list query perf (53%!) Summary: - Use a sub-select query with much better performance to display the thread list - Perform analyze on tables after launch The new query is: ``` SELECT `Thread`.`data` FROM `Thread` WHERE `Thread`.`id` IN (SELECT `id` FROM `ThreadCategory` AS `M26` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','dsnn660af0pmou2gg3nstga8a','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0) ORDER BY `Thread`.`last_message_received_timestamp` DESC; ` 0|0|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) 0|0|0|EXECUTE LIST SUBQUERY 1 1|0|0|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 1|0|0|EXECUTE LIST SUBQUERY 2 0|0|0|USE TEMP B-TREE FOR (only on 200 result items) ``` Which is twice as performant as: ``` SELECT `Thread`.`data` FROM `Thread` INNER JOIN `ThreadCategory` AS `M26` ON `M26`.`id` = `Thread`.`id` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0; 0|0|1|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 0|0|0|EXECUTE LIST SUBQUERY 1 0|1|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) ``` Test Plan: Broken! Reviewers: evan, juan Reviewed By: juan Differential Revision: https://phab.nylas.com/D2869
2016-04-12 04:29:05 +08:00
@m = new TestModel(id: 'local-6806434c-b0cd', other: 'other')
DatabaseTransaction / QueryBuilder spec fixes
2016-01-26 03:11:57 +08:00
@m.categories = [new Category(id: 'a'),new Category(id: 'b')]
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
@transaction._writeModels([@m])
it "should delete all association records for the model from join tables", ->
fix(db): Remove `-` from table names, which was a bad idea
2016-04-05 08:44:45 +08:00
expect(@performed[1].query).toBe('DELETE FROM `TestModelCategory` WHERE `id` IN (\'local-6806434c-b0cd\')')
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
perf(db): Use subselect to improve thread list query perf (53%!) Summary: - Use a sub-select query with much better performance to display the thread list - Perform analyze on tables after launch The new query is: ``` SELECT `Thread`.`data` FROM `Thread` WHERE `Thread`.`id` IN (SELECT `id` FROM `ThreadCategory` AS `M26` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','dsnn660af0pmou2gg3nstga8a','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0) ORDER BY `Thread`.`last_message_received_timestamp` DESC; ` 0|0|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) 0|0|0|EXECUTE LIST SUBQUERY 1 1|0|0|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 1|0|0|EXECUTE LIST SUBQUERY 2 0|0|0|USE TEMP B-TREE FOR (only on 200 result items) ``` Which is twice as performant as: ``` SELECT `Thread`.`data` FROM `Thread` INNER JOIN `ThreadCategory` AS `M26` ON `M26`.`id` = `Thread`.`id` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0; 0|0|1|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 0|0|0|EXECUTE LIST SUBQUERY 1 0|1|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) ``` Test Plan: Broken! Reviewers: evan, juan Reviewed By: juan Differential Revision: https://phab.nylas.com/D2869
2016-04-12 04:29:05 +08:00
it "should insert new association records into join tables in a single query, and include queryableBy columns", ->
expect(@performed[2].query).toBe('INSERT OR IGNORE INTO `TestModelCategory` (`id`,`value`,`other`) VALUES (?,?,?),(?,?,?)')
expect(@performed[2].values).toEqual(['local-6806434c-b0cd', 'a', 'other','local-6806434c-b0cd', 'b', 'other'])
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
describe "model collection attributes query building", ->
beforeEach ->
TestModel.configureWithCollectionAttribute()
perf(db): Use subselect to improve thread list query perf (53%!) Summary: - Use a sub-select query with much better performance to display the thread list - Perform analyze on tables after launch The new query is: ``` SELECT `Thread`.`data` FROM `Thread` WHERE `Thread`.`id` IN (SELECT `id` FROM `ThreadCategory` AS `M26` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','dsnn660af0pmou2gg3nstga8a','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0) ORDER BY `Thread`.`last_message_received_timestamp` DESC; ` 0|0|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) 0|0|0|EXECUTE LIST SUBQUERY 1 1|0|0|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 1|0|0|EXECUTE LIST SUBQUERY 2 0|0|0|USE TEMP B-TREE FOR (only on 200 result items) ``` Which is twice as performant as: ``` SELECT `Thread`.`data` FROM `Thread` INNER JOIN `ThreadCategory` AS `M26` ON `M26`.`id` = `Thread`.`id` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0; 0|0|1|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 0|0|0|EXECUTE LIST SUBQUERY 1 0|1|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) ``` Test Plan: Broken! Reviewers: evan, juan Reviewed By: juan Differential Revision: https://phab.nylas.com/D2869
2016-04-12 04:29:05 +08:00
@m = new TestModel(id: 'local-6806434c-b0cd', other: 'other')
DatabaseTransaction / QueryBuilder spec fixes
2016-01-26 03:11:57 +08:00
@m.categories = []
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
it "should page association records into multiple queries correctly", ->
DatabaseTransaction / QueryBuilder spec fixes
2016-01-26 03:11:57 +08:00
@m.categories.push(new Category(id: "id-#{i}")) for i in [0..199]
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
@transaction._writeModels([@m])
collectionAttributeQueries = _.filter @performed, (i) ->
fix(db): Remove `-` from table names, which was a bad idea
2016-04-05 08:44:45 +08:00
i.query.indexOf('INSERT OR IGNORE INTO `TestModelCategory`') == 0
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
expect(collectionAttributeQueries.length).toBe(1)
perf(db): Use subselect to improve thread list query perf (53%!) Summary: - Use a sub-select query with much better performance to display the thread list - Perform analyze on tables after launch The new query is: ``` SELECT `Thread`.`data` FROM `Thread` WHERE `Thread`.`id` IN (SELECT `id` FROM `ThreadCategory` AS `M26` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','dsnn660af0pmou2gg3nstga8a','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0) ORDER BY `Thread`.`last_message_received_timestamp` DESC; ` 0|0|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) 0|0|0|EXECUTE LIST SUBQUERY 1 1|0|0|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 1|0|0|EXECUTE LIST SUBQUERY 2 0|0|0|USE TEMP B-TREE FOR (only on 200 result items) ``` Which is twice as performant as: ``` SELECT `Thread`.`data` FROM `Thread` INNER JOIN `ThreadCategory` AS `M26` ON `M26`.`id` = `Thread`.`id` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0; 0|0|1|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 0|0|0|EXECUTE LIST SUBQUERY 1 0|1|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) ``` Test Plan: Broken! Reviewers: evan, juan Reviewed By: juan Differential Revision: https://phab.nylas.com/D2869
2016-04-12 04:29:05 +08:00
expect(collectionAttributeQueries[0].values[200*3-2]).toEqual('id-199')
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
it "should page association records into multiple queries correctly", ->
DatabaseTransaction / QueryBuilder spec fixes
2016-01-26 03:11:57 +08:00
@m.categories.push(new Category(id: "id-#{i}")) for i in [0..200]
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
@transaction._writeModels([@m])
collectionAttributeQueries = _.filter @performed, (i) ->
fix(db): Remove `-` from table names, which was a bad idea
2016-04-05 08:44:45 +08:00
i.query.indexOf('INSERT OR IGNORE INTO `TestModelCategory`') == 0
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
expect(collectionAttributeQueries.length).toBe(2)
perf(db): Use subselect to improve thread list query perf (53%!) Summary: - Use a sub-select query with much better performance to display the thread list - Perform analyze on tables after launch The new query is: ``` SELECT `Thread`.`data` FROM `Thread` WHERE `Thread`.`id` IN (SELECT `id` FROM `ThreadCategory` AS `M26` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','dsnn660af0pmou2gg3nstga8a','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0) ORDER BY `Thread`.`last_message_received_timestamp` DESC; ` 0|0|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) 0|0|0|EXECUTE LIST SUBQUERY 1 1|0|0|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 1|0|0|EXECUTE LIST SUBQUERY 2 0|0|0|USE TEMP B-TREE FOR (only on 200 result items) ``` Which is twice as performant as: ``` SELECT `Thread`.`data` FROM `Thread` INNER JOIN `ThreadCategory` AS `M26` ON `M26`.`id` = `Thread`.`id` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0; 0|0|1|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 0|0|0|EXECUTE LIST SUBQUERY 1 0|1|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) ``` Test Plan: Broken! Reviewers: evan, juan Reviewed By: juan Differential Revision: https://phab.nylas.com/D2869
2016-04-12 04:29:05 +08:00
expect(collectionAttributeQueries[0].values[200*3-2]).toEqual('id-199')
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
expect(collectionAttributeQueries[1].values[1]).toEqual('id-200')
it "should page association records into multiple queries correctly", ->
DatabaseTransaction / QueryBuilder spec fixes
2016-01-26 03:11:57 +08:00
@m.categories.push(new Category(id: "id-#{i}")) for i in [0..201]
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
@transaction._writeModels([@m])
collectionAttributeQueries = _.filter @performed, (i) ->
fix(db): Remove `-` from table names, which was a bad idea
2016-04-05 08:44:45 +08:00
i.query.indexOf('INSERT OR IGNORE INTO `TestModelCategory`') == 0
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
expect(collectionAttributeQueries.length).toBe(2)
perf(db): Use subselect to improve thread list query perf (53%!) Summary: - Use a sub-select query with much better performance to display the thread list - Perform analyze on tables after launch The new query is: ``` SELECT `Thread`.`data` FROM `Thread` WHERE `Thread`.`id` IN (SELECT `id` FROM `ThreadCategory` AS `M26` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','dsnn660af0pmou2gg3nstga8a','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0) ORDER BY `Thread`.`last_message_received_timestamp` DESC; ` 0|0|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) 0|0|0|EXECUTE LIST SUBQUERY 1 1|0|0|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 1|0|0|EXECUTE LIST SUBQUERY 2 0|0|0|USE TEMP B-TREE FOR (only on 200 result items) ``` Which is twice as performant as: ``` SELECT `Thread`.`data` FROM `Thread` INNER JOIN `ThreadCategory` AS `M26` ON `M26`.`id` = `Thread`.`id` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0; 0|0|1|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 0|0|0|EXECUTE LIST SUBQUERY 1 0|1|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) ``` Test Plan: Broken! Reviewers: evan, juan Reviewed By: juan Differential Revision: https://phab.nylas.com/D2869
2016-04-12 04:29:05 +08:00
expect(collectionAttributeQueries[0].values[200*3-2]).toEqual('id-199')
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
expect(collectionAttributeQueries[1].values[1]).toEqual('id-200')
perf(db): Use subselect to improve thread list query perf (53%!) Summary: - Use a sub-select query with much better performance to display the thread list - Perform analyze on tables after launch The new query is: ``` SELECT `Thread`.`data` FROM `Thread` WHERE `Thread`.`id` IN (SELECT `id` FROM `ThreadCategory` AS `M26` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','dsnn660af0pmou2gg3nstga8a','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0) ORDER BY `Thread`.`last_message_received_timestamp` DESC; ` 0|0|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) 0|0|0|EXECUTE LIST SUBQUERY 1 1|0|0|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 1|0|0|EXECUTE LIST SUBQUERY 2 0|0|0|USE TEMP B-TREE FOR (only on 200 result items) ``` Which is twice as performant as: ``` SELECT `Thread`.`data` FROM `Thread` INNER JOIN `ThreadCategory` AS `M26` ON `M26`.`id` = `Thread`.`id` WHERE `M26`.`value` IN ('9m9ks71k06n5rmx82kgues09p','9s9k25q6j1krjgpkovbcjm7d','13b7ufruoymvih07ki0uahlto','dtmhlzz6phr47zp512knhjgf8','16dvjb84bszfh15kgfrjj37i3','aclwmgncdqjfibp51bvgbeik','17qad7jhbp6tozog3klm5zagt','4x4bkbawiq825u4eu3aus8tll','7axr9f5f1lzpwm2rw2ghkirhq','361qr5rva1ieby2r0ec3sn0bm','10fyvba7pjyjgeyr5i65i1zri') AND `M26`.`in_all_mail` = 1 ORDER BY `M26`.`last_message_received_timestamp` DESC LIMIT 200 OFFSET 0; 0|0|1|SCAN TABLE Thread-Category AS M26 USING COVERING INDEX ThreadFancyIndex 0|0|0|EXECUTE LIST SUBQUERY 1 0|1|0|SEARCH TABLE Thread USING INDEX Thread_id (id=?) ``` Test Plan: Broken! Reviewers: evan, juan Reviewed By: juan Differential Revision: https://phab.nylas.com/D2869
2016-04-12 04:29:05 +08:00
expect(collectionAttributeQueries[1].values[4]).toEqual('id-201')
feat(transactions): Explicit (and faster) database transactions Summary: Until now, we've been hiding transactions beneath the surface. When you call persistModel, you're implicitly creating a transaction. You could explicitly create them with `atomically`..., but there were several critical problems that are fixed in this diff: - Calling persistModel / unpersistModel within a transaction could cause the DatabaseStore to trigger. This could result in other parts of the app making queries /during/ the transaction, potentially before the COMMIT occurred and saved the changes. The new, explicit inTransaction syntax holds all changes until after COMMIT and then triggers. - Calling atomically and then calling persistModel inside that resulted in us having to check whether a transaction was present and was gross. - Many parts of the code ran extensive logic inside a promise chained within `atomically`: BAD: ``` DatabaseStore.atomically => DatabaseStore.persistModel(draft) => GoMakeANetworkRequestThatReturnsAPromise ``` OVERWHELMINGLY BETTER: ``` DatabaseStore.inTransaction (t) => t.persistModel(draft) .then => GoMakeANetworkRequestThatReturnsAPromise ``` Having explicit transactions also puts us on equal footing with Sequelize and other ORMs. Note that you /have/ to call DatabaseStore.inTransaction (t) =>. There is no other way to access the methods that let you alter the database. :-) Other changes: - This diff removes Message.labels and the Message-Labels table. We weren't using Message-level labels anywhere, and the table could grow very large. - This diff changes the page size during initial sync from 250 => 200 in an effort to make transactions a bit faster. Test Plan: Run tests! Reviewers: juan, evan Reviewed By: juan, evan Differential Revision: https://phab.nylas.com/D2353
2015-12-18 03:46:05 +08:00
describe "when the model has joined data attributes", ->
beforeEach ->
TestModel.configureWithJoinedDataAttribute()
it "should not include the value to the joined attribute in the JSON written to the main model table", ->
@m = new TestModel(clientId: 'local-6806434c-b0cd', serverId: 'server-1', body: 'hello world')
@transaction._writeModels([@m])
expect(@performed[0].values).toEqual(['server-1', '{"client_id":"local-6806434c-b0cd","server_id":"server-1","id":"server-1"}', 'local-6806434c-b0cd', 'server-1'])
it "should write the value to the joined table if it is defined", ->
@m = new TestModel(id: 'local-6806434c-b0cd', body: 'hello world')
@transaction._writeModels([@m])
expect(@performed[1].query).toBe('REPLACE INTO `TestModelBody` (`id`, `value`) VALUES (?, ?)')
expect(@performed[1].values).toEqual([@m.id, @m.body])
it "should not write the value to the joined table if it undefined", ->
@m = new TestModel(id: 'local-6806434c-b0cd')
@transaction._writeModels([@m])
expect(@performed.length).toBe(1)
Reference in a new issue Copy permalink