Mailspring/internal_packages/thread-list/lib/thread-list-data-source.coffee

76 lines
3.1 KiB
CoffeeScript
Raw Normal View History

_ = require 'underscore'
Merging in new observables for thread list commit 7a67c1fd349c575a91b162024cc03050e86574c9 Author: Ben Gotow <bengotow@gmail.com> Date: Fri Jan 8 11:14:07 2016 -0800 WIP commit 891f23487827a447ec95406ef26f1473a0c07de6 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Jan 6 15:25:09 2016 -0800 WIP commit 3c323cd4beb2df2fae2439a556d3129404d942cc Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:46:11 2016 -0800 WIP commit ec7090ea9e1969fea2ea583f80a9a2ac41e6c8b0 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:22:07 2016 -0800 Remove unused LRUCache commit e10c3919559d3c364cb7bb94d19094a2444c10f3 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 16:21:37 2016 -0800 rm(database-view): Performance refactor of thread-list Summary: This diff removes the old DatabaseView class, which included lots of gross optimizations that have since been duplicated in QuerySubscription and makes the thread list use the QuerySubscription class. This diff also substantially replaces the QuerySubscription class. The new implementation actually makes more queries but is less gross and more straightforward. It leverages a couple findings from database profiling: - Because of the sqlite page cache, asking for ids you've previously asked for is very fast. + Don't bother sorting in memory to avoid a query, just ask for ids again and fill in any missing objects. - Loading and inflating models is 4x+ slower than just grabbing ids I've also added more convenience classes around database queries: - QueryRange: Represents {offset, limit}, and can do boolean intersections - QueryResultSet: Better than passing an array of 50 items when you really mean items 150-200. Also tries hard to be immutable. This diff doesn't fully remove the concept of a "ModelView" because it's used /everywhere/ in the multiselect list source. There's a small shim that we can remove when we refactor that code. Ideally, I think we should rename ModelView to "MultiselectListDataSource" to follow iOS conventions (eg UITableViewDataSource). RIP 80char lines? Test Plan: They've gone to hell. WIP. Reviewers: evan, juan Differential Revision: https://phab.nylas.com/D2408 commit 32607eee8aafb7fa98b866347bdd2c0b963a602c Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 09:56:34 2016 -0800 WIP commit 5ab5fe74e94db6904bd77d224720ad9fc69fe6a7 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 22:56:46 2015 -0800 redo scrollbars to not require counts commit 361bb192d072dc8a69fd3ef143cad7bed214ebdc Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 17:50:57 2015 -0800 wip commit 079394de1cc3344fb6568efe00a52d7fc97fbd27 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 13:49:11 2015 -0800 wip commit 65142be03c27c653fe1147fdde6c2f9b046ade22 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 01:23:20 2015 -0800 wip commit 5d412ec276be1104175ad0f43c9d54e1cea857bf Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 22:49:58 2015 -0800 Refactor start commit d2b6eea884fcd2bd81ebe3985f2b2636a510c493 Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 18:51:53 2015 -0800 RIP DatabaseView
2016-01-09 06:31:33 +08:00
Rx = require 'rx-lite'
{ObservableListDataSource,
Merging in new observables for thread list commit 7a67c1fd349c575a91b162024cc03050e86574c9 Author: Ben Gotow <bengotow@gmail.com> Date: Fri Jan 8 11:14:07 2016 -0800 WIP commit 891f23487827a447ec95406ef26f1473a0c07de6 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Jan 6 15:25:09 2016 -0800 WIP commit 3c323cd4beb2df2fae2439a556d3129404d942cc Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:46:11 2016 -0800 WIP commit ec7090ea9e1969fea2ea583f80a9a2ac41e6c8b0 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:22:07 2016 -0800 Remove unused LRUCache commit e10c3919559d3c364cb7bb94d19094a2444c10f3 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 16:21:37 2016 -0800 rm(database-view): Performance refactor of thread-list Summary: This diff removes the old DatabaseView class, which included lots of gross optimizations that have since been duplicated in QuerySubscription and makes the thread list use the QuerySubscription class. This diff also substantially replaces the QuerySubscription class. The new implementation actually makes more queries but is less gross and more straightforward. It leverages a couple findings from database profiling: - Because of the sqlite page cache, asking for ids you've previously asked for is very fast. + Don't bother sorting in memory to avoid a query, just ask for ids again and fill in any missing objects. - Loading and inflating models is 4x+ slower than just grabbing ids I've also added more convenience classes around database queries: - QueryRange: Represents {offset, limit}, and can do boolean intersections - QueryResultSet: Better than passing an array of 50 items when you really mean items 150-200. Also tries hard to be immutable. This diff doesn't fully remove the concept of a "ModelView" because it's used /everywhere/ in the multiselect list source. There's a small shim that we can remove when we refactor that code. Ideally, I think we should rename ModelView to "MultiselectListDataSource" to follow iOS conventions (eg UITableViewDataSource). RIP 80char lines? Test Plan: They've gone to hell. WIP. Reviewers: evan, juan Differential Revision: https://phab.nylas.com/D2408 commit 32607eee8aafb7fa98b866347bdd2c0b963a602c Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 09:56:34 2016 -0800 WIP commit 5ab5fe74e94db6904bd77d224720ad9fc69fe6a7 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 22:56:46 2015 -0800 redo scrollbars to not require counts commit 361bb192d072dc8a69fd3ef143cad7bed214ebdc Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 17:50:57 2015 -0800 wip commit 079394de1cc3344fb6568efe00a52d7fc97fbd27 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 13:49:11 2015 -0800 wip commit 65142be03c27c653fe1147fdde6c2f9b046ade22 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 01:23:20 2015 -0800 wip commit 5d412ec276be1104175ad0f43c9d54e1cea857bf Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 22:49:58 2015 -0800 Refactor start commit d2b6eea884fcd2bd81ebe3985f2b2636a510c493 Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 18:51:53 2015 -0800 RIP DatabaseView
2016-01-09 06:31:33 +08:00
DatabaseStore,
Message,
Merging in new observables for thread list commit 7a67c1fd349c575a91b162024cc03050e86574c9 Author: Ben Gotow <bengotow@gmail.com> Date: Fri Jan 8 11:14:07 2016 -0800 WIP commit 891f23487827a447ec95406ef26f1473a0c07de6 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Jan 6 15:25:09 2016 -0800 WIP commit 3c323cd4beb2df2fae2439a556d3129404d942cc Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:46:11 2016 -0800 WIP commit ec7090ea9e1969fea2ea583f80a9a2ac41e6c8b0 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:22:07 2016 -0800 Remove unused LRUCache commit e10c3919559d3c364cb7bb94d19094a2444c10f3 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 16:21:37 2016 -0800 rm(database-view): Performance refactor of thread-list Summary: This diff removes the old DatabaseView class, which included lots of gross optimizations that have since been duplicated in QuerySubscription and makes the thread list use the QuerySubscription class. This diff also substantially replaces the QuerySubscription class. The new implementation actually makes more queries but is less gross and more straightforward. It leverages a couple findings from database profiling: - Because of the sqlite page cache, asking for ids you've previously asked for is very fast. + Don't bother sorting in memory to avoid a query, just ask for ids again and fill in any missing objects. - Loading and inflating models is 4x+ slower than just grabbing ids I've also added more convenience classes around database queries: - QueryRange: Represents {offset, limit}, and can do boolean intersections - QueryResultSet: Better than passing an array of 50 items when you really mean items 150-200. Also tries hard to be immutable. This diff doesn't fully remove the concept of a "ModelView" because it's used /everywhere/ in the multiselect list source. There's a small shim that we can remove when we refactor that code. Ideally, I think we should rename ModelView to "MultiselectListDataSource" to follow iOS conventions (eg UITableViewDataSource). RIP 80char lines? Test Plan: They've gone to hell. WIP. Reviewers: evan, juan Differential Revision: https://phab.nylas.com/D2408 commit 32607eee8aafb7fa98b866347bdd2c0b963a602c Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 09:56:34 2016 -0800 WIP commit 5ab5fe74e94db6904bd77d224720ad9fc69fe6a7 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 22:56:46 2015 -0800 redo scrollbars to not require counts commit 361bb192d072dc8a69fd3ef143cad7bed214ebdc Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 17:50:57 2015 -0800 wip commit 079394de1cc3344fb6568efe00a52d7fc97fbd27 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 13:49:11 2015 -0800 wip commit 65142be03c27c653fe1147fdde6c2f9b046ade22 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 01:23:20 2015 -0800 wip commit 5d412ec276be1104175ad0f43c9d54e1cea857bf Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 22:49:58 2015 -0800 Refactor start commit d2b6eea884fcd2bd81ebe3985f2b2636a510c493 Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 18:51:53 2015 -0800 RIP DatabaseView
2016-01-09 06:31:33 +08:00
QueryResultSet,
QuerySubscription} = require 'nylas-exports'
Merging in new observables for thread list commit 7a67c1fd349c575a91b162024cc03050e86574c9 Author: Ben Gotow <bengotow@gmail.com> Date: Fri Jan 8 11:14:07 2016 -0800 WIP commit 891f23487827a447ec95406ef26f1473a0c07de6 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Jan 6 15:25:09 2016 -0800 WIP commit 3c323cd4beb2df2fae2439a556d3129404d942cc Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:46:11 2016 -0800 WIP commit ec7090ea9e1969fea2ea583f80a9a2ac41e6c8b0 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:22:07 2016 -0800 Remove unused LRUCache commit e10c3919559d3c364cb7bb94d19094a2444c10f3 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 16:21:37 2016 -0800 rm(database-view): Performance refactor of thread-list Summary: This diff removes the old DatabaseView class, which included lots of gross optimizations that have since been duplicated in QuerySubscription and makes the thread list use the QuerySubscription class. This diff also substantially replaces the QuerySubscription class. The new implementation actually makes more queries but is less gross and more straightforward. It leverages a couple findings from database profiling: - Because of the sqlite page cache, asking for ids you've previously asked for is very fast. + Don't bother sorting in memory to avoid a query, just ask for ids again and fill in any missing objects. - Loading and inflating models is 4x+ slower than just grabbing ids I've also added more convenience classes around database queries: - QueryRange: Represents {offset, limit}, and can do boolean intersections - QueryResultSet: Better than passing an array of 50 items when you really mean items 150-200. Also tries hard to be immutable. This diff doesn't fully remove the concept of a "ModelView" because it's used /everywhere/ in the multiselect list source. There's a small shim that we can remove when we refactor that code. Ideally, I think we should rename ModelView to "MultiselectListDataSource" to follow iOS conventions (eg UITableViewDataSource). RIP 80char lines? Test Plan: They've gone to hell. WIP. Reviewers: evan, juan Differential Revision: https://phab.nylas.com/D2408 commit 32607eee8aafb7fa98b866347bdd2c0b963a602c Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 09:56:34 2016 -0800 WIP commit 5ab5fe74e94db6904bd77d224720ad9fc69fe6a7 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 22:56:46 2015 -0800 redo scrollbars to not require counts commit 361bb192d072dc8a69fd3ef143cad7bed214ebdc Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 17:50:57 2015 -0800 wip commit 079394de1cc3344fb6568efe00a52d7fc97fbd27 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 13:49:11 2015 -0800 wip commit 65142be03c27c653fe1147fdde6c2f9b046ade22 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 01:23:20 2015 -0800 wip commit 5d412ec276be1104175ad0f43c9d54e1cea857bf Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 22:49:58 2015 -0800 Refactor start commit d2b6eea884fcd2bd81ebe3985f2b2636a510c493 Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 18:51:53 2015 -0800 RIP DatabaseView
2016-01-09 06:31:33 +08:00
_flatMapJoiningMessages = ($threadsResultSet) =>
# DatabaseView leverages `QuerySubscription` for threads /and/ for the
# messages on each thread, which are passed to out as `thread.metadata`.
$messagesResultSets = {}
# 2. when we receive a set of threads, we check to see if we have message
# observables for each thread. If threads have been added to the result set,
# we make a single database query and load /all/ the message metadata for
# the new threads at once. (This is a performance optimization -it's about
# ~80msec faster than making 100 queries for 100 new thread ids separately.)
Rx.Observable.zip([
$threadsResultSet,
$threadsResultSet.flatMapLatest (threadsResultSet) =>
missingIds = threadsResultSet.ids().filter (id) -> not $messagesResultSets[id]
return Rx.Observable.from([[]]) if missingIds.length is 0
Rx.Observable.fromPromise(DatabaseStore.findAll(Message, threadId: missingIds))
])
# 3. when that finishes, we group the loaded messsages by threadId and create
# the missing observables. Creating a query subscription would normally load
# an initial result set. To avoid that, we just hand new subscriptions the
# results we loaded in #2.
.flatMapLatest ([threadsResultSet, messagesForNewThreads]) =>
messagesGrouped = {}
for message in messagesForNewThreads
messagesGrouped[message.threadId] ?= []
messagesGrouped[message.threadId].push(message)
oldSets = $messagesResultSets
$messagesResultSets = {}
sets = threadsResultSet.ids().map (id) =>
$messagesResultSets[id] = oldSets[id] || _observableForThreadMessages(id, messagesGrouped[id])
$messagesResultSets[id]
sets.unshift(Rx.Observable.from([threadsResultSet]))
# 4. We use `combineLatest` to merge the message observables into a single
# stream (like Promise.all). When /any/ of them emit a new result set, we
# trigger.
Rx.Observable.combineLatest(sets)
.flatMapLatest ([threadsResultSet, messagesResultSets...]) =>
threadsWithMetadata = {}
threadsResultSet.models().map (thread, idx) ->
thread = new thread.constructor(thread)
thread.metadata = messagesResultSets[idx]?.models()
threadsWithMetadata[thread.id] = thread
Rx.Observable.from([QueryResultSet.setByApplyingModels(threadsResultSet, threadsWithMetadata)])
_observableForThreadMessages = (id, initialModels) ->
subscription = new QuerySubscription(DatabaseStore.findAll(Message, threadId: id), {
asResultSet: true,
initialModels: initialModels
})
Rx.Observable.fromPrivateQuerySubscription('message-'+id, subscription)
class ThreadListDataSource extends ObservableListDataSource
constructor: (subscription) ->
$resultSetObservable = Rx.Observable.fromPrivateQuerySubscription('thread-list', subscription)
$resultSetObservable = _flatMapJoiningMessages($resultSetObservable)
super($resultSetObservable, subscription.replaceRange)
Merging in new observables for thread list commit 7a67c1fd349c575a91b162024cc03050e86574c9 Author: Ben Gotow <bengotow@gmail.com> Date: Fri Jan 8 11:14:07 2016 -0800 WIP commit 891f23487827a447ec95406ef26f1473a0c07de6 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Jan 6 15:25:09 2016 -0800 WIP commit 3c323cd4beb2df2fae2439a556d3129404d942cc Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:46:11 2016 -0800 WIP commit ec7090ea9e1969fea2ea583f80a9a2ac41e6c8b0 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 17:22:07 2016 -0800 Remove unused LRUCache commit e10c3919559d3c364cb7bb94d19094a2444c10f3 Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 16:21:37 2016 -0800 rm(database-view): Performance refactor of thread-list Summary: This diff removes the old DatabaseView class, which included lots of gross optimizations that have since been duplicated in QuerySubscription and makes the thread list use the QuerySubscription class. This diff also substantially replaces the QuerySubscription class. The new implementation actually makes more queries but is less gross and more straightforward. It leverages a couple findings from database profiling: - Because of the sqlite page cache, asking for ids you've previously asked for is very fast. + Don't bother sorting in memory to avoid a query, just ask for ids again and fill in any missing objects. - Loading and inflating models is 4x+ slower than just grabbing ids I've also added more convenience classes around database queries: - QueryRange: Represents {offset, limit}, and can do boolean intersections - QueryResultSet: Better than passing an array of 50 items when you really mean items 150-200. Also tries hard to be immutable. This diff doesn't fully remove the concept of a "ModelView" because it's used /everywhere/ in the multiselect list source. There's a small shim that we can remove when we refactor that code. Ideally, I think we should rename ModelView to "MultiselectListDataSource" to follow iOS conventions (eg UITableViewDataSource). RIP 80char lines? Test Plan: They've gone to hell. WIP. Reviewers: evan, juan Differential Revision: https://phab.nylas.com/D2408 commit 32607eee8aafb7fa98b866347bdd2c0b963a602c Author: Ben Gotow <bengotow@gmail.com> Date: Mon Jan 4 09:56:34 2016 -0800 WIP commit 5ab5fe74e94db6904bd77d224720ad9fc69fe6a7 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 22:56:46 2015 -0800 redo scrollbars to not require counts commit 361bb192d072dc8a69fd3ef143cad7bed214ebdc Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 17:50:57 2015 -0800 wip commit 079394de1cc3344fb6568efe00a52d7fc97fbd27 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 13:49:11 2015 -0800 wip commit 65142be03c27c653fe1147fdde6c2f9b046ade22 Author: Ben Gotow <bengotow@gmail.com> Date: Wed Dec 30 01:23:20 2015 -0800 wip commit 5d412ec276be1104175ad0f43c9d54e1cea857bf Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 22:49:58 2015 -0800 Refactor start commit d2b6eea884fcd2bd81ebe3985f2b2636a510c493 Author: Ben Gotow <bengotow@gmail.com> Date: Tue Dec 29 18:51:53 2015 -0800 RIP DatabaseView
2016-01-09 06:31:33 +08:00
module.exports = ThreadListDataSource