Summary:
FTS tables don't support indices, so doing UPDATEs and DELETEs based on
the `content_id` was very slow on large FTS tables. Fortunately, it seems
that `UPDATE`s and `DELETE`s based on the `rowid` are much faster, so now we
store that info hanging off the searchable models. Also fixes a random bug
where after reaching the `MAX_INDEX_SIZE` we would clear the Thread search
index on startup.
Test Plan: Run locally, time how long it takes to delete when receiving new mail
Reviewers: spang, evan, juan
Reviewed By: evan, juan
Differential Revision: https://phab.nylas.com/D3847
Summary:
Sometimes you have to perform a query that needs to do a table scan. For those
times you don't want to silence entire classes of queries (e.g. by table name),
you can now silence individual queries.
Test Plan: Run locally, verify that noisy stuff disappears from logs
Reviewers: evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3763
Summary:
This diff modifies the SearchIndexer class to handle limiting the search
index size. It does this by periodically re-evaluating the window of
the n most recent items in a particular index where n is the max size of
the index. It then unindexes the items which are marked as indexed but
are no longer in the window and indexes the things that are in the window
but aren't marked as indexed.
Test Plan:
Run locally with a reduced thread index size, verify that the index
includes the most recent items and that it is the correct size. Also verify that
the queries used properly use fast sqlite indices.
Reviewers: evan, juan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3741
Summary: See title
Test Plan: tested locally
Reviewers: juan, mark, evan
Reviewed By: juan, mark
Subscribers: juan
Differential Revision: https://phab.nylas.com/D3744
Summary:
We changed the index format (splitting participants into to and from) so
we need to update the index version so that it gets rebuilt.
Test Plan: Run locally, make sure index gets rebuilt
Reviewers: evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3728
Summary:
Previously we indexed all participants together for a single thread.
This lead to confusing search results when people would search for
'from' and get back lots of results that were only 'to'. This diff
splits the participants index into separate to and from indices and
updates the query generator appropriately.
Fixes T7580
Test Plan: Run locally, verify from doesn't return to and vice versa.
Reviewers: evan, juan
Reviewed By: juan
Maniphest Tasks: T7580
Differential Revision: https://phab.nylas.com/D3724
Summary:
We had to add the id to the index as well (along with is_search_indexed)
so that the query worked correctly since it sorts by id. Also added a
default value of 0 for boolean attributes so that we don't have to query
for two different values (0 and null), which was causing sqlite to have to
do a final b-tree sort which is slow.
Test Plan: Run locally, verify query plan with EXPLAIN QUERY PLAN
Reviewers: evan, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D3712
Summary:
This diff adds a new SearchIndexer class that each of the concrete
indexer implementations register with. This new class uses the
`isSearchIndexed` field in searchable classes to split indexing work
into chunks. It then times how long it takes to index each chunk and
schedules more work based on a target CPU percent. For example, if it
takes 150 ms to index the last chunk of work and the target CPU fraction
is 0.05 (i.e. 5% CPU use) then it will schedule the next increment of
indexing 3 seconds in the future.
Test Plan: Run locally, verify that indexing occurs in increments
Reviewers: juan, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3707
Summary:
This is a very expensive operation and there seemed to be few downsides
to including quoted portions in the index. We can revisit this to
improve how we index when we have a proper scheduler.
Test Plan: Run locally
Reviewers: juan, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3682
Summary: See title
Test Plan: Run locally, search for specific labels
Reviewers: evan, juan
Reviewed By: evan, juan
Differential Revision: https://phab.nylas.com/D3656
Summary:
Previously we waited to build the local ThreadSearch index until we were
done with the initial mail sync. This is undesirable because inboxes can
be very large, making local sync useless for a number of hours after a
user adds an account. This diff removes that restriction.
This diff also adds a new rudimentary new grammar (along with accompanying lexer
and parser) for local thread search queries. This grammar is then
translated into the appropriate SQL queries on the ThreadSearch index
table. More advanced features (e.g. in:category, date ranges, etc) can be added
easily in the future by augmenting this simple search query language.
Test Plan: Run locally, new unit tests
Reviewers: juan, evan, khamidou
Reviewed By: khamidou
Differential Revision: https://phab.nylas.com/D3614
Summary:
Depends on D3544 (K2 diff)
This commit ensures that auth notifications are showed when the
underlying sync worker fails and are cleared when an account is
successfully reconnected
To achieve this, we manually keep track and update syncStates where
appropriate via `Actions.updateAccount`, given that we have access to
N1's version of the account directly from local-sync.
Initially I was considering account delta stream to the cloud-api and the local-api, but that
just complicated things more than it helped.
This commit also fixes a bug with refreshing the gmail token in which we
we were only attempting a token refresh upon restarting the app
This addresses: T7346, T7305, T7335
Test Plan: Manual
Reviewers: halla, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D3545
Summary:
This is an initial attempt to fix an issue we’ve had with long-running queries interrupting the N1 user experience. Node-sqlite3 used an async approach that ran sqlite’s synchronous query methods on a worker thread, but doing that involves copying memory more and node-sqlite3 was just generally slow.
However, moving to better-sqlite3 made /everything/ synchronous. Even with the right indexes some of our queries just suck.
This diff adds `DatabaseStore.findAll(…).background().then()` which allows you to mark a query as “unimportant”. These queries are run in a separate process which is forked from the window and can take an extra 10-50ms to complete. That said, they’re totally async and don’t jam up the app.
I’m personally a fan of the flag and less a fan of the implementation. The “agent” process can handle many queries in it’s lifetime if they keep coming and quits after 10 seconds of inactivity. (Both to save memory and to avoid scenarios where it might end up oprhaned and running forever). While running it uses about 40MB of RAM, which is a bit on the crazy side.
Test Plan: No new tests yet
Reviewers: evan, juan
Reviewed By: evan, juan
Differential Revision: https://phab.nylas.com/D3420
Summary:
This commit adds the ability to perform event search in the calendar window.
To achieve this, SearchBar has been moved from the thread-search package to be a part
of the nylas-component-kit, and reused by both thread-search and nylas-calendar
When an event is selected from search results, the calendar view is moved to the selected
event. Depends on D3396 to open the event popover.
Test Plan: Manual
Reviewers: halla, bengotow
Reviewed By: bengotow
Differential Revision: https://phab.nylas.com/D3424
