Summary:
Previously we stored message bodies uncompressed inside two different
databases. This was bad for a few reasons:
* Duplicate data in multiple places is an obvious waste of disk space
* Uncompressed data also made the disk footprint bigger than it might
otherwise be
* Storing these large message bodies in the database made the db file
larger than it otherwise could have been, increasing the size of
tables and slowing down queries.
This diff adds support for storing message bodies outside of the
database in compressed flat files. It changes the use of the body column
in the K2 database and the MessageBody table in the edgehill database to
contain a blob of JSON that contains a path to the file on disk. We
use the new format in an incremental fashion without having to perform an
actual database migration by first checking if the body matches our expected
JSON format and treating it appropriately if it doesn't. Both databases refer
to the same file on disk, thus deduplicating the messages bodies. We also
transparently support gzipping the message bodies stored on disk when we
read from/write to the files. The real world space savings depends on the
compressibility of the messages, but we've seen up to ~60% improvement in disk
space usage for certain inboxes. Typical savings are closer to 20%. Also, by
storing messages as separate files on disk we can potentially integrate with
Spotlight search at some point in the future.
Test Plan:
Run locally, make sure that upgrade to this doesn't hose things,
look at size of DB, read emails
Reviewers: evan, spang, halla, juan
Reviewed By: halla, juan
Differential Revision: https://phab.nylas.com/D4422
Summary:
We weren't passing benchmarkMode through in the default window options,
so the worker window (or any other spawned window) would think it wasn't in
benchmark mode.
Test Plan:
Run locally, verify that the worker window thinks it's in
benchmark mode
Reviewers: halla, spang, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4427
Summary:
Given that this error is unrecoverable it shouldn't be retryable.
Additionally, this commit improves error handling for this error in a
few ways:
- Don't delete all databases, just edgehill.db which is the one we know is corrupted. (Use `handleDatabaseUnrecoverableError` instead of `Actions.resetEmailCache`)
- Message the user about the database being reset so the app doesn't restart without notice, and make sure that this message is only displayed once by moving it to the main process
Test Plan: manual
Reviewers: mark, spang, evan, halla
Reviewed By: evan, halla
Differential Revision: https://phab.nylas.com/D4433
Summary:
This commit addressess issue T8135, which prevents the app from
starting.
This was happening because when 3111c166 landed, we added database
access from the main (backend) electron process to be able to read the
identity now stored in the database: https://github.com/nylas/nylas-mail/commit/3111c166#diff-1efa26fa0ae1603366b2c0033d971028R44
However, we omitted to add any error handling, so if the database failed to open
due to a database malformed error (which it does:
https://sentry.io/nylas/nylas-mail/?query=is%3Aunresolved+release%3A2.0.14+malformed&statsPeriod=14d), the app will just fail to start,
given that this happens during the initialization of the main process.
Additionally, the fact that we had no error handling increased the error
reports for malformed errors given that we would never handle them, so
every-time we opened the app we would report the same error
This commit adds the same error handling we have in the DatabaseStore
and moves the code around so it's available both in the main and
renderer processes.
After this commit, if the database fails to open during main process
initialization, due to malformed errors or others, we will correctly
inform the user that the database is corrupted, rebuild it, and restart
the app.
Test Plan:
manually throw errors during setup, verify that we handle them
correctly
Reviewers: mark, spang, evan, halla
Reviewed By: evan, halla
Differential Revision: https://phab.nylas.com/D4431
Summary:
Which was arguably already a code smell being inside the
DatabaseStore
Test Plan: manual
Reviewers: mark, halla, evan
Reviewed By: evan
Differential Revision: https://phab.nylas.com/D4430
Summary:
After https://github.com/nylas/nylas-mail-all/commit/008cb4c, the shape of
account.connectionSettings changed, which means that ids for accounts
will also change, given that they are based on the connection settings (see
Account.hash()).
This is fine in most cases, except for accounts that were running on a
version of Nylas Mail before 008cb4c, then upgraded to a version
after 008cb4c, and then re-authed their account.
In this scenario, re-authing the account will create a second account with
a different `id` but with the same email address, along with an extra
sequelize database, and will start a second SyncWorker for the same
account. App-side, the `AccountStore` will correctly overwrite the first account,
so users would correctly see just 1 account in the sidebar. However, given
that 2 SyncWorkers would be running for the same account,
message ids would collide in edgehill.db and this would cause
threads to disappear from the ui as if they were being deleted.
To fix this, we need to do 2 things:
- Upon app start, we remove any duplicate accounts that might have been created due to this bug before starting any sync workers
- If we detect that we are going to create a duplicate account upon auth success, we delete the old account first. This will effectively cause sync to restart for the account.
Test Plan:
Verified problem: Checked out a commit before 008cb4c, authed an account, checked out master, re-authed account, verified that duplicate accounts are created.
Then test 2 scenarios:
- With duplicate accounts present, checked out this commit, verified that duplicate account would be removed and sync would function normally.
- With an account authed on a build before 008cb4c, checked out this commit, re-authed the account, and verified that duplicate account wouldn't be created
Reviewers: mark, halla, khamidou, spang
Reviewed By: spang
Subscribers: tomasz
Differential Revision: https://phab.nylas.com/D4425
Summary:
By nullifying the key inside `workersByAccountId` we would attempt to
access accountIds that no longer existed in other parts of the code by
using Object.keys()
This makes it so we correctly completely remove the record from the map
Test Plan: manual
Reviewers: halla, spang, mark
Reviewed By: mark
Differential Revision: https://phab.nylas.com/D4426
Summary:
This was showing up in Sentry. Check to see if the recipient has an email
before calling toLowerCase.
Test Plan: yolo
Reviewers: halla, juan
Reviewed By: juan
Differential Revision: https://phab.nylas.com/D4423
Summary:
It seems that sometimes we get events that don't have `event.detail.reason`.
This commits tries to make our best guess, or report info about the event shape
Test Plan: manual
Reviewers: mark, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4420
Summary:
This changes the `name` option we pass to `electron-windows-installer`,
which makes it so that the app is installed under
`AppData\Local\NylasMail` instead of `AppData\Local\Nylas` where the
old N1 binary lives
Test Plan: manually tested the builds from `ci-test-4` branch on the surface machine
Reviewers: mark, spang, khamidou, halla
Reviewed By: khamidou, halla
Differential Revision: https://phab.nylas.com/D4419
Summary:
This makes grepping the codebase really painful. We don't need these
docs here
Test Plan: manual
Reviewers: mark halla spang evan
Subscribers:
Summary:
When we had to refresh our credentials for the mail listener connection we
weren't disposing of the old connection which caused us to slowly leak
connections from the IMAP connection pool, eventually requiring a
restart. Now we always dispose of the connection prior to requesting a
new one from the pool.
Test Plan: Run locally always refreshing credentials, verify we don't get stuck
Reviewers: spang, evan, halla, juan
Reviewed By: halla, juan
Subscribers: halla
Differential Revision: https://phab.nylas.com/D4414
Summary:
We listen for unhandled rejections on both the `process` and the `window`. From
my understanding, both are necessary because some errors might escape
one or the other. However, most errors get caught by both handlers, so
we don't want to double report them to sentry (or the console).
To prevent this, this commit debounces the unhandled rejection handler
for a very short time so that the same error only gets reported once.
Additionally, it rate limits reports to sentry based on the stack trace
Test Plan: manual
Reviewers: mark, halla
Reviewed By: mark, halla
Differential Revision: https://phab.nylas.com/D4413
Summary:
Previously, we were listening for unhanlded rejections on both the
`window` and the `process`. However, the window handler didn't receive
an error as a parameter, but rather a window event. When trying to
report this event and sending it over IPC we would get an ipc parse
error.
This commit makes it so we correctly report the error instead of the
window event.
Test Plan: manual
Reviewers: mark, halla
Reviewed By: halla
Differential Revision: https://phab.nylas.com/D4412
Summary:
This reverts commit c808438ee9.
Adding an overarching transaction to `processMessage` is causing
database locked errors when processing deltas, specifically when trying
to write to the Transaction table, given that that write happens
asynchronously inside the execution of `processMessage`, but doesn't
have access to its transaction. See T8117
Given that the performance gains from wrapping messgae processing in a
transaction were not significant, let's revert this to prevent db
locking issues
Test Plan: manual
Reviewers: halla, mark
Reviewed By: mark
Differential Revision: https://phab.nylas.com/D4411
Summary:
We were reporting 1m+ of these events, heavily contributing towards our
quota
Test Plan: manual
Reviewers: halla, mark
Reviewed By: halla, mark
Differential Revision: https://phab.nylas.com/D4406
