Commit graph

37 commits

Author SHA1 Message Date
Mark Hahnenberg
9270ff6c4c [client-app] Add script that benchmarks new commits
Summary:
We want to setup the initial sync benchmark to run automatically with each
commit on a separate machine. This script does just that and dumps the
results to a directory. We can then have a separate script process them
and upload them to whichever service we want.

Test Plan: Run locally

Reviewers: evan, spang, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4296
2017-03-30 16:11:49 -07:00
Mark Hahnenberg
fa50657af1 [client-app] Add initial sync benchmarking script
Summary:
We want to be able to benchmark initial sync, so this diff adds two
scripts. The first, drop-stuff.sh, drops all data from the app and sync
databases that isn't related to account credentials. The second,
benchmark-initial-sync.sh, runs a fixed number of iterations (current 5) that
invokes drop-stuff.sh then opens the app, waits a fixed amount of time
(currently 120 seconds), and then kills the app and measures how many messages
it synced which it prints to the console. This is sufficient for us to start
measuring how quickly we can sync messages. This diff also includes the
sqlite3 amalgamation which drop-stuff.sh requires to function correctly
due to depending on the FTS5 extension which doesn't come built-in on
some platforms.

Test Plan: Run benchmark locally

Reviewers: evan, spang, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4275
2017-03-29 16:11:44 -07:00
Juan Tejada
b8883cd2b7 [dev] Only bootstrap specific pkgs in postinstall for faster installs
Summary:
This commit makes it so postinstall.es6 only bootstraps the packages we
are interested in based on a new env variable, `INSTALL_TARGET`.

`INSTALL_TARGET` can be set to `cloud`, `client` or `all`, or not set at all,
and our postinstall will bootstrap the packages accordingly. This will make
build and deploy times faster.

Test Plan: manual

Reviewers: spang, halla, evan, khamidou

Reviewed By: evan, khamidou

Differential Revision: https://phab.nylas.com/D4271
2017-03-28 13:59:55 -07:00
Evan Morikawa
44e51a0ed5 [client-app] only re-install for optional deps on mac 2017-03-27 17:00:38 -07:00
Evan Morikawa
b7036574d5 [*] add ONLY_CLIENT=false support for cloud & regular install 2017-03-27 14:40:28 -07:00
Karim Hamidou
1f9c5120cd Fix build condition
On the server the env var is defined but set to false.
2017-03-22 11:06:59 -07:00
Juan Tejada
d025aaf369 [dev] Actually fix tmp dir path in daily 2017-03-21 09:29:46 -07:00
Juan Tejada
17da34caed [dev] Fix temp dir path in daily 2017-03-21 09:25:46 -07:00
Juan Tejada
f1572f4bfc [dev] Fix path for tmp dir in daily script 2017-03-17 10:08:00 -07:00
Evan Morikawa
093f315f59 [client-app] don't bootsrap cloud stuff when building the client
Summary:
We don't need to spend the time running script/bootstrap in our cloud-*
folders!

Test Plan: manual

Reviewers: spang, juan, halla

Reviewed By: juan, halla

Differential Revision: https://phab.nylas.com/D4214
2017-03-14 13:39:56 -07:00
Karim Hamidou
3a92eaf857 [send-later] Add support for attachments
Summary:
This is going to be a diff way shorter than the previous one! Basically, it adds a new endpoint, `/blobs` to our API to store send later attachments. When a user schedules a draft to be sent, we send all attachments to this endpoint. Separately, we store the rest of the message as metadata.

When it's time to send the message, we fetch the attachments from S3, fetch the metadata and merge them together to get a message we can send.

Test Plan: Tested manually. Will make a final QA pass before landing.

Reviewers: juan, halla, evan

Reviewed By: halla, evan

Differential Revision: https://phab.nylas.com/D4196
2017-03-14 12:24:04 -07:00
Evan Morikawa
1f2370c2a9 [client-app] run windows build separately
Summary:
build-client, aka packaging, aka bundling is now separate from uploading.
This is both to compartmentalize our tasks a bit more and so we can add a
non-grunt windows task inbetween packaging and uploading.

No more heavily-overloaded PUBLISH_BUILD flag.
Added SIGN_BUILD flag instead.
No more TRAVIS and TRAVIS_PULL_REQUEST flag.

Test Plan: Manual

Reviewers: halla, juan, spang

Reviewed By: juan, spang

Differential Revision: https://phab.nylas.com/D4208
2017-03-13 15:48:45 -07:00
Juan Tejada
3b08632c19 [none] (dev) Fix changelog path in daily script 2017-03-10 14:06:30 -08:00
Juan Tejada
7649c8ed53 [none] (dev) Fix npm version bump path in script/daily 2017-03-08 22:32:47 -08:00
Juan Tejada
a7e5d814df [none] (dev) Fix paths in scripts/daily 2017-03-08 22:24:17 -08:00
Juan Tejada
c0715bfb6d [none] (dev) Add script/daily
Summary:
Generating the daily build always takes a bit of effort, so I wrote the
script to automate the process and allow anyone to do it as well.

This script will:

- Check your current working directory is in a good state
- Bump the version number in package.json
- Edit CHANGELOG.md with the git log diff between current master and the last version tag (you can also manually edit it). This is so writing the CHANGELOG before a release takes less effort; you will just need to open the CHANGELOG file and edit the commit messages.
- Commit the changes with a commit message of: bump(version) <next_version>
- Add new version tags to nylas-mail and k2
- Push the changes to Github
- If provided with the flag, it will build the app. This is intended to be used in our mac mini machine

There are a few TODO's left in here, like updating the daily channel,
and emailing a notification when the channel is updated with the buid urls.

When this lands, we will configure our mac mini to run this script daily/nightly.
Also, any engineer should be able to trigger a daily build easily by just
running this script.

For sample output, see this commit which was generated by this script:
000fa88ebb

Test Plan: manually running it

Reviewers: halla, spang, mark, khamidou, evan

Reviewed By: evan

Subscribers: tomasz

Differential Revision: https://phab.nylas.com/D4123
2017-03-08 11:54:49 -08:00
Evan Morikawa
67b7a335de [dev] use .cmd versions for Windows
Since spawn on Windows can't accept scripts like exec can :(
2017-03-07 14:50:34 -05:00
Evan Morikawa
fc3dde9111 [dev] fix AppVeyor lerna bootstrap command
Needs to include `.cmd` for windows lerna builds
2017-03-07 14:43:45 -05:00
Evan Morikawa
3c1c70e262 [client-app] use forked lerna for better npm logging 2017-03-06 15:35:56 -05:00
Evan Morikawa
c0510f6015 [*] increase loglevel for lerna bootstrap 2017-03-06 12:17:30 -05:00
Juan Tejada
c088e86110 [client-app] Include node-mac-notifier in build
Summary:
Given that `lerna bootstrap` does not install optional dependencies, we
need to manually run `npm install` inside `client-app` so
`node-mac-notifier` get's correctly installed and included in the build

See https://github.com/lerna/lerna/issues/121

Fixes T7887

Test Plan: manual

Reviewers: halla, evan

Reviewed By: evan

Maniphest Tasks: T7887

Differential Revision: https://phab.nylas.com/D4076
2017-03-06 09:15:17 -08:00
Halla Moore
4c6848d5df [*] Change the symlink location for the isomorphic-core specs
Summary:
Put them in the client-app/spec folder instead of
client-app/internal_packages

Test Plan: Ran the tests

Reviewers: juan, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D4080
2017-03-02 10:56:03 -08:00
Halla Moore
82e7a276a3 [*] Move Jasmine setup into isomorphic-core
Summary:
Move the base Jasmine spec runner into isomorphic-core to prevent
code duplication. Jasmine will look for the config file relative to
the directory it's being run in though, so we need to symlink the
config file into each package that will need it.

Test Plan: Run tests once the suites are integrated

Reviewers: evan, spang, juan

Reviewed By: spang, juan

Differential Revision: https://phab.nylas.com/D4056
2017-03-01 11:08:37 -08:00
Halla Moore
953a8e438e [*] Run the isomorphic-core specs as part of the client test suite
Summary:
Convert the isomorphic-core specs to Jasmine 1 and symlink them into
client-app/internal_packages so they are run as part of the client
test suite.

Test Plan: Ran the suite with fdescribes in iso-core to ensure they were being called properly

Reviewers: evan, spang, juan

Reviewed By: spang, juan

Differential Revision: https://phab.nylas.com/D4055
2017-03-01 11:07:48 -08:00
Halla Moore
87aa1a86ad [*] Make sure cloud deploy can access .babelrc
Summary:
Now that .babelrc is in client-app, we need to make it an exception in the
.dockerignore file. Additionally, this means that the client-app directory will
exist during the postinstall script in the cloud environment, so we need a
stricter check to figure out if the client-app needs to be built or not.

Test Plan: deployed to staging

Reviewers: evan, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D4008
2017-02-22 12:09:14 -08:00
Juan Tejada
1485575e48 [client-app] Fix build
Partially revert 3071856a8e

Symlinking instead of copying the error-logger-extensions prevents the
app from finding the appropriate dependencies required in the error
logger extensions in the main process and in the renderer processes
because globalPaths are only set in renderer processes and /after/
error-logger-extensions are loaded.

We should fix this properly in a separate diff, and move the error
logger to isomorphic-core instead of copying it or symlinking it
2017-02-21 15:57:38 -08:00
Halla Moore
1105b2621f [*] Rename postinstall.js -> postinstall.es6
The n1Cloud machines try to compile this script, even though they
never run it, and they didn't recognize it was supposed to be es6
2017-02-21 15:55:08 -08:00
Evan Morikawa
3071856a8e [client-app] symlink in babelrc and resolve links
Summary:
The prod build was failing because it couldn't find the .babelrc.

I decided to put the babelrc in the main nylas-mail repo with the
expectation that it'll need to be in the open source version too. To DRY
up this for us building, we sylink the root one to the client-app babelrc

Also since we now dereference symlinks we don't need to do the full copy
of nylas-private-resources so we don't have two error reporters floating
around

Test Plan: npm run build-client

Reviewers: juan, spang

Reviewed By: juan, spang

Differential Revision: https://phab.nylas.com/D3993
2017-02-21 16:08:59 -05:00
Evan Morikawa
4582acc2e5 [client-app] actually npm install inside the /apm dir with env vars
Summary:
We weren't actually running npm install inside of apm. This meant that
/apm/node_modules was empty causing the build to fail. This will install
with the proper env required for old apm

Wow we need to get rid of this thing soon…

Test Plan: Run `npm install`

Reviewers: jerm, spang, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3986
2017-02-21 14:34:30 -05:00
Juan Tejada
c8a299ed9b [*] Correctly copy error logger extensions in postinstall script
Summary: Also fix linter errors

Test Plan: manual

Reviewers: evan, mark, halla

Reviewed By: halla

Subscribers: halla

Differential Revision: https://phab.nylas.com/D3960
2017-02-17 12:12:56 -08:00
Evan Morikawa
b686b969cd [cloud-*] Get cloud-api and cloud-workers running on new repo
[dev] remove .tern-project

[cloud-*] move into cloud-core/build

[cloud-core] convert to launch properly via Docker
2017-02-17 11:47:45 -08:00
Evan Morikawa
918fa0b6dd [*] move to monorepo
[*] update babel

[client-app] remove flow-typed

[client-app] Move build/package.json to main package.json

[client-app] remove spec_integration

[client-app] fix babel support

Add client-private-plugins package.json

[client-app] add node_modules to global path for private-plugins

Move client-sync dependencies to client-app root

fix electron rebuild

[*] moved to monorepo

Summary: App now runs in monorepo

Test Plan: npm test

Reviewers: juan, mark, khamidou, halla, spang

Differential Revision: https://phab.nylas.com/D3947
2017-02-16 18:46:26 -08:00
Karim Hamidou
e55c36a79a [cloud-api] Add support for database migrations
Summary:
This diff adds support for database migration to our cloud API. It's partially inspired by Halla's local-sync migration diff (D3809). You can run a migration by calling "node-babel scripts/migrate-db up|down" or by calling "npm script upgrade-db|downgrade-db".

Note that for simplicity reasons we assume that we're only writing migrations for our MySQL database – people developing locally may have to blow up there dbs whenever there's a schema change, though in practice `ALTER TABLE ADD COLUMN`statements work the same on both dbs.

Test Plan: Tested locally. Will run the metadata migration on staging.

Reviewers: evan, spang, halla

Reviewed By: halla

Differential Revision: https://phab.nylas.com/D3840
2017-02-06 13:38:59 -08:00
Christine Spang
5e99f7fa21 run-redis.sh must be run under bash 2017-02-06 13:13:22 -08:00
Karim Hamidou
ae32666609 [cloud-api] Base infrastructure for running cloud workers
Summary:
I've decided to break my snooze patches in multiple parts to make it easier to review. This diff contains all the code related to running workers in the cloud. Cloud workers all inherit from the `Worker` class which defines a bunch of useful things like error handling.

What's left to do:
- spawn workers based on the plugin type
- add monitoring (I'm going to add a simple HTTP endpoint for that)
- writing a migration for the local sync db and the prod metadata db.

Test Plan: Tested manually.

Reviewers: halla, evan

Reviewed By: evan

Differential Revision: https://phab.nylas.com/D3814
2017-02-02 16:14:59 -08:00
Juan Tejada
f56a807792 [cloud-*] Fix n1-cloud build
We added new presets and plugins to the k2 .babelrc but didn't add the
respective dependencies in our package.json, so the build was failing.

Add the required dependencies, and make sure that `build-n1-cloud.js`
uses the correct babel presets and plugins
2017-01-28 02:00:16 -08:00
Evan Morikawa
1d254a7aaa [*] Add basic babel toolchain
Summary:
Adds babel to K2
Creates a simple build script so it'll run on prod.

Test Plan: manual

Reviewers: jackie, halla, juan

Reviewed By: juan

Differential Revision: https://phab.nylas.com/D3527
2016-12-16 13:08:21 -05:00