simple-login/app
1
1
Fork 0
mirror of https://github.com/simple-login/app.git synced 2024-11-17 22:21:38 +08:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #204 from simple-login/domain-trash

Browse source 
Domain trash
This commit is contained in:
Son Nguyen Kim 2020-05-23 12:23:02 +02:00 committed by GitHub
commit 96502c677d
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 470 additions and 181 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

316
README.md
View file

@ -77,6 +77,10 @@ the following section will show a step-by-step guide on how to get your own emai
[3. Contributing Guide](#contributing)
[4. API](#api)
[5. OAuth2/OpenID Connect](#oauth)
## General Architecture
@ -615,7 +619,49 @@ then open http://localhost:7777, you should be able to login with the following
john@wick.com / password
```
### API
### Database migration
The database migration is handled by `alembic`
Whenever the model changes, a new migration has to be created.
If you have Docker installed, you can create the migration by the following script:
```bash
sh new_migration.sh
```
Make sure to review the migration script before committing it.
Sometimes (very rarely though), the automatically generated script can be incorrect.
We cannot use the local database to generate migration script as the local database doesn't use migration.
It is created via `db.create_all()` (cf `fake_data()` method). This is convenient for development and
unit tests as we don't have to wait for the migration.
### Code structure
The repo consists of the three following entry points:
- wsgi.py and server.py: the webapp.
- email_handler.py: the email handler.
- cron.py: the cronjob.
Here are the small sum-ups of the directory structures and their roles:
- app/: main Flask app. It is structured into different packages representing different features like oauth, api, dashboard, etc.
- local_data/: contains files to facilitate the local development. They are replaced during the deployment.
- migrations/: generated by flask-migrate. Edit these files will be only edited when you spot (very rare) errors on the database migration files.
- static/: files available at `/static` url.
- templates/: contains both html and email templates.
- tests/: tests. We don't really distinguish unit, functional or integration test. A test is simply here to make sure a feature works correctly.
The code is formatted using https://github.com/psf/black, to format the code, simply run
```
black .
```
## API
SimpleLogin current API clients are Chrome/Firefox/Safari extension and mobile (iOS/Android) app.
These clients rely on `API Code` for authentication.
@ -638,6 +684,90 @@ Some errors should be fixed during development however: for example error like `
All following endpoint return `401` status code if the API Key is incorrect.
### Authentication endpoints
#### POST /api/auth/login
Input:
- email
- password
- device: device name. Used to create the API Key. Should be humanly readable so user can manage later on the "API Key" page.
Output:
- name: user name, could be an empty string
- mfa_enabled: boolean
- mfa_key: only useful when user enables MFA. In this case, user needs to enter their OTP token in order to login.
- api_key: if MFA is not enabled, the `api key` is returned right away.
The `api_key` is used in all subsequent requests. It's empty if MFA is enabled.
If user hasn't enabled MFA, `mfa_key` is empty.
Return 403 if user has enabled FIDO. The client can display a message to suggest user to use the `API Key` instead.
#### POST /api/auth/mfa
Input:
- mfa_token: OTP token that user enters
- mfa_key: MFA key obtained in previous auth request, e.g. /api/auth/login
- device: the device name, used to create an ApiKey associated with this device
Output:
- name: user name, could be an empty string
- api_key: if MFA is not enabled, the `api key` is returned right away.
The `api_key` is used in all subsequent requests. It's empty if MFA is enabled.
If user hasn't enabled MFA, `mfa_key` is empty.
#### POST /api/auth/facebook
Input:
- facebook_token: Facebook access token
- device: device name. Used to create the API Key. Should be humanly readable so user can manage later on the "API Key" page.
Output: Same output as for `/api/auth/login` endpoint
#### POST /api/auth/google
Input:
- google_token: Google access token
- device: device name. Used to create the API Key. Should be humanly readable so user can manage later on the "API Key" page.
Output: Same output as for `/api/auth/login` endpoint
#### POST /api/auth/register
Input:
- email
- password
Output: 200 means user is going to receive an email that contains an *activation code*. User needs to enter this code to confirm their account -> next endpoint.
#### POST /api/auth/activate
Input:
- email
- code: the activation code
Output:
- 200: account is activated. User can login now
- 400: wrong email, code
- 410: wrong code too many times. User needs to ask for an reactivation -> next endpoint
#### POST /api/auth/reactivate
Input:
- email
Output:
- 200: user is going to receive an email that contains the activation code.
#### POST /api/auth/forgot_password
Input:
- email
Output: always return 200, even if email doesn't exist. User need to enter correctly their email.
#### GET /api/user_info
Given the API Key, return user name and whether user is premium.
@ -659,6 +789,7 @@ Output: if api key is correct, return a json with user name and whether user is
If api key is incorrect, return 401.
### Alias endpoints
#### GET /api/v4/alias/options
@ -751,115 +882,6 @@ If success, 201 with the new alias, for example
}
```
#### POST /api/auth/login
Input:
- email
- password
- device: device name. Used to create the API Key. Should be humanly readable so user can manage later on the "API Key" page.
Output:
- name: user name, could be an empty string
- mfa_enabled: boolean
- mfa_key: only useful when user enables MFA. In this case, user needs to enter their OTP token in order to login.
- api_key: if MFA is not enabled, the `api key` is returned right away.
The `api_key` is used in all subsequent requests. It's empty if MFA is enabled.
If user hasn't enabled MFA, `mfa_key` is empty.
Return 403 if user has enabled FIDO. The client can display a message to suggest user to use the `API Key` instead.
#### POST /api/auth/mfa
Input:
- mfa_token: OTP token that user enters
- mfa_key: MFA key obtained in previous auth request, e.g. /api/auth/login
- device: the device name, used to create an ApiKey associated with this device
Output:
- name: user name, could be an empty string
- api_key: if MFA is not enabled, the `api key` is returned right away.
The `api_key` is used in all subsequent requests. It's empty if MFA is enabled.
If user hasn't enabled MFA, `mfa_key` is empty.
#### POST /api/auth/facebook
Input:
- facebook_token: Facebook access token
- device: device name. Used to create the API Key. Should be humanly readable so user can manage later on the "API Key" page.
Output: Same output as for `/api/auth/login` endpoint
#### POST /api/auth/google
Input:
- google_token: Google access token
- device: device name. Used to create the API Key. Should be humanly readable so user can manage later on the "API Key" page.
Output: Same output as for `/api/auth/login` endpoint
#### POST /api/auth/register
Input:
- email
- password
Output: 200 means user is going to receive an email that contains an *activation code*. User needs to enter this code to confirm their account -> next endpoint.
#### POST /api/auth/activate
Input:
- email
- code: the activation code
Output:
- 200: account is activated. User can login now
- 400: wrong email, code
- 410: wrong code too many times. User needs to ask for an reactivation -> next endpoint
#### POST /api/auth/reactivate
Input:
- email
Output:
- 200: user is going to receive an email that contains the activation code.
#### POST /api/auth/forgot_password
Input:
- email
Output: always return 200, even if email doesn't exist. User need to enter correctly their email.
#### GET /api/mailboxes
Get user verified mailboxes.
Input:
- `Authentication` header that contains the api key
Output:
List of mailboxes. Each mailbox has id, email field.
```json
{
"mailboxes": [
{
"email": "a@b.c",
"id": 1
},
{
"email": "m1@example.com",
"id": 2
}
]
}
```
#### GET /api/v2/aliases
Get user aliases.
@ -1087,7 +1109,6 @@ If success, 200 with the list of contacts, for example:
Please note that last_email_sent_timestamp and last_email_sent_date can be null.
#### POST /api/aliases/:alias_id/contacts
Create a new contact for an alias.
@ -1113,6 +1134,35 @@ Return 409 if contact is already added.
}
```
### Mailbox endpoints
#### GET /api/mailboxes
Get user verified mailboxes.
Input:
- `Authentication` header that contains the api key
Output:
List of mailboxes. Each mailbox has id, email field.
```json
{
"mailboxes": [
{
"email": "a@b.c",
"id": 1
},
{
"email": "m1@example.com",
"id": 2
}
]
}
```
### Contact endpoints
#### DELETE /api/contacts/:contact_id
Delete a contact
@ -1131,6 +1181,7 @@ If success, 200.
}
```
### Misc endpoints
#### POST /api/apple/process_payment
Process payment receipt
@ -1144,49 +1195,8 @@ Output:
200 if user is upgraded successfully
4** if any error.
### Database migration
The database migration is handled by `alembic`
Whenever the model changes, a new migration has to be created.
If you have Docker installed, you can create the migration by the following script:
```bash
sh new_migration.sh
```
Make sure to review the migration script before committing it.
Sometimes (very rarely though), the automatically generated script can be incorrect.
We cannot use the local database to generate migration script as the local database doesn't use migration.
It is created via `db.create_all()` (cf `fake_data()` method). This is convenient for development and
unit tests as we don't have to wait for the migration.
### Code structure
The repo consists of the three following entry points:
- wsgi.py and server.py: the webapp.
- email_handler.py: the email handler.
- cron.py: the cronjob.
Here are the small sum-ups of the directory structures and their roles:
- app/: main Flask app. It is structured into different packages representing different features like oauth, api, dashboard, etc.
- local_data/: contains files to facilitate the local development. They are replaced during the deployment.
- migrations/: generated by flask-migrate. Edit these files will be only edited when you spot (very rare) errors on the database migration files.
- static/: files available at `/static` url.
- templates/: contains both html and email templates.
- tests/: tests. We don't really distinguish unit, functional or integration test. A test is simply here to make sure a feature works correctly.
The code is formatted using https://github.com/psf/black, to format the code, simply run
```
black .
```
### OAuth flow
## OAuth
SL currently supports code and implicit flow.

23
app/alias_utils.py
View file

@ -16,6 +16,7 @@ from app.models import (
Directory,
User,
DeletedAlias,
DomainDeletedAlias,
)
@ -130,15 +131,27 @@ def try_auto_create_catch_all_domain(address: str) -> Optional[Alias]:
def delete_alias(alias: Alias, user: User):
email = alias.email
Alias.delete(alias.id)
db.session.commit()
# try to save deleted alias
# save deleted alias to either global or domain trash
if alias.custom_domain_id:
try:
DeletedAlias.create(email=email)
DomainDeletedAlias.create(
user_id=user.id, email=alias.email, domain_id=alias.custom_domain_id
)
db.session.commit()
# this can happen when a previously deleted alias is re-created via catch-all or directory feature
except IntegrityError:
LOG.error("alias %s has been added before to DeletedAlias", email)
LOG.error(
"alias %s domain %s has been added before to DeletedAlias",
alias.email,
alias.custom_domain_id,
)
db.session.rollback()
else:
try:
DeletedAlias.create(email=alias.email)
db.session.commit()
except IntegrityError:
LOG.error("alias %s has been added before to DeletedAlias", alias.email)
db.session.rollback()

31
app/api/views/new_custom_alias.py
View file

@ -9,7 +9,14 @@ from app.config import MAX_NB_EMAIL_FREE_PLAN
from app.dashboard.views.custom_alias import verify_prefix_suffix, signer
from app.extensions import db
from app.log import LOG
from app.models import Alias, AliasUsedOn, User, CustomDomain, DeletedAlias
from app.models import (
Alias,
AliasUsedOn,
User,
CustomDomain,
DeletedAlias,
DomainDeletedAlias,
)
from app.utils import convert_to_id
@ -137,15 +144,25 @@ def new_custom_alias_v2():
LOG.d("full alias already used %s", full_alias)
return jsonify(error=f"alias {full_alias} already exists"), 409
alias = Alias.create(
user_id=user.id, email=full_alias, mailbox_id=user.default_mailbox_id, note=note
)
custom_domain_id = None
if alias_suffix.startswith("@"):
alias_domain = alias_suffix[1:]
domain = CustomDomain.get_by(domain=alias_domain)
LOG.d("set alias %s to domain %s", full_alias, domain)
alias.custom_domain_id = domain.id
# check if the alias is currently in the domain trash
if domain and DomainDeletedAlias.get_by(domain_id=domain.id, email=full_alias):
LOG.d(f"Alias {full_alias} is currently in the {domain.domain} trash. ")
return jsonify(error=f"alias {full_alias} in domain trash"), 409
custom_domain_id = domain.id
alias = Alias.create(
user_id=user.id,
email=full_alias,
mailbox_id=user.default_mailbox_id,
note=note,
custom_domain_id=custom_domain_id,
)
db.session.commit()

5
app/dashboard/templates/dashboard/domain_detail/base.html
View file

@ -15,6 +15,11 @@
class="list-group-item list-group-item-action {{ 'active' if domain_detail_page == 'dns' }}">
<span class="icon mr-3"><i class="fe fe-cloud"></i></span>DNS
</a>
<a href="{{ url_for('dashboard.domain_detail_trash', custom_domain_id=custom_domain.id) }}"
class="list-group-item list-group-item-action {{ 'active' if domain_detail_page == 'trash' }}">
<span class="icon mr-3"><i class="fe fe-trash"></i></span>Deleted Alias
</a>
</div>
</div>

50
app/dashboard/templates/dashboard/domain_detail/trash.html Normal file
View file

@ -0,0 +1,50 @@
{% extends 'dashboard/domain_detail/base.html' %}
{% set domain_detail_page = "trash" %}
{% block title %}
{{ custom_domain.domain }} deleted aliases
{% endblock %}
{% block domain_detail_content %}
<h1 class="h3"> {{ custom_domain.domain }} deleted alias (aka Trash)
<a class="ml-3 text-info" style="font-size: 12px" data-toggle="collapse" href="#howtouse" role="button"
aria-expanded="false" aria-controls="collapseExample">
How to use <i class="fe fe-chevrons-down"></i>
</a>
</h1>
<div class="alert alert-primary collapse" id="howtouse" role="alert">
On this page you can view all aliases that have been deleted and belong to the domain
<b>{{ custom_domain.domain }}</b>. <br>
When an alias is in the trash, it cannot be re-created, either via the alias creation page or on-the-fly with the
domain catch-all option.
</div>
{% if domain_deleted_aliases | length > 0 %}
<form method="post">
<input type="hidden" name="form-name" value="empty-all">
<button class="btn btn-outline-danger">Empty Trash</button>
<div class="small-text">
Remove all deleted aliases from the trash, allowing them to be re-created.
That operation is irreversible.
</div>
</form>
{% else %}
There's no deleted alias recorded for this domain.
{% endif %}
{% for deleted_alias in domain_deleted_aliases %}
<hr>
<b>{{ deleted_alias.email }}</b> -
deleted {{ deleted_alias.created_at | dt }}
<form method="post">
<input type="hidden" name="form-name" value="remove-single">
<input type="hidden" name="deleted-alias-id" value="{{ deleted_alias.id }}">
<button class="btn btn-sm btn-outline-warning">Remove from trash</button>
</form>
{% endfor %}
{% endblock %}

37
app/dashboard/views/custom_alias.py
View file

@ -11,7 +11,15 @@ from app.dashboard.base import dashboard_bp
from app.email_utils import email_belongs_to_alias_domains
from app.extensions import db
from app.log import LOG
from app.models import Alias, CustomDomain, DeletedAlias, Mailbox, User, AliasMailbox
from app.models import (
Alias,
CustomDomain,
DeletedAlias,
Mailbox,
User,
AliasMailbox,
DomainDeletedAlias,
)
from app.utils import convert_to_id, random_word, word_exist
signer = TimestampSigner(CUSTOM_ALIAS_SECRET)
@ -101,11 +109,31 @@ def custom_alias():
"warning",
)
else:
custom_domain_id = None
# get the custom_domain_id if alias is created with a custom domain
if alias_suffix.startswith("@"):
alias_domain = alias_suffix[1:]
domain = CustomDomain.get_by(domain=alias_domain)
# check if the alias is currently in the domain trash
if domain and DomainDeletedAlias.get_by(
domain_id=domain.id, email=full_alias
):
flash(
f"Alias {full_alias} is currently in the {domain.domain} trash. "
f"Please remove it from the trash in order to re-create it.",
"warning",
)
return redirect(url_for("dashboard.custom_alias"))
custom_domain_id = domain.id
alias = Alias.create(
user_id=current_user.id,
email=full_alias,
note=alias_note,
mailbox_id=mailboxes[0].id,
custom_domain_id=custom_domain_id,
)
db.session.flush()
@ -114,13 +142,6 @@ def custom_alias():
alias_id=alias.id, mailbox_id=mailboxes[i].id,
)
# get the custom_domain_id if alias is created with a custom domain
if alias_suffix.startswith("@"):
alias_domain = alias_suffix[1:]
domain = CustomDomain.get_by(domain=alias_domain)
LOG.d("Set alias %s domain to %s", full_alias, domain)
alias.custom_domain_id = domain.id
db.session.commit()
flash(f"Alias {full_alias} has been created", "success")

56
app/dashboard/views/domain_detail.py
View file

@ -10,7 +10,7 @@ from app.dns_utils import (
get_cname_record,
)
from app.extensions import db
from app.models import CustomDomain, Alias
from app.models import CustomDomain, Alias, DomainDeletedAlias
@dashboard_bp.route("/domains/<int:custom_domain_id>/dns", methods=["GET", "POST"])
@ -171,3 +171,57 @@ def domain_detail(custom_domain_id):
nb_alias = Alias.filter_by(custom_domain_id=custom_domain.id).count()
return render_template("dashboard/domain_detail/info.html", **locals())
@dashboard_bp.route("/domains/<int:custom_domain_id>/trash", methods=["GET", "POST"])
@login_required
def domain_detail_trash(custom_domain_id):
custom_domain = CustomDomain.get(custom_domain_id)
if not custom_domain or custom_domain.user_id != current_user.id:
flash("You cannot see this page", "warning")
return redirect(url_for("dashboard.index"))
if request.method == "POST":
if request.form.get("form-name") == "empty-all":
DomainDeletedAlias.filter_by(domain_id=custom_domain.id).delete()
db.session.commit()
flash("All deleted aliases can now be re-created", "success")
return redirect(
url_for(
"dashboard.domain_detail_trash", custom_domain_id=custom_domain.id
)
)
elif request.form.get("form-name") == "remove-single":
deleted_alias_id = request.form.get("deleted-alias-id")
deleted_alias = DomainDeletedAlias.get(deleted_alias_id)
if not deleted_alias or deleted_alias.domain_id != custom_domain.id:
flash("Unknown error, refresh the page", "warning")
return redirect(
url_for(
"dashboard.domain_detail_trash",
custom_domain_id=custom_domain.id,
)
)
DomainDeletedAlias.delete(deleted_alias.id)
db.session.commit()
flash(
f"{deleted_alias.email} can now be re-created", "success",
)
return redirect(
url_for(
"dashboard.domain_detail_trash", custom_domain_id=custom_domain.id
)
)
domain_deleted_aliases = DomainDeletedAlias.filter_by(
domain_id=custom_domain.id
).all()
return render_template(
"dashboard/domain_detail/trash.html",
domain_deleted_aliases=domain_deleted_aliases,
custom_domain=custom_domain,
)

17
app/models.py
View file

@ -1088,6 +1088,9 @@ class DeletedAlias(db.Model, ModelMixin):
email = db.Column(db.String(256), unique=True, nullable=False)
def __repr__(self):
return f"<Deleted Alias {self.email}>"
class EmailChange(db.Model, ModelMixin):
"""Used when user wants to update their email"""
@ -1179,6 +1182,20 @@ class CustomDomain(db.Model, ModelMixin):
return f"<Custom Domain {self.domain}>"
class DomainDeletedAlias(db.Model, ModelMixin):
"""Store all deleted alias for a domain"""
__table_args__ = (
db.UniqueConstraint("domain_id", "email", name="uq_domain_trash"),
)
email = db.Column(db.String(256), nullable=False)
domain_id = db.Column(
db.ForeignKey("custom_domain.id", ondelete="cascade"), nullable=False
)
user_id = db.Column(db.ForeignKey(User.id, ondelete="cascade"), nullable=False)
class LifetimeCoupon(db.Model, ModelMixin):
code = db.Column(db.String(128), nullable=False, unique=True)
nb_used = db.Column(db.Integer, nullable=False)

40
migrations/versions/2020_052312_0e08145f0499_.py Normal file
View file

@ -0,0 +1,40 @@
"""empty message
Revision ID: 0e08145f0499
Revises: ce15cf3467b4
Create Date: 2020-05-23 12:06:25.707402
"""
import sqlalchemy_utils
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '0e08145f0499'
down_revision = 'ce15cf3467b4'
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('domain_deleted_alias',
sa.Column('id', sa.Integer(), autoincrement=True, nullable=False),
sa.Column('created_at', sqlalchemy_utils.types.arrow.ArrowType(), nullable=False),
sa.Column('updated_at', sqlalchemy_utils.types.arrow.ArrowType(), nullable=True),
sa.Column('email', sa.String(length=256), nullable=False),
sa.Column('domain_id', sa.Integer(), nullable=False),
sa.Column('user_id', sa.Integer(), nullable=False),
sa.ForeignKeyConstraint(['domain_id'], ['custom_domain.id'], ondelete='cascade'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='cascade'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('domain_id', 'email', name='uq_domain_trash')
)
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('domain_deleted_alias')
# ### end Alembic commands ###

23
shell.py
View file

@ -2,8 +2,8 @@ import flask_migrate
from IPython import embed
from sqlalchemy_utils import create_database, database_exists, drop_database
from app.config import DB_URI
from app.email_utils import send_email, render
from app.config import DB_URI, ALIAS_DOMAINS
from app.email_utils import send_email, render, get_email_domain_part
from app.models import *
from server import create_app
from time import sleep
@ -95,6 +95,25 @@ def send_mobile_newsletter():
sleep(1)
def migrate_domain_trash():
"""Move aliases from global trash to domain trash if applicable"""
for deleted_alias in DeletedAlias.query.all():
alias_domain = get_email_domain_part(deleted_alias.email)
if alias_domain not in ALIAS_DOMAINS:
domain = CustomDomain.get_by(domain=alias_domain)
if domain:
LOG.d("move %s to domain %s trash", deleted_alias, domain)
DomainDeletedAlias.create(
user_id=domain.user_id,
email=deleted_alias.email,
domain_id=domain.id,
created_at=deleted_alias.created_at,
)
DeletedAlias.delete(deleted_alias.id)
db.session.commit()
app = create_app()
with app.app_context():

45
tests/api/test_new_custom_alias.py
View file

@ -1,9 +1,10 @@
from flask import url_for
from app.alias_utils import delete_alias
from app.config import EMAIL_DOMAIN, MAX_NB_EMAIL_FREE_PLAN
from app.dashboard.views.custom_alias import signer
from app.extensions import db
from app.models import User, ApiKey, Alias
from app.models import User, ApiKey, Alias, CustomDomain
from app.utils import random_word
@ -139,3 +140,45 @@ def test_success_v2(flask_client):
new_ge = Alias.get_by(email=r.json["alias"])
assert new_ge.note == "test note"
def test_cannot_create_alias_in_trash(flask_client):
user = User.create(
email="a@b.c", password="password", name="Test User", activated=True
)
db.session.commit()
# create api_key
api_key = ApiKey.create(user.id, "for test")
db.session.commit()
# create a custom domain
CustomDomain.create(user_id=user.id, domain="ab.cd", verified=True)
db.session.commit()
# create new alias with note
suffix = f"@ab.cd"
suffix = signer.sign(suffix).decode()
r = flask_client.post(
url_for("api.new_custom_alias_v2", hostname="www.test.com"),
headers={"Authentication": api_key.code},
json={"alias_prefix": "prefix", "signed_suffix": suffix, "note": "test note",},
)
# assert alias creation is successful
assert r.status_code == 201
assert r.json["alias"] == f"prefix@ab.cd"
# delete alias: it's going to be moved to ab.cd trash
alias = Alias.get_by(email="prefix@ab.cd")
assert alias.custom_domain_id
delete_alias(alias, user)
# try to create the same alias, will fail as the alias is in trash
r = flask_client.post(
url_for("api.new_custom_alias_v2", hostname="www.test.com"),
headers={"Authentication": api_key.code},
json={"alias_prefix": "prefix", "signed_suffix": suffix, "note": "test note",},
)
assert r.status_code == 409