Commit graph

33 commits

Author SHA1 Message Date
Felipe Martin
876d27f337
refactor: remove gin and use stdlib http server (#1064)
* refactor: base http server stdlib

* refactor: swagger and frontend routes

* fix: use global middlewares

* refactor: removed gin from testutils

* fix: object references in legacy webserver

* refactor: legacy, swagger and system handlers

* fix: added verbs to handlers

* fix: server handlers ordering

* refactor: bookmarks handlers

* refactor: system api routes

* tests: bookmark handlers

* refactor: migrated api auth routes

* chore: remove unused middlewares

* docs: add swagger docs to refactored system api

* chore: remove old auth routes

* refactor: account apis

* chore: removed old handlers

* fix: api v1 handlers missing middlewares

* refactor: migrated tag list route

* refactor: bookmark routes

* refactor: remove gin

* chore: make styles

* test: fixed tests

* test: generate binary file without text

* fix: global middleware missing from system api handler

* fix: incorrect api handler

* chore: avoid logging screenshot contents

* tests: bookmarks domain

* tests: shortcuts

* test: missing tests

* tests: server tests

* test: remove test using syscall to avoid windows errors

* chore: added middlewares
2025-02-26 20:50:48 +01:00
Felipe Martin
73a5239753
refactor(apiv1): accounts api (#825)
* list account and create account

* deleteaccount (wip)

* remove old accounts code

* fix from merge

* remove serve method from makefile

* ListAccounts, password hash on domain

* make lint

* more permissive assertion

* rename test

* update account

* Authorization

* updated api calls

* apis, pointers, auth

* swagger

* stylecheck

* domain validation

* tests

* swagger

* error handling

* fix system account changes

* Cleanup database interface

* test cleanup

* fixed nil references

* feat: Add logout endpoint to auth routes

* feat: Add logoutHandler for stateless JWT token logout

* fixed some bug catched in tests

* auth/account patch

* prettier

* remove test logs

* fixed incorrect number of parameters

* fixed swagger docs

* enable swagger in dev environment

* errors.Wrap -> fmt.Errorf

* test: Add comprehensive test cases for accounts API handlers

* fix: Resolve test failures in accounts_test.go

* test: Add tests for duplicate username handling in account creation and update

* feat: Add username uniqueness checks for account creation and update

refactor: Improve username existence checks in SQLite account methods

* linted

* test: Add comprehensive tests for auth domain token and credential validation

* test: Add comprehensive test cases for auth domain token creation and validation

* test: Add comprehensive error handling test cases for accounts domain

* refactor: Remove `SaveAccountSettings` method from database implementations

* test: Add test cases for password update functionality

* test(e2e): auth login

* lint

* send regular context to domain

* fixed e2e auth tests

* test: Add auth_test.go for end-to-end authentication testing

* feat: Add comprehensive authentication tests using Playwright and testcontainers

* fix: Handle multiple return values in Playwright test methods

* error message

* e2e playwrigth tests

* ci: setup playwrigth

* refactor: Update Playwright tests to use locator-based API

* refactor: Remove unnecessary alias for playwright-go expect import

* refactor: Replace deprecated expect package with WaitFor() method in Playwright tests

* fix: Resolve linting issues in e2e Playwright tests

* remove npm ci from e2e ci

* make playwright available in path

* typo

* re enabled ci

* base e2e accounts test

* more account e2e

* feat: Add HTML test reporter with screenshots and detailed results

* feat: Embed screenshots as base64 in HTML test report

* refactor: Remove GitHub step summary functionality from test helper

* refactor: Make reporter global to share test results across test helpers

* refactor: Add HandleSuccess method to TestHelper for consistent test result reporting

* feat: Add descriptive messages to all test assertions in TestHelper

* test: Add descriptive messages to assertions in accounts_test.go

* test: Add descriptive error messages to assertions in accounts_test.go

* feat: Add descriptive messages to assertions in accounts_test.go

* refactor: Update assertion functions to receive *testing.T as first argument

* refactor: Update accounts_test.go assertions to pass *testing.T argument

* refactor: Update accounts_test.go assertions to use *testing.T argument

* refactor: Update `accounts_test.go` to use `*testing.T` argument in `Require()` calls

* refactor: Update `th.Require()` calls with `t *testing.T` argument in accounts_test.go

* assert helper

* refactor: Refactor `False` test helper to use `Assert` function consistently

* refactor: Refactor `Equal` test helper to use `Assert` function

* refactor: Simplify Error test helper to use Assert function

* refactor: Refactor `NoError` to use `Assert` function for consistent error handling

* typo

* refactor: Differentiate between test cases and assertions in reporter

* refactor: Simplify AddResult method signature and use error message for assertion

* refactor: Simplify test report with focused failure details and screenshots

* refactor: Ensure assertions are always called in PlaywrightRequire helper methods

* refactor: Update test error messages to be action-oriented

* refactor: Update error messages to be more action-oriented in accounts_test.go

* refactor: Update error messages to be action-oriented in accounts_test.go

* refactor: Improve error messages in auth_test.go for better test readability

* refactor: Improve screenshot handling and test result reporting in Playwright test helper

* fix: Improve test reporting with detailed error messages and logging

* refactor: Remove unused runningInCI field from TestHelper struct

* fix: Improve message formatting in Assert method for better reporting

* assertions

* test: Add `Require()` calls to 007 test for improved error handling

* refactor: Update test reporter to include error details and improve HTML rendering

* fix: Properly escape and render base64 screenshot in HTML report

* fix: Correct base64 screenshot rendering in test reporter

* fixed tests + html report

* feat: Add artifact upload for e2e test report

* make lint

* chore: use correct version in user agent

* ci: run e2e after other checks

* chore: remove pre-commit
2025-02-22 20:38:36 +01:00
Felipe Martin
c58c35509b
feat: improve SQLite performance (#1024)
* refactor: Improve SQLite performance with connection pooling and retry logic

* feat: Add withTx and withTxRetry methods to SQLiteDatabase for handling database locks

* refactor: add Init command to all databases

* refactor: Improve transaction handling with retry and error management

* refactor: Remove panic/recover pattern in transaction handling

* refactor: Replace `errors.WithStack` with `fmt.Errorf` in transaction methods

* docs: Add docstrings to `withTx` and `withTxRetry` methods in SQLite database implementation

* feat: use new withTxRetry in SaveBookmarks

* feat: sqlite inmmediate transactions by default

* refactor: Split SQLiteDatabase into separate writer and reader dbbase instances

* refactor: Update Init method to configure both reader and writer database connections

* feat: use writer/reader sqlite databases

* refactor: Replace all read calls to use the `reader` database instance

* refactor: Replace errors.WithStack with fmt.Errorf and add nil checks

refactor: Replace errors.WithStack with fmt.Errorf and add proper error handling

fix: Handle potential database connection errors with improved error wrapping

refactor: Replace errors.WithStack with fmt.Errorf and improve error handling

refactor: Replace error handling with fmt.Errorf and proper nil checks

refactor: Replace errors.WithStack with fmt.Errorf and add nil error checks

refactor: Replace errors.WithStack with fmt.Errorf and add nil checks in sqlite.go

refactor: Replace errors.WithStack with fmt.Errorf and add nil checks

refactor: Replace errors.WithStack with fmt.Errorf and improve error handling

refactor: Replace remaining errors.WithStack with fmt.Errorf in sqlite.go

* refactor: Use withTxRetry for SetDatabaseSchemaVersion method

* fix: Simplify error handling in GetBookmark and GetAccount methods

* refactor: Remove duplicated non-nil error checks in sqlite.go

fix: duplicated non-nil checks

* tests: use testutil instead of a manual in memory sqlite db

* fix: openbsd sqlite connection
2025-01-02 09:46:39 +01:00
Felipe Martin
d75de89701
fix: use mysql json default as expression (#1048) 2025-01-01 16:12:07 +01:00
Monirzadeh
4a5564d60b
feat: store created and modified time separately on database for bookmarks (#896)
* sqlite migrate script

* create time just when bookmark added and modified update if change happen

* show added and modified time in footer instead of header

* add bun.lockb that missing

* add migrate for postgres

* add pg support of created time

* change modifed to modifed_at and create to created_at in sqlite

* change modifed to modifed_at and create to created_at in postgre

* add created_at to mariadb

* fix migration file names

* better variable name and more clear code for add modified time if created and modified is not in same day

* add unittest

* add unittest to sure filters work as expected

* index for created_at and modified_at

* build new styles.css

* update swagger documents

* make styles

* change Created and Modified to CreatedAt and ModifiedAt

* fix missing Modified

* fix typo

* missing Modified

* fix typo

* make swagger

* run tests parallel

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>

* remove t.Parallel()

* remove dayjs dependency and combine two function

* better unittest name

* fix typo

* diffrnt footer style for login and content page

* use class instead of id

* back parallel

* change duplicate url

* remvoe run Parallel

* make styles

---------

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>
2024-06-26 20:17:51 +02:00
Felipe Martin
72aecd2b60
feat: new system info panel in webui settings (#926)
* frontend

* fixed Database.DBx return value

* api endpoint

* updated swagger

* fix openbsd variable dereference

* tests

* only load information if user is owner

* memory improvement for other routes
2024-06-08 17:48:55 +02:00
Felipe Martin
a6e55705da
fix: not checking for nil-pointer errors on migrations (#906) 2024-05-12 19:07:16 +02:00
Felipe Martin
31767f75e6
feat: new migrations system (#876)
* feat: new migration system

* use newFuncMigration

* database version -> database schema version

* column name

* use path instead of filepath for goembed

* simplified migrations, added backwards compatible migrations
2024-04-27 07:46:36 +02:00
Felipe Martin
cc7c75116d
refactor: migrate bookmark static pages to new http server (#775)
* migrate bookmark content route to new http server

* new archive page

* remove unused go generate comment

* database mock

* utils cleanup

* unused var

* domains refactor and tests

* fixed secret key type

* redirect to login on ui errors

* fixed archive folder with storage domain

* webroot documentation

* some bookmark route tests

* fixed error in bookmark domain for non existant bookmarks

* centralice errors

* add coverage data to unittests

* added tests, refactor storage to use afero

* removed mock to avoid increasing complexity

* using deps to copy files around

* remove config usage (to deps)

* remove handler-ui file
2023-12-28 18:18:32 +01:00
Yuta Hayashibe
c64e858a23
Fix typos (#756)
Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>
2023-10-22 17:43:49 +02:00
Monirzadeh
c05d617fbd
feat: allow per-user settings and store them in database (#639)
* create needed field in sqlite database

* update account model

* update Account struct for save Account options

* update sqlite database return account settings

* save configure in sqlite as text and return that

* read configure from user account and defualt configure for shiori

* add api/ui for update settings in database user can save settings in database (in sqlite database)

* check configures be in json format before save in database

* support MariaDB

* fix wrong comment

* support PostgreSQL

* revert unneeded change in new logic

* change configures to config

* change SaveAccount to SaveAccountSettings

* add migrate database scripts

* change default in migration scrtipts

* update model

* read config field as json from database

* fix parse value config value & update config update

* update default value for new user

* update settings variable name to reflect database value in UI

* fix typo

* not panic if user not exist and update worng comment

* visitor user can update there settings now

* remove unneeded loading dialog

* fix typo

* update function for pg and mysql

* remove IsJson

* move scan method to model

* simplify jsonify

* simplify assignees value to account.Config

* missing part of function

* fix some typo and unneeded field in struct

* add down migrate script for all database

* change createEbook to CreateEbook

* use json instead of text in mysql and postgres

* implement

* remove unneeded part

* remove unneeded jsonify in code

* return SelectContext and GetContext

* remove defualt config in reques for new user it will be set in backend

* New API

* remove legacy API

* remove validateSessionWithoutOwnerStatus

* remove Jsonify function don't need that anymore

* add unit test for database

* update migrate script name

* change put to patch

* return PUT

* fix Patch problem and now use PATCH instead of PUT

* remove unneeded retuen

* more cleaner code for request new settings

* fix bug to handle string in Scan method thanks to fmartingr

* fix Authorization & use GetAccount & remove username from request

* shiori-settings remove and it read from shiori-account

* add swagger documentation

* API unit test

* fix typo

* remove unneeded coment

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>

* better Documentation

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>

* shiori-toke remove on logout

* fix typo

* add unit test check update config in database

* update swag documentation

* fix swag formaing error

---------

Co-authored-by: Felipe Martin <812088+fmartingr@users.noreply.github.com>
2023-10-15 15:27:54 +02:00
Felipe Martin
6f19c12c95
Start working on new REST API. Refactor logic in domains. (#497)
* added 404 template

* added auth domain

* added embed file for frontend

* added base config and dependencies

* added basic new http server

* added separated server command

* updated go modules

* removed modd file

* Added shortcut to send internal server error response

* Added JWT support to Auth Domain

* Added JWT support to API

* docs: added comments to response struct

* naming

* inline returns

* updated dependencies

* production logger

* bookmarks endpoint

* reverted old views api path

* frontend for api v1

* proper 404 error (not working atm)

* use response

* removed 404 html

* server error handler

* login and basic auth

* adjusted session duration

* properly retrieve tags

* properly delete bookmark

* cleanup

* archiver domain

* debug routes

* bookmark routes

* expiration by parameter

* move to logrus

* logout

* frontend cache

* updated dependencies

* http: migrated to gin

* linted

* Added version command

* unit tests, docs

* response test utils and tests

* remove logout handler

* auth

* createtag

* improved http test utilities

* assert message equals

* Remove 1.19 from test matrix

* moved api to v1 folder

* docs: contribute docs

* updated makefile

* updated usage docs

* warn in server command

* updaed docs with shiori version command

* Updated documentation

* deps: update
2023-07-17 14:30:18 +01:00
Felipe Martin
3fcb295d72
Make migrations transparent to the user (#530)
* refactor: remove migrate command

* reafactor: avoid migration errors on no changes
2022-11-27 15:39:27 +01:00
Felipe Martin
c86cf12638
fix: remove createnewid usages (#520)
* remove CreateNewID usages

* remove CreateNewID implementations and definition

* added missing sqlite envvar to compose file
2022-10-15 23:01:52 +02:00
Felipe Martin
d1f0ce8dbb
fix(api-ext): use same save logic as the api (#518)
* switched session priority to header then cookie

* fix(db): GetBookmarks handle no rows error

* fix(api-ext): using same save logic as the api
2022-10-15 14:16:12 +02:00
Acelya
ed5a3bcbb9
fix: failed to save bookmark: constraint failed: UNIQUE constraint failed: bookmark.url (#515)
After a first bookmark save, all subsequent ones fail because the update
query does not include an identifier and the update is done on all rows.

Introduced by 05fee53bd0
2022-10-13 19:18:40 +02:00
Felipe Martin Garcia
05fee53bd0
fix: saving bookmarks inconsistencies (#500)
* chore: updated go-migrate dependencies

* fix: specify if we're saving bookmarks expecting a creation

up until now the SaveBookmarks method was doing some "magic" to do
"upserts" on the databases, but consistency between engines was scarce
and not knowing if we were expecting saving a new bookmark or updating
an existing one was leading to errors and inconsistencies in logic all
around the place. Now we need to specify a creation boolean when
saving and a differnt query will be make (INSERT vs UPDATE).

* fix(api): using incorrect bookmark for content downlaod

* test(db): added test pipeline for databases

Added functions that will share logic among the engines and will be
called on fresh databases on each test run

* dev: added basic docker-compose for development

* chore: uncommented tests

* ci(test): added mysql service

* typo

* test(mysql): select database after reset

* fix(mysql): ignore empty row errors when parsing tags

* fix(mysql): handle insert errors

* chore: added mysql variables to compose

* ci: explicit mysql service port exposed
2022-10-11 23:47:38 +02:00
Felipe Martin Garcia
d0210447d2
fix: import should respect add_date or last_modified fields (#490)
* fix: handle last_modified and add_date in imports

* refactor: moved date format to a constant

* fix: use provided modified date when set
2022-10-09 17:05:52 +02:00
hulb
09f2465065
refactor:pass ctx to db layer, return err instead of recover panic (#450) 2022-09-30 12:19:36 +02:00
hulb
8732fd7b6d
implement migration for mysql & postgres (#422) 2022-05-03 23:30:28 +02:00
Felipe Martin Garcia
81d52a2e24
feat: sqlite migrations (#398)
- Moved migrations from code to SQL files
- Using golang-migrate/v4
- Added a new CLI command: migrate
2022-03-26 08:33:57 +01:00
Felipe Martin Garcia
a76b121098
fix: golangci-lint errors (#366) 2022-02-13 16:38:27 +01:00
Dean Jackson
32f9d2695f Explicit charset for MySQL connection 2019-12-13 08:52:26 +01:00
Radhi Fadlillah
9ba9d1eced Fix: wrong query in PostgreSQL 2019-10-03 21:47:03 +07:00
Radhi Fadlillah
a4dd1c8687 Specify character set when creating MySQL table #188 2019-10-03 21:02:25 +07:00
Dean Jackson
988f2d6502 Use single quotes for better compatibility 2019-09-22 19:15:04 +02:00
Dean Jackson
ddf7c37208 Fix MySQL column definitions
- Add AUTO_INCREMENT to `id` fields
- Fix DEFAULT definitions on TEXT/BLOB fields

TEXT/BLOB field default values must be expressions:
https://dev.mysql.com/doc/refman/8.0/en/data-type-defaults.html
2019-09-22 18:58:26 +02:00
Dean Jackson
76a7856f80 Add specific size to bookmark_url key
MySQL can't index an entire TEXT field; the key must be truncated
https://stackoverflow.com/a/1827099
2019-09-22 18:49:11 +02:00
Radhi Fadlillah
df98c281d3 Make MySQL address changeable #156 2019-09-22 08:25:34 +07:00
Radhi Fadlillah
c22f904bdd If there are no owner yet, allow to use default account 2019-08-22 14:34:36 +07:00
Radhi Fadlillah
0c4d75f773 Add basic account level management 2019-08-12 19:57:18 +07:00
Radhi Fadlillah
4e76288e09 Add method to exclude or include some tags 2019-08-11 15:55:31 +07:00
Radhi Fadlillah
667e5b2e74 Initial MySQL support 2019-08-10 08:13:13 +07:00