Commit graph

68 commits

Author SHA1 Message Date
Frederik Ring
83fa0aae48 Refactor handling of runtime configuration to prepare for reloading 2024-02-16 20:19:58 +01:00
Frederik Ring
9a1e885138
Env vars should propagate when using conf.d (#358)
* Extend confd test case to test for env var propagation

* Env vars set in conf.d files are expected to propagate

* Lock needs to be acquired when instantiating script
2024-02-13 15:43:04 +01:00
Frederik Ring
4c74313222 Periodically collect runtime info when requested 2024-02-12 16:04:12 +01:00
Frederik Ring
65626dd3d4 Hoist control for exiting script a level up (#348)
* Hoist control for exiting script a level up

* Do not accidentally nil out errors

* Log when running schedule

* Remove duplicate log line

* Warn on cron schedule that will never run
2024-02-12 16:04:12 +01:00
pixxon
1d45062100 Move cron scheduling inside application (#338)
* Move cron scheduling inside application

* Make envvar a fallback and check for errors

* Panic significantly less

* propagate error out of runBackup

* Add structured logging

* FIx error propagation to exit

* Enable the new scheduler by default

* Review fixes

* Added docs and better error propagation
2024-02-12 16:04:12 +01:00
Frederik Ring
63b545787e
Exclusive file lock is released prematurely (#339) 2024-02-01 18:14:18 +01:00
Frederik Ring
c3daeacecb
Improve Swarm support (#333)
* Query for labeled services as well

* Try scaling down services

* Scale services back up

* Use progress tool from Docker CLI

* In test, label both services

* Clean up error and log messages

* Document scale-up/down approach in docs

* Downgrade Docker CLI to match client

* Document services stats

* Do not rely on PreviousSpec for storing desired replica count

* Log warnings from Docker when updating services

* Check whether container and service labels collide

* Document script behavior on label collision

* Add additional check if all containers have been removed

* Scale services concurrently

* Move docker interaction code into own file

* Factor out code for service updating

* Time out after five minutes of not reaching desired container count

* Inline handling of in-swarm container level restart

* Timer is more suitable for timeout race

* Timeout when scaling down services should be configurable

* Choose better filename

* Reflect changes in naming

* Rename and deprecate BACKUP_STOP_CONTAINER_LABEL

* Improve logging

* Further simplify logging
2024-01-31 12:17:41 +01:00
Frederik Ring
ed5abd5ba8
Panic handling does not log reason for script being halted (#334) 2024-01-26 20:02:09 +01:00
MaxJa4
5fcc96edf9
Cleanup: Lint warnings and deprecated packages (#263)
* Fix lint warnings and std lib deprecations

* Replace deprecated std lib with maintained drop-in replacement fork

Backwartds compatible with original package and suggested by std lib due to security and stability issues.

* OAuth2 is now a direct dependency due to Dropbox

* Undo change

* Revert "Replace deprecated std lib with maintained drop-in replacement fork"

This reverts commit 2887bd409f.

* Update channel handling

* Add linter for PRs

* Rename CI, fetch all issues, add govet
2023-08-27 18:14:55 +02:00
Frederik Ring
67d978f515
Drop logrus dependency, log using slog package from stdlib (#247) 2023-08-10 19:41:03 +02:00
Frederik Ring
b441cf3e2b
Fine grained labels (#115)
* Refactor label command mechanism to be more flexible

* Run all steps wrapped in labeled commands

* Rename methods to be in line with lifecycle

* Deprecate exec-pre and exec-post labels

* Add documentation

* Use type alias for lifecycle phases

* Fix bad imports

* Fix command lookup for deprecated labels

* Use more generic naming for lifecycle phase

* Fail on erroneous post command

* Update documentation
2022-07-10 10:36:56 +02:00
Frederik Ring
da8c63f755
Support identical cron schedule (#87)
* Retry on lock being unavailable

* Refactor locking to return plain error

* Collect LockedTime in stats

* Add test case

* Add documentation for LOCK_TIMEOUT

* Log in case lock needs to be awaited

* Release resources created for awaiting lock
2022-03-25 18:26:34 +01:00
Frederik Ring
0504a92a1f
Add option to run pre/post commands for any container (#73)
* Add option to run pre commands on arbitrary container

* Correctly handle quoted args in commands

* Provide defaults for test version arg

* Allow filtering of target containers

* Add documentation on exec commands

* Use mysqldump in exec test

* Add mysqldump section to recipes

* Also run commands test in swarm mode

* Use name instead of id

* Add syntax highlighting

* Add missing license headers
2022-02-22 07:53:33 +01:00
Frederik Ring
a57e93d01e
Split source into multiple files, deduplicate pruning logic, do not parse templates when notifications are not used (#63)
* Split code into multiple files

* Deduplicate logic for pruning backups against different storages

* Only parse templates when notifications are enabled

* Use better description
2022-02-13 10:52:19 +01:00
Frederik Ring
3e17d1b123
Ensure end time is recorded for unsuccessful runs too (#62)
* Ensure end time is also recorded for unsuccessful runs

* Clean up integration tests
2022-02-13 09:41:36 +01:00
Mauro Molin
8dfdd14527
Added custom notification messages using text/template (#60)
* Added custom notification messages using text/template

* Change notification template path and removed automatic newline trim

* Added stats and changed structure of template params

* Stat file hotfix

* Embedded and fixed default notification templates


Fix

* Changed Output to LogOutput

* Changed stats integer to unsigned

* Bytes formatting in template func


fix

* Changed Archives to Storages

* Removed unecessary sleep for pruning leeway

* Set EndTime after pruning is completed

* Added custom notifications documentation

* Added 5s sleep in swarm test

* Fixed documentation

* Dockerfile copies all files in cmd/backup
2022-02-11 20:05:16 +01:00
Frederik Ring
6fe629ce87
Allow path to be set for bucket storage (#52) 2022-01-25 21:16:16 +01:00
Frederik Ring
1db896f7cf Tweak README, improve client naming, tidy go.mod file 2022-01-22 13:35:13 +01:00
Kaerbr
6ded00aa06
Support Nextcloud / WebDav (#48)
* add studio-b12/gowebdav to be able to upload to webdav server

* make sure all env variables are present for webdav upload

* implement file upload to WebDav server

directory defaults to the base directory

* docs: add the new feature to the documentation

* if no WebDav env variable are given throw no error

* docs: use more elegant english :D

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>

* docs: use official spelling of "WebDAV"

* perf: golang likes to return early instead of having an else block

* use WEBDAV_PATH instead of WEBDAV_DIRECTORY

* use split_words for more convenience

like shown here: https://github.com/kelseyhightower/envconfig#struct-tag-support

* simplify

* feat: add pruning of files in WebDAV remote

Based on / Inspired by the minio/S3 implementation of pruning remote files.

* remove logging from the development

* test: first try implementing tests

Sandly I have to use the remote pipeline -- local wont work for me.

* test: adapt used volume names

* test: specify image only once!

* test: minio AND webdav data should be present

* test: backups on WebDAV remote are laying in the root directory

* test: the webdav server stores date in /var/lib/dav

* trying with data subfolder

* test: 1 container was added so the number raised from 3 to 4

* webdav  subfolder is "data" not "backup"

* fix: password AND username must be defined

not password OR username

* improve logging

* feat: if the given path on the server isnt preset it will be created

* test: add creation of new folder for webdav to tests

Co-authored-by: Frederik Ring <frederik.ring@gmail.com>
2022-01-22 13:29:21 +01:00
Frederik Ring
32575c831e Also expand env vars in pruning prefix if configured 2021-12-23 09:22:56 +01:00
Frederik Ring
c062710ce8
Allow for env substitution in backup filename (#39) 2021-12-22 14:39:46 +01:00
Frederik Ring
4207146fb6 Refactor calling of hooks on exit 2021-12-18 10:31:12 +01:00
Frederik Ring
1f727f698f Run hooks in order of severity 2021-12-18 10:31:12 +01:00
Frederik Ring
88c90a206c Use int comparison for checking hooks 2021-12-18 10:31:12 +01:00
Frederik Ring
8bad0656b3 Enable notifications on multiple levels 2021-12-18 10:31:12 +01:00
Frederik Ring
08d78a0bd6 allow sending notifications to multiple channels 2021-12-18 10:31:12 +01:00
Frederik Ring
5a6ce81b58 update github.com/otai/copy, use PreserveOwner option 2021-11-29 08:40:55 +01:00
Frederik Ring
7bc5b2ccef fix minor error scoping mistakes 2021-11-28 20:06:24 +01:00
Frederik Ring
210c7d4540
Reuse hook mechanism for scheduling clean up tasks (#33)
* reuse hook mechanism for scheduling clean up tasks

* register hooks before creating files or dirs

* fix logging order

* use typed hook levels
2021-11-08 19:10:10 +01:00
schwannden
411c39ee72
create a snapshot before creating tar archive (#32)
* create a snapshot before creating tar archive

* safeguard snapshot removal and make snapshot optional

* fix typo, make sure remove snapshot failure triggers failure hook

Co-authored-by: Schwannden Kuo <schwannden@mobagel.com>
2021-11-08 08:39:18 +01:00
Frederik Ring
0c666d0c88 use lstat when checking whether file is a symlink 2021-11-03 18:07:55 +01:00
Frederik Ring
a0402b407d fix fileinfo mode comparison when checking for symlinks 2021-11-03 18:03:44 +01:00
Frederik Ring
3193e88fc0 os.FileInfo cannot be used for deleting files as it does not contain a full path 2021-11-02 06:40:37 +01:00
Frederik Ring
f946f36fb0 exclude symlinks from candidates when pruning local files
Previously, symlinks would be included in the set of candidates, but would
be skipped when pruning. This could lead to a wrong number of candidates
being printed in the log messages.
2021-10-29 09:00:37 +02:00
Frederik Ring
5245b5882f update README, save some indentation 2021-10-28 19:55:39 +02:00
schwannden
7f0f173115
adding option to skip tls verification error (#30)
* adding option to skip tls verification error

* merge options

* removed merged option from README

Co-authored-by: Schwannden Kuo <schwannden@mobagel.com>
2021-10-28 19:51:35 +02:00
Frederik Ring
c59b40f2df automatically create symlink to latest local backup if configured 2021-10-01 18:19:24 +02:00
Frederik Ring
bd73a2b5e4 allow s3 authentication via IAM role 2021-09-30 19:24:43 +02:00
Frederik Ring
53c257065e ensure script always tries to remove local artifacts even when backup failed 2021-09-12 10:48:19 +02:00
Frederik Ring
160a47e90b allow registering hooks at different levels 2021-09-09 16:55:49 +02:00
Frederik Ring
59660ec5c7 include exit log message in notification 2021-09-09 11:08:05 +02:00
Frederik Ring
88368197c1 implement email notifications on failed backup runs 2021-09-09 09:00:23 +02:00
Frederik Ring
e46968ed79 call error hooks on script failure 2021-09-09 08:12:07 +02:00
Frederik Ring
2c06f81503 collect all log output in buffer so it could be used in notifications 2021-09-09 07:24:18 +02:00
Frederik Ring
8aba98c012 use forked version of package targz 2021-09-04 10:08:06 +02:00
Frederik Ring
ede94bcd88 display all error messages instead of first one 2021-08-29 19:39:51 +02:00
Frederik Ring
aae97a5617 try restarting even when stopping some containers failed 2021-08-29 18:51:05 +02:00
Frederik Ring
825cbb50ef always use background context directly 2021-08-29 18:26:40 +02:00
Frederik Ring
bea203af3d improve documentation 2021-08-29 18:16:04 +02:00
Frederik Ring
6034e6a902 print proper local archive in log message 2021-08-29 08:36:45 +02:00