From 1526495b625d965fbc682361b31cb11b078ec9ba Mon Sep 17 00:00:00 2001 From: deajan Date: Wed, 26 Feb 2025 12:27:38 +0100 Subject: [PATCH] Rework README file --- README.md | 308 ++++++++++++++++++++---------------------------------- 1 file changed, 111 insertions(+), 197 deletions(-) diff --git a/README.md b/README.md index 1e9e168..fcddfdf 100644 --- a/README.md +++ b/README.md @@ -12,18 +12,31 @@ Includes an orchestrator that can schedule checks and retention policy operation Works on x64 **Linux** , **NAS** solutions based on arm/arm64, **Windows** x64 and x86 and MacOS X. -![image](img/interface_v3.0.0.png) - -## V3 Release candidate available -Please check https://github.com/netinvent/npbackup/releases +![image](img/grafana_dashboard_20250211.png) ## Features -- Multiple repositories support - - Repository group settings - - Group operations -- Data deduplication and fast zstd compression* -- Client side data encryption* +#### User experience + +- Multi repository support (with group settings inheritance) +- Group operations via orchestrator +- Housekeeping operations that run checks and retention policies on all requested backup repositories +- First class Windows support + - VSS snapshots* + - VSS snapshot fallback + - Automatic cloud file exclusions (reparse points) + - Windows pre-built executables* +- Easy configuration via GUI or CLI yaml files +- Remote automatic self upgrade capabilities + - Included upgrade server ready to run in production +- Fullblown CLI interface with optional --json API mode +- End User GUI + - Optional viewer only binaries + - Internationalization support (en, fr as of Apr 2024) +- Comes with full exclusion lists for Linux, Synology and Windows OSes + +#### Availability + - Wide storage backend support* - local files* - SFTP* @@ -34,43 +47,42 @@ Please check https://github.com/netinvent/npbackup/releases - Google Cloud Storage* - OpenStack Swift* - Alibaba Cloud (Aliyun) Object Storage System (OSS)* + - Rclone support - Resume on interrupted backups* -- Full CLI interface with all previous options, including --json API mode - - Checks for recent backups before launching a backup -- End User GUI - - Backups create, list, viewer and restore - - Full configuration interface - - Internationalization support (en, fr as of Apr 2024) -- Performance - - Backup process and IO priority settings - - Upload / download speed limits* - - Remote connectivity concurrency settings* -- Per repo / group retention policies - - Will also avoid wrong date data destruction via optional NTP queries -- Housekeeping option which will perform check, forget and prune in one run -- Comes with full exclusion lists for Linux and Windows -- First class prometheus support - - Restic results metric generation - - Grafana dashboard included - - node_exporter file collector support - - Optional push gateway metrics uploading -- First class Windows support - - VSS snapshots* - - Automatic cloud file exclusions (reparse points) - - Windows pre-built executables* -- Additional security - - Repository uri / password, http metrics and upgrade server passwords are AES-256 encrypted - - Repository permissions allowing to limit clients + +#### Performance + +- Data deduplication and fast zstd compression* +- Backup process and IO priority settings +- Upload / download speed limits* +- Remote connectivity concurrency settings* + +#### Security + +- Client side data encryption* +- Repository uri / password, http metrics and upgrade server passwords are AES-256 encrypted +- Permission system to limit client operations: - Backup only permission - Backup, list and restore permissions - Restore only permissions - Full permissions including destructive operations - - Encrypted data viewing requires additional password - - AES-256 keys can't be guessed in executables thanks to Nuitka Commercial compiler - - External AES-256 keys are obfuscated -- Easy configuration via YAML file (or via GUI) -- Remote automatic self upgrade capacity - - Included upgrade server ready to run in production +- Optional manager password to edit permissions & view encrypted data +- AES-256 keys can't be easily guessed in executables thanks to Nuitka Commercial compiler and programming hygiene +- External AES-256 keys obfuscation + +#### Integrity + +- Housekeeping operation that checks all repositories before running retention policies +- Retention policies can be executed with opt-in NTP queries in order to avoid date attacks +- Check / repair repository options* + +#### Logging + +- First class prometheus monitoring support + - Restic results metric generation + - Grafana dashboard included + - node_exporter file collector support + - Optional push gateway metrics uploading (*) Feature provided by [restic](https://restic.net) backup backend @@ -78,14 +90,52 @@ Please check https://github.com/netinvent/npbackup/releases So, a new backup solution out of nowhere, packed with too much features for it's own good ? Not really ! -NPBackup relies on the well known [restic](https://restic.net) backup program, which has been battle proven for years. -While restic is a fanstastic program, NPBackup expands restic by offering a wider set of features. -Still, NPBackup repos are basically managed by restic, and can be viewed / restored / maintained by restic. +NPBackup relies on the well known [restic](https://restic.net) backup program, which has been battle proven for years. +While restic is a fanstastic program, NPBackup expands restic by offering a wider set of features and ecosystem integration. +Still, NPBackup repos are basically managed by restic, and can be viewed / restored / maintained by standalone restic, so data availability is guaranteed. +## Philosophy + +NPBackup is a multiparadigm backup solution which tries to solve two major backup problems, server and laptop backups ! +The core design idea is to make backups available (obviously !), even partial, on system / network failures. + +> It's always better to have some data than none + +The NPBackup design is the result of a multitude of reald world experience. Examples include: + +- For instance, one can configure a "minimum backup size" under which NPBackup reports the backup as failed. Imagine a user moving all it's data from folder A to folder B, while only folder A is setup for backup. Most backup solutions would not complain, but when the time comes for a restore operation, there would be an empty folder. NPBackup informs the monitoring in that scenarios. + +- pre and post exec commands have timeouts. There are way too many situations where a pre command is stuck, so backup never happens. + +- retention policies can check an optional NTP server. Imagine the scenario where you keep last 12 months of backups, but someone setups your computer's date to 2037 ! On next retention run, you would loose all backups. Also, retention policy is configured by default to always keep at least the 3 most recents backups, regardless of their date. + +> It's always better to have too many alerts than none + +It is a set and forget solution. Still, every operation can send a metric to a prometheus monitoring system. Because forgetting the admin burden is different + +#### The difficulty of laptop backups + +As a matter of fact, achieving laptop backups isn't easy. No one can predict when a laptop is on, and when it has internet access. A scheduled backup strategy will fail randomly in that case. + +NPBackup solves this by checking every couple of minutes if a recent backup exists. +If a recent backup already exists, it will recheck later, or else, it will launch a backup immediately. +In order to avoid a sluggish user experience while the backup is in progress, process and io priority are set to low by default. +Note that both the check interval and what's called recent backup (like a less than 24h old backup) options can be configured. +NPBackup also triggers Windows VSS, but still will proceed with backups on VSS failures, allowing a backup to exist in all scenarios. + +#### The usual server backup solution + +It's always most advisable to do server backups as complete disk image, which is especially easy when run as virtual machine. +But in a world of cloud servers, one needs a backup solution that will run inside a VM (or baremetal), and backup various services likes files and/or databases where snapshots/dump operations are required. + +NPBackup can run pre-exec and post-exec commands (with timeouts and failure action), can accept direct database dumps via stdin_from_command parameter, can trigger VSS on windows, can filter + +Backups are run by setting up a scheduled task / cron job manually or via integrated task creation. + ## Quickstart NPBackup consists of four programs: -- npbackup-cli: CLI version of the backup program +- npbackup-cli: CLI version of the backup program (see --help) - npbackup-gui: GUI version of the backup program, useful to create YAML config files and for end users, can also act as cli - npbackup-viewer: View and restore restic repositories without configuration - upgrade_server: Separate server to provide npbackup clients with newer binaries @@ -93,47 +143,27 @@ NPBackup consists of four programs: You may install npbackup via PyPI or use the pre-built executables. ### Prebuilt executables -On linux, uncompress the NPBackup tar.gz corresponding to your platform to `/usr/local/bin`, or wherever you want. + +On linux, uncompress the NPBackup tar.gz archive corresponding to your platform to `/usr/local/bin`, or wherever you want. You'll get `/usr/local/bin/npbackup-cli`, `/usr/local/bin/npbackup-gui` or `/usr/local/bin/npbackup-viewer` depending on the flavor you have chosen. You'll need to have write permissionds to `/var/log`. If not, you can specify an alternative log file via `--log-file /my/writable/path/npbackup.log` -Any distribution with glibc >= 2.17 should do. + +Linux distributions comes in two flavors: +- legacy: any distribution with glibc >= 2.17 should work (compiled on RHEL 7) +- non legacy: any distribution with glibc >= 2.34 should work (compiled on RHEL 9) +- arm and arm64 builds are compiled on Debian stretch for use with glibc > 2.2 On Windows, you should decompress zip archive and run the executable inside it. The x64 binary is compatible with Windows 10+. The x86 binary is compatible with windows Vista and higher. On those old systems, you might need to install Visual C runtime 2015. -### PyPI installation +For other install methods, please see the [wiki](https://github.com/netinvent/npbackup/wiki) -If you don't want to use the pre-built executables, you can install via pip with `pip install npbackup` or `pip install npbackup==3.0.0rc14` currently. - -Python minimal requirement: 3.7+ -Note that using Python 3.8+ is recommended in order to have better memory usage. - -Note that if you want to use the GUI, you'll also need to install tkinter via `yum install python-tkinter` or `apt install python3-tk`. - -### Usage - -You can check the documentation on the [wiki](https://github.com/netinvent/npbackup/wiki) page in order to learn to use NPBackup. - -Copy the example config from model `examples/npbackup.conf.dist` into the directory where npbackup is installed. -Also copy the `excludes` directory if you plan to use the prefilled bigger exclusion lists for your backups. - -You can adjust the parameters directly in the file, or via a config GUI by launching `npbackup-gui --config-file=npbackup.conf` - -Once configured, you can launch manual backups via `npbackup-cli --backup`. Those can be scheduled. -Windows schedule can be created from the configuration page. On Linux, you'll have to create a cronjob or a systemd timer. - -Since NPBackup is configured to only proceed with backups when no recent backups are detected, you should consider scheduling npbackup executions quite often. -The default schedule should be somewhere around 15 minutes. - -You can use `npbackup-cli --list` or the GUI to list backups. - -The GUI allows an end user to check current backups & restore files.rom backups: - -The YAML configuration file encrypts sensitive data so the end user doesn't have to know repository URI or password. ## Quickstart GUI -Just run the npbackup-gui executable and configure it. +![image](img/interface_v3.0.0.png) + +Launch the npbackup-gui executable and setup a new configuration. Prebuilt binaries can be found [here](https://github.com/netinvent/npbackup/releases) ![image](img/restore_window_v3.0.0.png) @@ -149,18 +179,6 @@ Configuration allows to edit the YAML configuration files directly as end user Orchestrator GUI allows to run commands on multiple repositories or groups. -**Security** -NPBackup' security model relies on symmetric encryption of all sensitive data that allows to access a repository. -In order to achieve this, NPBackup contains an AES-KEY that can be set: - - at compile time - - at run time via an AES-KEY file - -Please note that right clicking on "" in the configuration GUI will allow to decrypt that data, by prompting a backup admin password. -That password is set at compile-time and should be different depending on the organization. - -This allows a system admin to see repo URI and passwords, without leaving this information available on the computer. - -The configuration file should never be world readable, as one could change the backup admin password, allowing to decrypt other parts of the conf file. ### NPBackup viewer @@ -171,23 +189,6 @@ If no configuration file nor environment variables are set, it will ask for repo ![image](img/viewer_v3.0.0.png) -## The difficulty of laptop backups - -As a matter of fact, laptop backups are the hardest. No one can predict when a laptop is on, and if it has access to internet. -Creating a backup strategy in those cases isn't a simple task. - -NPBackup solves this by checking every 15 minutes if a backup newer than 24h exists. -If so, nothing is done. In the case no recent backup exists, a backup is immediately launched. -OF course, both time options are configurable. -In order to avoid sluggish user experience while backing up, process and io priority can be lowered. -Once done, NPBackup can send backup results in Prometheus format directly to a push gateway, for prometheus to catch them. - -## A good server backup solution - -Server backups can be achieved by setting up a scheduled task / cron job manually or via integrated task creation. - -Of course, since NPBackup supports pre-exec and post-exec commands, it can be used to backup various services like virtual hosts or databases where snapshot/dump operations are required (especially with new stdin_from_command parameter). - ## Monitoring NPBackup includes full prometheus support, including grafana dashboard. @@ -196,7 +197,7 @@ On servers, we'll configure a prometheus file that gets written on each backup, On laptops, since we might be away from our usual network, we'll push the backup metrics to a remote push gateway which laters gets collected by prometheus itself. The current NPBackup dashboard: -![image](img/grafana_dashboard_2.2.0.png) +![image](img/grafana_dashboard_20250211.png) ## End user expericence @@ -204,53 +205,7 @@ While admin user experience is important, NPBackup also offers a GUI for end use ## CLI usage -`npbackup-cli` has all the functions the GUI has, and can run on any headless server. -It also has a `--json` parameter which guarantees parseable output. - -You may run operations on multiple repositories, or repositories groups by specifying parameter `--repo-group` or `--repo-name`. -`--repo-name` allows to specify one or multiple comma separated repo names, also allows special `__all__` argument which selects all repositories. -`--repo-group` allows to specify one or multiple comme separated repo group names, also allows special `__all__` argument which selects all groups. - -Multi-repo example: -``` -npbackup-cli --housekeeping --repo-group default_group -``` - -`npbackup-gui` can also act as cli if run with `--run-as-cli` parameter, allowing to use a single executable as GUI and scheduled tasks. - -## Security - -NPBackup inherits all security measures of it's backup backend (currently restic with AES-256 client side encryption including metadata) and all security options from it's storage backends. - -On top of those, NPBackup itself encrypts sensitive information like the repo uri and password, as well as the metrics http username and password. -This ensures that end users can backup/restore data without the need to know any password, avoiding secret compromission. -Note that NPBackup uses an AES-256 key itself, in order to encrypt sensitive data. The public (git) version of NPBackup uses the default encryption key that comes with the official NPBackup repo. - -You can generate a new AES-256 key with `npbackup-cli --create-key npbackup.key` and use it via an environment variable: - -Use a file -``` -export NPBACKUP_KEY_LOCATION=/path/to/npbackup.key -``` - -Use a command that provides the key as it's output -``` -export NPBACKUP_KEY_COMMAND=my_key_command -``` - -You may also compile your own NPBackup executables that directly contain the AES-256 key. See instructions in PRIVATE directory to setup keys. - -## Permission restriction system - -By default, npbackup is allowed to execute all operations on a repo. -There are some situations where an administrator needs to restrict repo operations for end users. -In that case, you can set permissions via the GUI or directly in the configuration file. - -Permissions are: -- full: Set by default, allows all including destructive operations -- restore: Allows everything backup does plus restore, check and dump operations -- restore_noly: Allows only restoring backups, but not creating ones -- backup: Allows, backup, snapshot/object listing operations and repo unlocking +Please refer to the wiki for CLI usage details. ## Logs @@ -261,53 +216,11 @@ You can also use `--log-file` to specify an alternative log file path. ## Upgrade server NPBackup comes with integrated auto upgrade function that will run regardless of program failures, in order to lessen the maintenance burden. -The upgrade server runs a python asgi web server with integrated HTTP basic authentication, that can further be put behind an SSL proxy like HaProxy. +The upgrade server runs a python asgi web server with integrated HTTP basic authentication, that can further be put behind an SSL proxy like HaProxy. See the wiki for more. ## Branding -NPBackup is fully customizable and branding can be done easily. -You'll find all images and icons in the `resources` directory. Feel free to update those files. Once updated, you must run `update_custom_resources.py` to regenerate the `customization.py` file corresponding entries. -The file `customization.py` also contains OEM strings that can be safely changed, except for the license text which must stay unaltered. - -## Compilation - -In order to fully protect the AES key that is needed to support NPBackup, one can compile the program with Nuitka. -Compiling needs restic binary for the target platform in `RESTIC_SOURCE_FILES` folder, files must be named `restic_{version}_{platform}_{arch}[.extension]` like provided by restic.net or [github](https://github.com/restic/restic) -Linux binaries need to be made executable in the `RESTIC_SOURCE_FILES` folder. -You can use `update_restic.sh` script in `RESTIC_SOURCE_FILES` to download binaries. - -You'll need to change the default AES key in `secrets.py`, see the documentation in the file itself. - -One you're setup, you need to install compilation tools, see `requirements-compilation.txt` file. - -Compile options are available in `compile.py`. - -We maintain a special 32 bit binary for Windows 7 which allows to backup those old machines until they get replaced. - -We also compile our linux target on RHEL 7 in order to be compatible with reasonably old distributions (>= glibc 2.17). - -arm and arm64 builds are compiled on Debian stretch for use with glibc > 2.24. - -On most Linux distributions, you might get your glibc version by running `find /usr -name "libc.so.6" -exec "{}" \;` - -## Smart shield, antivirus and reputation - -Official binaries for Windows provided by NetInvent are signed with a certificate, allowing to gain trust and reputation in antivirus analysis. -Also, official binaries are compiled using Nuitka Commercial grade, which is more secure in storing secrets. - -Pre-compiled builds for Windows have been code signed with NetInvent's EV certificate, using [windows_tools.signtool](github.com/netinvent/windows_tools) -Signing on a Windows machine with Windows SDK installed can be done via `windows\sign_windows.py` script. -Alternatively, you can sign executables via: -``` -from windows_tools.signtool import SignTool -signer = SignTool() -signer.sign(r"c:\path\to\executable", bitness=64) -``` - -Or as singleliner to use in scripts: -``` -python.exe -c "from windows_tools.signtool import SignTool; s=SignTool(); s.sign(r'C:\GIT\npbackup\BUILDS\public\windows\x64\npbackup-viewer-x64.exe')" -``` +NPBackup is fully customizable and branding can be done easily. See the compilation instructions in the wiki for more. ## Misc @@ -327,3 +240,4 @@ Currently supported: `en-US`, `fr-FR` - Thanks to the Restic Team without which this program would not be possible - Thanks to https://github.com/Krutyi-4el who packaged i18nice internationalization for us - Special thanks to the BTS SIO 2nd year class of 2022 at Lycee Marillac / Perpignan who volunteered as GUI Q&A team +- Thanks to @Guitarbilly for his efforts in reporting bugs, asking questions, and general improvement suggestions \ No newline at end of file