netinvent/npbackup
1
1
Fork 0
mirror of https://github.com/netinvent/npbackup.git synced 2025-09-30 16:54:52 +08:00
Code Issues Projects Releases Packages Wiki Activity

Rework README file

Browse source
This commit is contained in:
deajan 2025-02-26 12:27:38 +01:00
parent 6699388e27
commit 1526495b62
1 changed files with 111 additions and 197 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

308
README.md
View file

@ -12,18 +12,31 @@ Includes an orchestrator that can schedule checks and retention policy operation
Works on x64 **Linux** , **NAS** solutions based on arm/arm64, **Windows** x64 and x86 and MacOS X.
![image](img/interface_v3.0.0.png)
## V3 Release candidate available
Please check https://github.com/netinvent/npbackup/releases
![image](img/grafana_dashboard_20250211.png)
## Features
- Multiple repositories support
- Repository group settings
- Group operations
- Data deduplication and fast zstd compression*
- Client side data encryption*
#### User experience
- Multi repository support (with group settings inheritance)
- Group operations via orchestrator
- Housekeeping operations that run checks and retention policies on all requested backup repositories
- First class Windows support
- VSS snapshots*
- VSS snapshot fallback
- Automatic cloud file exclusions (reparse points)
- Windows pre-built executables*
- Easy configuration via GUI or CLI yaml files
- Remote automatic self upgrade capabilities
- Included upgrade server ready to run in production
- Fullblown CLI interface with optional --json API mode
- End User GUI
- Optional viewer only binaries
- Internationalization support (en, fr as of Apr 2024)
- Comes with full exclusion lists for Linux, Synology and Windows OSes
#### Availability
- Wide storage backend support*
- local files*
- SFTP*
@ -34,43 +47,42 @@ Please check https://github.com/netinvent/npbackup/releases
- Google Cloud Storage*
- OpenStack Swift*
- Alibaba Cloud (Aliyun) Object Storage System (OSS)*
- Rclone support
- Resume on interrupted backups*
- Full CLI interface with all previous options, including --json API mode
- Checks for recent backups before launching a backup
- End User GUI
- Backups create, list, viewer and restore
- Full configuration interface
- Internationalization support (en, fr as of Apr 2024)
- Performance
- Backup process and IO priority settings
- Upload / download speed limits*
- Remote connectivity concurrency settings*
- Per repo / group retention policies
- Will also avoid wrong date data destruction via optional NTP queries
- Housekeeping option which will perform check, forget and prune in one run
- Comes with full exclusion lists for Linux and Windows
- First class prometheus support
- Restic results metric generation
- Grafana dashboard included
- node_exporter file collector support
- Optional push gateway metrics uploading
- First class Windows support
- VSS snapshots*
- Automatic cloud file exclusions (reparse points)
- Windows pre-built executables*
- Additional security
- Repository uri / password, http metrics and upgrade server passwords are AES-256 encrypted
- Repository permissions allowing to limit clients
#### Performance
- Data deduplication and fast zstd compression*
- Backup process and IO priority settings
- Upload / download speed limits*
- Remote connectivity concurrency settings*
#### Security
- Client side data encryption*
- Repository uri / password, http metrics and upgrade server passwords are AES-256 encrypted
- Permission system to limit client operations:
- Backup only permission
- Backup, list and restore permissions
- Restore only permissions
- Full permissions including destructive operations
- Encrypted data viewing requires additional password
- AES-256 keys can't be guessed in executables thanks to Nuitka Commercial compiler
- External AES-256 keys are obfuscated
- Easy configuration via YAML file (or via GUI)
- Remote automatic self upgrade capacity
- Included upgrade server ready to run in production
- Optional manager password to edit permissions & view encrypted data
- AES-256 keys can't be easily guessed in executables thanks to Nuitka Commercial compiler and programming hygiene
- External AES-256 keys obfuscation
#### Integrity
- Housekeeping operation that checks all repositories before running retention policies
- Retention policies can be executed with opt-in NTP queries in order to avoid date attacks
- Check / repair repository options*
#### Logging
- First class prometheus monitoring support
- Restic results metric generation
- Grafana dashboard included
- node_exporter file collector support
- Optional push gateway metrics uploading
(*) Feature provided by [restic](https://restic.net) backup backend
@ -78,14 +90,52 @@ Please check https://github.com/netinvent/npbackup/releases
So, a new backup solution out of nowhere, packed with too much features for it's own good ? Not really !
NPBackup relies on the well known [restic](https://restic.net) backup program, which has been battle proven for years.
While restic is a fanstastic program, NPBackup expands restic by offering a wider set of features.
Still, NPBackup repos are basically managed by restic, and can be viewed / restored / maintained by restic.
NPBackup relies on the well known [restic](https://restic.net) backup program, which has been battle proven for years.
While restic is a fanstastic program, NPBackup expands restic by offering a wider set of features and ecosystem integration.
Still, NPBackup repos are basically managed by restic, and can be viewed / restored / maintained by standalone restic, so data availability is guaranteed.
## Philosophy
NPBackup is a multiparadigm backup solution which tries to solve two major backup problems, server and laptop backups !
The core design idea is to make backups available (obviously !), even partial, on system / network failures.
> It's always better to have some data than none
The NPBackup design is the result of a multitude of reald world experience. Examples include:
- For instance, one can configure a "minimum backup size" under which NPBackup reports the backup as failed. Imagine a user moving all it's data from folder A to folder B, while only folder A is setup for backup. Most backup solutions would not complain, but when the time comes for a restore operation, there would be an empty folder. NPBackup informs the monitoring in that scenarios.
- pre and post exec commands have timeouts. There are way too many situations where a pre command is stuck, so backup never happens.
- retention policies can check an optional NTP server. Imagine the scenario where you keep last 12 months of backups, but someone setups your computer's date to 2037 ! On next retention run, you would loose all backups. Also, retention policy is configured by default to always keep at least the 3 most recents backups, regardless of their date.
> It's always better to have too many alerts than none
It is a set and forget solution. Still, every operation can send a metric to a prometheus monitoring system. Because forgetting the admin burden is different
#### The difficulty of laptop backups
As a matter of fact, achieving laptop backups isn't easy. No one can predict when a laptop is on, and when it has internet access. A scheduled backup strategy will fail randomly in that case.
NPBackup solves this by checking every couple of minutes if a recent backup exists.
If a recent backup already exists, it will recheck later, or else, it will launch a backup immediately.
In order to avoid a sluggish user experience while the backup is in progress, process and io priority are set to low by default.
Note that both the check interval and what's called recent backup (like a less than 24h old backup) options can be configured.
NPBackup also triggers Windows VSS, but still will proceed with backups on VSS failures, allowing a backup to exist in all scenarios.
#### The usual server backup solution
It's always most advisable to do server backups as complete disk image, which is especially easy when run as virtual machine.
But in a world of cloud servers, one needs a backup solution that will run inside a VM (or baremetal), and backup various services likes files and/or databases where snapshots/dump operations are required.
NPBackup can run pre-exec and post-exec commands (with timeouts and failure action), can accept direct database dumps via stdin_from_command parameter, can trigger VSS on windows, can filter
Backups are run by setting up a scheduled task / cron job manually or via integrated task creation.
## Quickstart
NPBackup consists of four programs:
- npbackup-cli: CLI version of the backup program
- npbackup-cli: CLI version of the backup program (see --help)
- npbackup-gui: GUI version of the backup program, useful to create YAML config files and for end users, can also act as cli
- npbackup-viewer: View and restore restic repositories without configuration
- upgrade_server: Separate server to provide npbackup clients with newer binaries
@ -93,47 +143,27 @@ NPBackup consists of four programs:
You may install npbackup via PyPI or use the pre-built executables.
### Prebuilt executables
On linux, uncompress the NPBackup tar.gz corresponding to your platform to `/usr/local/bin`, or wherever you want.
On linux, uncompress the NPBackup tar.gz archive corresponding to your platform to `/usr/local/bin`, or wherever you want.
You'll get `/usr/local/bin/npbackup-cli`, `/usr/local/bin/npbackup-gui` or `/usr/local/bin/npbackup-viewer` depending on the flavor you have chosen.
You'll need to have write permissionds to `/var/log`. If not, you can specify an alternative log file via `--log-file /my/writable/path/npbackup.log`
Any distribution with glibc >= 2.17 should do.
Linux distributions comes in two flavors:
- legacy: any distribution with glibc >= 2.17 should work (compiled on RHEL 7)
- non legacy: any distribution with glibc >= 2.34 should work (compiled on RHEL 9)
- arm and arm64 builds are compiled on Debian stretch for use with glibc > 2.2
On Windows, you should decompress zip archive and run the executable inside it.
The x64 binary is compatible with Windows 10+. The x86 binary is compatible with windows Vista and higher. On those old systems, you might need to install Visual C runtime 2015.
### PyPI installation
For other install methods, please see the [wiki](https://github.com/netinvent/npbackup/wiki)
If you don't want to use the pre-built executables, you can install via pip with `pip install npbackup` or `pip install npbackup==3.0.0rc14` currently.
Python minimal requirement: 3.7+
Note that using Python 3.8+ is recommended in order to have better memory usage.
Note that if you want to use the GUI, you'll also need to install tkinter via `yum install python-tkinter` or `apt install python3-tk`.
### Usage
You can check the documentation on the [wiki](https://github.com/netinvent/npbackup/wiki) page in order to learn to use NPBackup.
Copy the example config from model `examples/npbackup.conf.dist` into the directory where npbackup is installed.
Also copy the `excludes` directory if you plan to use the prefilled bigger exclusion lists for your backups.
You can adjust the parameters directly in the file, or via a config GUI by launching `npbackup-gui --config-file=npbackup.conf`
Once configured, you can launch manual backups via `npbackup-cli --backup`. Those can be scheduled.
Windows schedule can be created from the configuration page. On Linux, you'll have to create a cronjob or a systemd timer.
Since NPBackup is configured to only proceed with backups when no recent backups are detected, you should consider scheduling npbackup executions quite often.
The default schedule should be somewhere around 15 minutes.
You can use `npbackup-cli --list` or the GUI to list backups.
The GUI allows an end user to check current backups & restore files.rom backups:
The YAML configuration file encrypts sensitive data so the end user doesn't have to know repository URI or password.
## Quickstart GUI
Just run the npbackup-gui executable and configure it.
![image](img/interface_v3.0.0.png)
Launch the npbackup-gui executable and setup a new configuration.
Prebuilt binaries can be found [here](https://github.com/netinvent/npbackup/releases)
![image](img/restore_window_v3.0.0.png)
@ -149,18 +179,6 @@ Configuration allows to edit the YAML configuration files directly as end user
Orchestrator GUI allows to run commands on multiple repositories or groups.
**Security**
NPBackup' security model relies on symmetric encryption of all sensitive data that allows to access a repository.
In order to achieve this, NPBackup contains an AES-KEY that can be set:
- at compile time
- at run time via an AES-KEY file
Please note that right clicking on "<encrypted data>" in the configuration GUI will allow to decrypt that data, by prompting a backup admin password.
That password is set at compile-time and should be different depending on the organization.
This allows a system admin to see repo URI and passwords, without leaving this information available on the computer.
The configuration file should never be world readable, as one could change the backup admin password, allowing to decrypt other parts of the conf file.
### NPBackup viewer
@ -171,23 +189,6 @@ If no configuration file nor environment variables are set, it will ask for repo
![image](img/viewer_v3.0.0.png)
## The difficulty of laptop backups
As a matter of fact, laptop backups are the hardest. No one can predict when a laptop is on, and if it has access to internet.
Creating a backup strategy in those cases isn't a simple task.
NPBackup solves this by checking every 15 minutes if a backup newer than 24h exists.
If so, nothing is done. In the case no recent backup exists, a backup is immediately launched.
OF course, both time options are configurable.
In order to avoid sluggish user experience while backing up, process and io priority can be lowered.
Once done, NPBackup can send backup results in Prometheus format directly to a push gateway, for prometheus to catch them.
## A good server backup solution
Server backups can be achieved by setting up a scheduled task / cron job manually or via integrated task creation.
Of course, since NPBackup supports pre-exec and post-exec commands, it can be used to backup various services like virtual hosts or databases where snapshot/dump operations are required (especially with new stdin_from_command parameter).
## Monitoring
NPBackup includes full prometheus support, including grafana dashboard.
@ -196,7 +197,7 @@ On servers, we'll configure a prometheus file that gets written on each backup,
On laptops, since we might be away from our usual network, we'll push the backup metrics to a remote push gateway which laters gets collected by prometheus itself.
The current NPBackup dashboard:
![image](img/grafana_dashboard_2.2.0.png)
![image](img/grafana_dashboard_20250211.png)
## End user expericence
@ -204,53 +205,7 @@ While admin user experience is important, NPBackup also offers a GUI for end use
## CLI usage
`npbackup-cli` has all the functions the GUI has, and can run on any headless server.
It also has a `--json` parameter which guarantees parseable output.
You may run operations on multiple repositories, or repositories groups by specifying parameter `--repo-group` or `--repo-name`.
`--repo-name` allows to specify one or multiple comma separated repo names, also allows special `__all__` argument which selects all repositories.
`--repo-group` allows to specify one or multiple comme separated repo group names, also allows special `__all__` argument which selects all groups.
Multi-repo example:
```
npbackup-cli --housekeeping --repo-group default_group
```
`npbackup-gui` can also act as cli if run with `--run-as-cli` parameter, allowing to use a single executable as GUI and scheduled tasks.
## Security
NPBackup inherits all security measures of it's backup backend (currently restic with AES-256 client side encryption including metadata) and all security options from it's storage backends.
On top of those, NPBackup itself encrypts sensitive information like the repo uri and password, as well as the metrics http username and password.
This ensures that end users can backup/restore data without the need to know any password, avoiding secret compromission.
Note that NPBackup uses an AES-256 key itself, in order to encrypt sensitive data. The public (git) version of NPBackup uses the default encryption key that comes with the official NPBackup repo.
You can generate a new AES-256 key with `npbackup-cli --create-key npbackup.key` and use it via an environment variable:
Use a file
```
export NPBACKUP_KEY_LOCATION=/path/to/npbackup.key
```
Use a command that provides the key as it's output
```
export NPBACKUP_KEY_COMMAND=my_key_command
```
You may also compile your own NPBackup executables that directly contain the AES-256 key. See instructions in PRIVATE directory to setup keys.
## Permission restriction system
By default, npbackup is allowed to execute all operations on a repo.
There are some situations where an administrator needs to restrict repo operations for end users.
In that case, you can set permissions via the GUI or directly in the configuration file.
Permissions are:
- full: Set by default, allows all including destructive operations
- restore: Allows everything backup does plus restore, check and dump operations
- restore_noly: Allows only restoring backups, but not creating ones
- backup: Allows, backup, snapshot/object listing operations and repo unlocking
Please refer to the wiki for CLI usage details.
## Logs
@ -261,53 +216,11 @@ You can also use `--log-file` to specify an alternative log file path.
## Upgrade server
NPBackup comes with integrated auto upgrade function that will run regardless of program failures, in order to lessen the maintenance burden.
The upgrade server runs a python asgi web server with integrated HTTP basic authentication, that can further be put behind an SSL proxy like HaProxy.
The upgrade server runs a python asgi web server with integrated HTTP basic authentication, that can further be put behind an SSL proxy like HaProxy. See the wiki for more.
## Branding
NPBackup is fully customizable and branding can be done easily.
You'll find all images and icons in the `resources` directory. Feel free to update those files. Once updated, you must run `update_custom_resources.py` to regenerate the `customization.py` file corresponding entries.
The file `customization.py` also contains OEM strings that can be safely changed, except for the license text which must stay unaltered.
## Compilation
In order to fully protect the AES key that is needed to support NPBackup, one can compile the program with Nuitka.
Compiling needs restic binary for the target platform in `RESTIC_SOURCE_FILES` folder, files must be named `restic_{version}_{platform}_{arch}[.extension]` like provided by restic.net or [github](https://github.com/restic/restic)
Linux binaries need to be made executable in the `RESTIC_SOURCE_FILES` folder.
You can use `update_restic.sh` script in `RESTIC_SOURCE_FILES` to download binaries.
You'll need to change the default AES key in `secrets.py`, see the documentation in the file itself.
One you're setup, you need to install compilation tools, see `requirements-compilation.txt` file.
Compile options are available in `compile.py`.
We maintain a special 32 bit binary for Windows 7 which allows to backup those old machines until they get replaced.
We also compile our linux target on RHEL 7 in order to be compatible with reasonably old distributions (>= glibc 2.17).
arm and arm64 builds are compiled on Debian stretch for use with glibc > 2.24.
On most Linux distributions, you might get your glibc version by running `find /usr -name "libc.so.6" -exec "{}" \;`
## Smart shield, antivirus and reputation
Official binaries for Windows provided by NetInvent are signed with a certificate, allowing to gain trust and reputation in antivirus analysis.
Also, official binaries are compiled using Nuitka Commercial grade, which is more secure in storing secrets.
Pre-compiled builds for Windows have been code signed with NetInvent's EV certificate, using [windows_tools.signtool](github.com/netinvent/windows_tools)
Signing on a Windows machine with Windows SDK installed can be done via `windows\sign_windows.py` script.
Alternatively, you can sign executables via:
```
from windows_tools.signtool import SignTool
signer = SignTool()
signer.sign(r"c:\path\to\executable", bitness=64)
```
Or as singleliner to use in scripts:
```
python.exe -c "from windows_tools.signtool import SignTool; s=SignTool(); s.sign(r'C:\GIT\npbackup\BUILDS\public\windows\x64\npbackup-viewer-x64.exe')"
```
NPBackup is fully customizable and branding can be done easily. See the compilation instructions in the wiki for more.
## Misc
@ -327,3 +240,4 @@ Currently supported: `en-US`, `fr-FR`
- Thanks to the Restic Team without which this program would not be possible
- Thanks to https://github.com/Krutyi-4el who packaged i18nice internationalization for us
- Special thanks to the BTS SIO 2nd year class of 2022 at Lycee Marillac / Perpignan who volunteered as GUI Q&A team
- Thanks to @Guitarbilly for his efforts in reporting bugs, asking questions, and general improvement suggestions