npbackup/CHANGELOG

## Shortly planned
 
  ! - Add policy like restic forget --keep-within-daily 30d --keep-within-weekly 1m --keep-within-monthly 1y --keep-within-yearly 3y
  default policy restic forget --keep-within-hourly 72h --keep-within-daily 30d --keep-within-weekly 1m --keep-within-monthly 1y --keep-within-yearly 3y
      - With button to select --keep-x or --keep-within-x
      - with button to select forget/prune operation
      - With explanation link to restic
    - prune operation: default max unsed ?
    - Operation planifier:
      - backups
      - cleaner (forget / check / prune, each having different planification (at least for check --read-data))
      - Launch now
   - NPBackup Operation mode
      - manages multiple repos with generic key (restic key add) or specified key
  !- NTP server
  - GUI --help parameter should intercept stdout and show results in a popup

 
## 3.0.0
 - This is a major rewrite that allows using multiple repositories, adds repository groups and implements repository settings inheritance from group settings

  !- New operation planifier for backups / cleaning / checking repos
  !- Implemented retention policies
  !  - Optional time server update to make sure we don't drift before doing retention operations
  !  - Optional repo check before doing retention operations
  !- Pre and post-execution scripts
  !  - Multiple pre and post execution scripts are now allowed
  !  - Post-execution script can now be force run on error / exit
  !  - Script result now has prometheus metrics


## Features
  - New viewer mode allowing to browse/restore restic repositories without any NPBackup configuation
    - Allows setting repository via environment variables, config file or directly in GUI
  - Multi repository support
  - Group settings for repositories
  !- Operation center
    - GUI operation center allowing to mass execute actions on repos / groups
    - CLI operation center via `--group-operation --repo-group=somegroup`
    !- Implemented retention policies
    !- Operation planifier allows to create scheduled tasks for operations
      !- Implemented scheduled task creator for Windows & Unix
         !(simple list of tasks, actions, stop on error)
    - Implemented repo quick check / full check / repair index / repair snapshots / unlock / forget / prune / dump / stats commands
  - Added per repo permission management
    - Repos now have backup, restore and full privileges, optionally allowing to restrict access for end users
  - Added snapshot tag to snapshot list on main window
  - Split npbackup into separate CLI and GUI
    - Status window has been refactored so GUI now has full stdout / stderr returns from runner and backend
  - Implemented file size based exclusion
  - CLI can now fully operate in API compatible mode via --json parameter
    - Parses non json compliant restic output
    - Always returns a result boolean and a reason when failing
  - CLI now accepts --stdin parameter to backup streams sent to CLI
  - Added minimum backup size upon which we declare that backup has failed
  - All bytes units now have automatic conversion of units (K/M/G/T/P bits/bytes or IEC bytes)
  - Refactored GUI and overall UX of configuration
  - New option --show-config to show inheritance in CLI mode (GUI has visual indicators)
  - Allow using external restic binary via --external-backend-binary parameter in CLI mode

## Fixes
  - Default exit code is now worst log level called unless specific errors are triggered
  - Show anonymized repo uri in GUI
  - Fix deletion failed message for en lang
  - Fix Google cloud storage backend detection in repository uri
  - Backup admin password is now stored in a more secure way

## Misc
  - Current backup state now shows more precise backup state, including last backup date when relevant
  - Concurrency checks (pidfile checks) are now directly part of the runner
  - Allow a 30 seconds grace period for child processes to close before asking them nicely, and than not nicely to quit
  - Fully refactored prometheus metrics parser to be able to read restic standard or json outputs
  - Reimplmented auto upgrade after CLI/GUI split
  - Added initial tests

## 2.2.2 - 14/12/2023 (internal build only)
  - Fixed backup paths of '/' root partitions
  - Properly display repository init errors
  - Logs didn't show proper error status
  - Fixed wrong init detection for S3 backend

## 2.2.1 - 28/08/2023
  - Added snapshot deletion option in GUI
  - Fix ignore_cloud_files parameter did not work on some non systems (still an issue, see https://github.com/restic/restic/issues/4155)
  - Configured default backend connections to 2 for local, 8 for remote when default value '0' is given
  - Fine adjust GOMAXPROCS environment variable (n for 1 core, n-1 for 2-4 cores, n-2 for 5+ cores)
  - Updated internal restic binary to 0.16.0 compiled with go1.20.6
  - Compiled with newer Nuitka 1.7.10 that allows use of Python 3.11
  - Updated python to version 3.11 for Windows x64 targets
  - Basic MacOS X compile support

## 2.2.0 - rtm - 03/06/2023
  - Fix potential deadlock in evaluate variables
  - Fix additionnal parameters should only apply for backup operations
  - Fix unnecessary exclude statements when no exclusions are given
  - Added source types (--files-from, --files-from-verbatim and --files-from-raw equivalent)
  - Add encrypted environment variables support
  - Add SSL certificate verification bypass for prometheus metrics
  - Make sure all multiline entries in config files are processed as lists
  - Add exclude-patterns to GUI (was present in CLI version already)
  - Make sure we always quote exclusions patterns or files
  - Updated internal restic binary to 0.15.2 compiled with go1.20.3
  - *SECURITY* Don't log password command, even in debug mode
  - Compiled with Nuitka commercial 1.5.8

## 2.2.0 - rc12 - 03/05/2023
  - Add grafana dashboard to examples directory
  - Implement password command functionality
  - Compiled with newer Nuitka commercial 1.5.7
  - Added windows executable signature script

## v2.2.0 - rc11 - 24/04/2023
  - Add default 1 minute timeout for init operations
  - Fix auto_upgrade_group wasn't transmitted to server
  - Make sure new binary is set as executable in linux upgrade
  - *SECURITY* Show unencrypted is only allowed on compiled builds where default AES KEY is not used
    - This prevents a password injection from another conf file
  - *SECURITY* Prevent raw password update in config file
  - Fix empty password makes GUI fail
  - Fix empty password makes CLI fail
  - Fix empty data is shown as encrypted
  - Enforce repository and passwords to be strings, also remove trailing and ending spaces
  - Add warning when source and destination are the same path
  - Compiled with newer Nuitka 1.5.6+ that leverages lesser virus detection issues

## v2.2.0 - rc10 - 30/03/2023
  - Add arm/arm64 compilation support
  - Add an option to disable GUI compilation

## v2.2.0 - rc9 - 28/03/2023
  - Allow AES key rotation by keeping old key and updating encryption on config loading
  - Make sure restic binary is executable for Nuitka builds
  - Improve compiler script
  - Add messages when no tcl/tk exists in GUI mode
  - Switch to i18nice fork of python-i18n

## v2.2.0 - rc8 - 22/03/2023
  - Highly optimize tree window creation times (x3.5) by replacing dateutil.parser.parse()
  - Fix missing locale might fail on certain linux builds
  - Don't build NPBackupInstaller on non Windows platforms
  - Add --gui-status parameter and more GUI related info
  - Make translation manager more resilient
  - Add an inline copy of maintained python-i18n to deal with portability issues (see requirements.txt for more)

## v2.2.0 - rc7 - 20/03/2023
  - New look and feel
  - Cosmetic fixes (button sizes)

## v2.2.0 - rc6 - 13/03/2023
  - Fixup bogus restic command line when empty tag lines were given
  - Added missing default value for ignoring cloud files in the example configuration
  - Made sure npbackup can run when no tkinter is installed (headless)

## v2.2.0 - rc5 - 03/03/2023
  - Fixed bogus restic command line when empty tag lines were given
  - Make sure the no config dialog stays on top
  - Make sure upgrade client checks current python arch instead of os arch
  - Make upgrade client catch referrer for /current_version calls
  - Upgrade python-i18n to a maintained version
  - Make sure we get the real IP when client downloads an upgrade and server is behind proxy server

## v2.2.0 - rc4 - 08/02/2023
  - Add gui popup when config cannot be saved
  - Make repo connection messages more clear
  - Remove default backup admin password from config
  - Fix empty restic results will make prometheus metrics fail

## v2.2.0 - rc3 - 07/02/2023
  - Auto minimize console window when run in GUI mode to not distract end user
  - Trigger GUI refresh after configuration changes
  - Main program now loads directly after first time GUI setup
  - A popup is now shown on non-readable repos
  - Various smaller fixes
  - Support for --config-file parameter in upgrade_server
  - Upgrade server now fetches the real client IP if behind a proxy
  - Upgrade server systemd service file example

## v2.2.0 - rc2 - 03/02/2023
  - Upgrade client now sends its hostname, group and version information to server
  - Add variable substitution in configuration
  - Add machine identification in configuration
  - Add variable ${RANDOM}[n] to config file in order to distinguish non unique hostnames
  - Refactor config variable handling
  - Make backup state and upgrade state messages more coherent
  - Fix config fails when restic password is an int
  - Fix empty config files did not show a proper error message
  - Fix various config file malformation will break execution
  - Fix backup hangs when no restic password is given (restic asks for password in backgroud job)
  - Fix error message in logs when repo is not initialized

## v2.2.0 - rc1 - 02/02/2023
  - Added a full auto-upgrade solution:
    - Aupgrade client integrated into NPBackup, that can be called manually via --auto-upgrade or automatically run every n backups
    - Upgrade server which servers files and their metadata
  - Added a gui to create a scheduled task under Windows
  - Added a gui to configure backup admin password (was compile time setting before)
  - Improved setup.py to provide launch scripts for both Linux and Windows platforms
  - Made windows cloud file filter optional (enabled by default)
  - Added default configuration settings
  - Rewrite configuration gui into a tabbed gui
  - Updated restic binary to restic 0.15.1 compiled with go1.19.5
  - Changed backup interval unit from seconds to minutes

## v2.1.0 - 29/01/2023
  - Added execution time information
  - Code cleanup
    - Refactor runner into a class
    - Refactor npbackup into package
    - Adding missing typing
    - Code is now formatted with black
    - Code must now pass pylint and flake8
  - Workaround for painfully slow sg.TreeData window.close()
  - Simplify npbackup shortcut
  - Shift default interval from 23H30 to 24H
  - Fixed missing translations for admin password popups

## v2.0.0 - 24/02/2023

  - First open public release
  - Refactor GUI so actions become non-blocking
    - Added loader animation for gui events
    - Added backup window with current state
    - Added internationalization support (currently en and fr)
  - Refactor restic.ls function to return a generator since restic ls command gives chunks of json, which is bad memory wise when parsing each chunk into a big list
  - On Windows, "The cloud operation is not supported on a read-only volume." and "The media is write protected." errors will not make backup fail, see #4155
    - This fix can be removed once #4155 is resolved, see # TEMP-FIX-4155 comment in restic_wrapper package
  - Added Additional backup parameter option
  - Added optional backup tags
  - Added optional prometheus group label and additional labels entries
  - Make sure backend connections number parameter works with all backends
  - Added priority configuration for restic operations, defaults to low priority to not disturb operations
  - Added WindowsApps folder (reparse points) to exclusion list (fix "The file cannot be accessed by the system." error)
  - Fixed backup did use verbose option twice
  - Updated restic binary to restic 0.15.0 compiled with go1.19.5 on windows/amd64
  - Made theme configurable
  - Make sure we exit all subprocesses when exiting npbackup
  - Improve config GUI for supporting elder config files properly
  - Fix restic metrics when 0 bytes of data have been added to the repo
  - Fix restoration must ignore case on Windows
  - Fix backup_job variable substitution in config file
  - Fix for ruamel.yaml global objects being modified out of current scope encrypting current config while saving
  - Fix npbackup could not start when no config file was present
  - Drastically improve ls operation speed by changing command_runner method from poller to monitor (no live output) except for backup operation
  - Compiler:
    - Compile python 3.7 and 3.10 targets so we also get to use Windows 7 which can't run python > 3.7
  - Installer:
    - Moved task creation process to npbackup so we can recreate tasks on systems where npbackup is already installed

## v1.8.0

  - Implement VSS support for Windows
  - Do not require UAC elevation for npbackup.exe (Still highly recommended, but at least we allow non admin users to backup their data (without VSS))
  - Search restic executable path when not found locally for both Windows and Linux platforms
  - Make exclusion lists always ignore case on Windows systems
  - Highly improve exclusion lists
  - Keep structure and comments in yaml configuration files
  - Simplify configuration file data encryption
  - Fix limit-download parameter not working
  - Fix error when rest_connections was set to 0
  - Add some config gui logging
  - Cosmetic gui fixes
  - Fix never enforce execution timeout on backup operation
  - Fix bogus filename encodings while showing backup contents / usernames
  - Added more restic metrics
  - Fixed bogus restric metrics
  - Allow setting additional prometheus labels
  - Fix gui snapshot contents don't show duplicate parents on Windows when backing up 'c:\foo' and 'C:\bar'
  - Internal code cleanup
  - Installer:
    - Make sure we don't overwrite destination configuration file if existing

## v1.5.3
  - First closed public release

## v0.1.0 - 1.5.2
  - Internal development and tests