diff --git a/CHANGELOG.md b/CHANGELOG.md index fa649530..60100d14 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,28 @@ All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](http://semver.org/). +## [0.7.0] - 2024-04-09 + +This version uses a different database layout and introduces multiple breaking changes in the configuration files. Please read the [UPGRADING.md](UPGRADING.md) file for more information on how to upgrade from previous versions. + +## Added +- Web-based administration interface. +- REST API for management and configuration. +- Automatic RSA and ED25519 DKIM key generation. +- Support for compressing binaries in the blob store (#227). +- Improved performance accessing IMAP mailboxes with a large number of messages. +- Support for custom DNS resolvers. +- Support for multiple loggers with different levels and outputs. + +### Changed + +### Fixed +- Store quotas as `u64` rather than `u32`. +- Second IDLE connections disconnects the first one (#280). +- Use relaxed DNS parsing, allowing underscores in DNS labels (#172). +- Escape regexes within `matches()` expressions (#155). +- ManageSieve LOGOUT should reply with `OK` instead of `BYE`. + ## [0.6.0] - 2024-02-14 This version introduces breaking changes in the configuration file. Please read the [UPGRADING.md](UPGRADING.md) file for more information on how to upgrade from previous versions. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f0495070..ec659418 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,26 +1,35 @@ # Contributing -Thank you for your interest in contributing to the Stalwart Mail Server project! We appreciate your support and enthusiasm. This document provides guidelines for contributing to the project. +When contributing to this repository, please first discuss the change you wish to make via issue, +email, or any other method with the owners of this repository before making a change. -## Not accepting Pull Requests at this time -As of now, the Stalwart Mail Server project is in a phase of rapid development and evolution. Until we reach version 1.0, **we are not open to direct code contributions** in the form of pull requests. This is due to the following reasons: -- The project is undergoing significant changes which might make integrating external contributions challenging. -- Our current resources do not allow for the efficient evaluation and testing of submitted pull requests. +Please note we have a code of conduct, please follow it in all your interactions with the project. -We understand that this might be disappointing, but rest assured, this measure is temporary and is aimed at maintaining the integrity and consistency of the project during its critical development phase. +## Any contributions you make will be under AGPL -## Other Ways to Contribute -While we're not accepting pull requests at this moment, there are several other valuable ways you can contribute to the Stalwart Mail Server project: +This software is licensed under the Affero General Public License (AGPL). Any contributions made to +this project will be under this license. Before any contributions can be made, contributors are +required to sign a Contributor License Agreement (CLA). The purpose of the agreement is to clarify +and document the rights granted by contributors to us. -- **Enhancement Requests**: If you have an idea for a new feature or an improvement, we encourage you to file an enhancement request on GitHub. This way, your idea can be evaluated and potentially included in future versions of the project. To do this, create a new issue on our GitHub repository and label it as an 'enhancement'. -- **Extensive Testing**: One of the most helpful contributions is to rigorously test the software in various environments and use-cases. Your testing can uncover important issues and help improve the stability of the project. -- **Writing Unit Tests**: Contributing new unit tests or improving existing ones is a great way to ensure the reliability and robustness of our codebase. -- **Reviewing Documentation**: As the project evolves, so does the need for accurate and up-to-date documentation. Reviewing, updating, or writing new documentation is a critical contribution that helps new and existing users understand and effectively use Stalwart Mail Server. +## Pull Request Process -## Stay Updated -We anticipate opening up for code contributions once version 1.0 is launched. We recommend keeping an eye on our project announcements for any updates regarding contribution opportunities. +1. Ensure any install or build dependencies are removed before the end of the layer when doing a + build. +2. Update the README.md with details of changes to the interface, this includes new environment + variables, exposed ports, useful file locations and container parameters. +3. Increase the version numbers in any examples files and the README.md to the new version that this + Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/). +4. You may merge the Pull Request in once you have the sign-off of two other developers, or if you + do not have permission to do that, you may request the second reviewer to merge it for you. -## Questions or Suggestions? -If you have any questions or suggestions regarding the contribution process, feel free to open a discussion on our GitHub repository. +## Code of Conduct -Thank you for your support and understanding. Together, we are building a robust and reliable Stalwart Mail Server! +We as members, contributors, and leaders pledge to make participation in our community a harassment-free +experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex +characteristics, gender identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity and orientation. +We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, +and healthy community. + +You can read the full Code of Conduct [here](https://github.com/stalwartlabs/.github/blob/main/CODE_OF_CONDUCT.md). diff --git a/README.md b/README.md index 595d2349..4094de3e 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,13 @@ Key features: - Sieve scripting language with support for all [registered extensions](https://www.iana.org/assignments/sieve-extensions/sieve-extensions.xhtml). - Email aliases, mailing lists, subaddressing and catch-all addresses support. - Integration with **OpenTelemetry** to enable monitoring, tracing, and performance analysis. +- **Web-based administration**: + - Account, domain, group and mailing list management. + - SMTP queue management for messages and outbound DMARC and TLS reports. + - Report visualization interface for received DMARC, TLS-RPT and Failure (ARF) reports. + - Configuration of every aspect of the mail server. + - Log viewer with search and filtering capabilities. + - Self-service portal for password reset and encryption-at-rest key management. - **Secure and robust**: - Encryption at rest with **S/MIME** or **OpenPGP**. - Automatic TLS certificate provisioning with [ACME](https://datatracker.ietf.org/doc/html/rfc8555). @@ -106,7 +113,7 @@ Additionally you may become a sponsor to obtain priority support from Stalwart L - [x] Performance enhancements - [x] Distributed SMTP queues -- [ ] Web-based admin panel +- [x] Web-based admin panel - [ ] JMAP Calendar, Contacts and Tasks support - [ ] CalDAV and CardDAV support diff --git a/SECURITY.md b/SECURITY.md index 9e8d73df..e1c6b18f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -6,9 +6,9 @@ We provide security updates for the following versions of Stalwart Mail Server: | Version | Supported | | ------- | ------------------ | -| 0.5.x | :white_check_mark: | -| 0.4.x | :white_check_mark: | -| < 0.3 | :x: | +| 0.7.x | :white_check_mark: | +| 0.6.x | :white_check_mark: | +| < 0.5 | :x: | ## Reporting a Vulnerability diff --git a/UPGRADING.md b/UPGRADING.md index 11f2fdb5..b891215b 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -1,3 +1,38 @@ +Upgrading from `v0.6.0` to `v0.7.0` +----------------------------------- + +Version `0.7.0` of Stalwart Mail Server introduces significant improvements and features that enhance performance and functionality. However, it also comes with multiple breaking changes in the configuration files and a revamped database layout optimized for accessing large mailboxes. Additionally, Stalwart now supports compression for binaries stored in the blob store, further increasing efficiency. +Due to these extensive changes, the recommended approach for upgrading is to perform a clean reinstallation of Stalwart and manually migrate your accounts to the new version. + +## Pre-Upgrade Steps +- Download the `v0.7.0` mail-server and CLI binaries for your platform from the [releases page](https://github.com/stalwartlabs/mail-server/releases/latest/). +- Initialize the setup on a distinct directory using the command `sudo ./stalwart-mail -- init /path/to/new-install`. This command will print the administrator password required to access the web-admin. +- Create the `bin` directory using `mkdir /path/to/new-install/bin`. +- Move the downloaded binaries to the `bin` directory using the command `mv stalwart-mail stalwart-cli /path/to/new-install/bin`. +- Open `/path/to/new-install/etc/config.toml` in a text editor and comment out all listeners except the HTTP listener for port `8080`. +- Start the new installation from the terminal using the command `sudo /path/to/new-install/bin/stalwart-mail --config /path/to/new-install/etc/config.toml`. +- Point your browser to the web-admin at `http://yourserver.org:8080` and login using the auto-generated administrator password. +- Configure the new installation with your domain, hostname, certificates, and other settings following the instructions at [stalw.art/docs/get-started](https://stalw.art/docs/get-started). Ignore the part about using the installation script, we are performing a manual installation. +- Add your user accounts. +- Configure Stalwart to run as the `stalwart-mail` user and `stalwart-mail` group from `Settings` > `Server` > `System`. This is not necessary if you are using Docker. +- Stop the new installation by pressing `Ctrl+C` in the terminal. + +## Upgrade Steps +- On your `v0.6.0` installation, open in a text editor the `smtp/listener.toml`, `imap/listener.toml` files and comment out all listeners except the JMAP/HTTP listener (we are going to need it to export the user accounts) and then restart the service. +- If you are using an external store, backup the database using the appropriate method for your database system. +- Create the `~/exports` directory, here we will store the exported accounts. +- Using the existing CLI tool (not the one you just downloaded as it is not compatible), export each user account using the command `./stalwart-cli -u https://your-old-server.org -c export account ~/exports`. +- Stop the `v0.6.0` installation using the command `sudo systemctl stop stalwart-mail`. +- Move the old `v0.6.0` installation to a backup directory, for example `mv /opt/stalwart-mail /opt/stalwart-mail-backup`. +- Move the new `v0.7.0` installation to the old installation directory, for example `mv /path/to/new-install /opt/stalwart-mail`. +- Set the right permissions for the new installation using the command `sudo chown -R stalwart:stalwart /opt/stalwart-mail`. +- Start the new installation using the command `sudo systemctl start stalwart-mail`. +- Import the accounts using the new CLI tool with the command `./stalwart-cli -u http://yourserver.org:8080-c import ~/exports/`. +- Using the admin tool, reactivate all the necessary listener (SMTP, IMAP, etc.) +- Restart the service using the command `sudo systemctl restart stalwart-mail`. + +We apologize for the complexity of the upgrade process associated with this version of Stalwart. We understand the challenges and inconveniences that the requirement for a clean reinstallation and manual account migration poses. Moving forward, an automated migration tool will be included in any future releases that necessitate changes to the database layout, aiming to streamline the upgrade process for you. Furthermore, as we approach the milestone of version 1.0.0, we anticipate that such foundational changes will become increasingly infrequent, leading to more straightforward updates. We appreciate your patience and commitment to Stalwart during this upgrade. + Upgrading from `v0.5.3` to `v0.6.0` ----------------------------------- diff --git a/install.sh b/install.sh index 740431af..f14e1007 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,7 @@ #!/usr/bin/env sh # shellcheck shell=dash -# Stalwart Mail install script -- based on the rustup installation script. +# Stalwart Mail Server install script -- based on the rustup installation script. set -e set -u @@ -34,26 +34,43 @@ main() { _account="_stalwart-mail" fi - # Start configuration mode - if [ "$#" -eq 1 ] && [ "$1" = "--init" ] ; then - init - configure - return 0 - fi + # Read arguments + local _dir="/opt/stalwart-mail" + + # Default component setting + local _component="stalwart-mail" + + # Loop through the arguments + for arg in "$@"; do + case "$arg" in + --fdb) + _component="stalwart-mail-foundationdb" + ;; + *) + if [ -n "$arg" ]; then + _dir=$arg + fi + ;; + esac + done # Detect platform architecture get_architecture || return 1 local _arch="$RETVAL" assert_nz "$_arch" "arch" + # Create directories + ensure mkdir -p "$_dir" "$_dir/bin" "$_dir/etc" "$_dir/logs" + # Download latest binary - say "⏳ Downloading installer for ${_arch}..." - local _dir - _dir="$(ensure mktemp -d)" - local _file="${_dir}/stalwart-install.tar.gz" - local _url="${BASE_URL}/stalwart-install-${_arch}.tar.gz" + say "⏳ Downloading ${_component} for ${_arch}..." + local _file="${_dir}/bin/stalwart-mail.tar.gz" + local _url="${BASE_URL}/${_component}-${_arch}.tar.gz" ensure mkdir -p "$_dir" ensure downloader "$_url" "$_file" "$_arch" + ensure tar zxvf "$_file" -C "$_dir/bin" + ignore chmod +x "$_dir/bin/stalwart-mail" + ignore rm "$_file" # Create system account if ! id -u ${_account} > /dev/null 2>&1; then @@ -86,16 +103,92 @@ main() { fi fi - # Copy binary - say "⬇️ Running installer..." - ensure tar zxvf "$_file" -C "$_dir" - ignore $_dir/stalwart-install - ignore rm "$_file" - ignore rm "$_dir/stalwart-install" + # Run init + ignore $_dir/bin/stalwart-mail --init "$_dir" + + # Set permissions + say "🔐 Setting permissions..." + ensure chown -R ${_account}:${_account} "$_dir" + ensure chmod -R 755 "$_dir" + ensure chmod 700 "$_dir/etc/config.toml" + + # Create service file + say "🚀 Starting service..." + if [ "${_os}" = "linux" ]; then + printf "\n[server.run-as]\nuser = \"stalwart-mail\"\ngroup = \"stalwart-mail\"\n" >> "$_dir/etc/config.toml" + create_service_linux "$_dir" + elif [ "${_os}" = "macos" ]; then + create_service_macos "$_dir" + fi + + # Installation complete + local _host=$(hostname) + say "🎉 Installation complete! Continue the setup at http://$_host:8080/login" return 0 } +# Functions to create service files +create_service_linux() { + local _dir="$1" + cat < /etc/systemd/system/stalwart-mail.service +[Unit] +Description=Stalwart Mail Server Server +Conflicts=postfix.service sendmail.service exim4.service +ConditionPathExists=__PATH__/etc/config.toml +After=network-online.target + +[Service] +Type=simple +LimitNOFILE=65536 +KillMode=process +KillSignal=SIGINT +Restart=on-failure +RestartSec=5 +ExecStart=__PATH__/bin/stalwart-mail --config=__PATH__/etc/config.toml +PermissionsStartOnly=true +StandardOutput=syslog +StandardError=syslog +SyslogIdentifier=stalwart-mail + +[Install] +WantedBy=multi-user.target +EOF + systemctl daemon-reload + systemctl enable stalwart-mail.service + systemctl restart stalwart-mail.service +} + +create_service_macos() { + local _dir="$1" + cat < /Library/LaunchAgents/stalwart.mail.plist + + + + + Label + stalwart.mail + ServiceDescription + Stalwart Mail Server + ProgramArguments + + __PATH__/bin/stalwart-mail + --config=__PATH__/etc/config.toml + + RunAtLoad + + KeepAlive + + + +EOF + launchctl load /Library/LaunchAgents/stalwart.mail.plist + launchctl enable system/stalwart.mail + launchctl start system/stalwart.mail +} + + get_architecture() { local _ostype _cputype _bitness _arch _clibtype _ostype="$(uname -s)" @@ -369,7 +462,7 @@ get_endianness() { } say() { - printf 'stalwart-mail: %s\n' "$1" + printf '%s\n' "$1" } err() { diff --git a/install_new.sh b/install_new.sh deleted file mode 100644 index f14e1007..00000000 --- a/install_new.sh +++ /dev/null @@ -1,734 +0,0 @@ -#!/usr/bin/env sh -# shellcheck shell=dash - -# Stalwart Mail Server install script -- based on the rustup installation script. - -set -e -set -u - -readonly BASE_URL="https://github.com/stalwartlabs/mail-server/releases/latest/download" - -main() { - downloader --check - need_cmd uname - need_cmd mktemp - need_cmd chmod - need_cmd mkdir - need_cmd rm - need_cmd rmdir - need_cmd tar - - # Make sure we are running as root - if [ "$(id -u)" -ne 0 ] ; then - err "❌ Install failed: This program needs to run as root." - fi - - # Detect OS - local _os="unknown" - local _uname="$(uname)" - _account="stalwart-mail" - if [ "${_uname}" = "Linux" ]; then - _os="linux" - elif [ "${_uname}" = "Darwin" ]; then - _os="macos" - _account="_stalwart-mail" - fi - - # Read arguments - local _dir="/opt/stalwart-mail" - - # Default component setting - local _component="stalwart-mail" - - # Loop through the arguments - for arg in "$@"; do - case "$arg" in - --fdb) - _component="stalwart-mail-foundationdb" - ;; - *) - if [ -n "$arg" ]; then - _dir=$arg - fi - ;; - esac - done - - # Detect platform architecture - get_architecture || return 1 - local _arch="$RETVAL" - assert_nz "$_arch" "arch" - - # Create directories - ensure mkdir -p "$_dir" "$_dir/bin" "$_dir/etc" "$_dir/logs" - - # Download latest binary - say "⏳ Downloading ${_component} for ${_arch}..." - local _file="${_dir}/bin/stalwart-mail.tar.gz" - local _url="${BASE_URL}/${_component}-${_arch}.tar.gz" - ensure mkdir -p "$_dir" - ensure downloader "$_url" "$_file" "$_arch" - ensure tar zxvf "$_file" -C "$_dir/bin" - ignore chmod +x "$_dir/bin/stalwart-mail" - ignore rm "$_file" - - # Create system account - if ! id -u ${_account} > /dev/null 2>&1; then - say "🖥️ Creating '${_account}' account..." - if [ "${_os}" = "macos" ]; then - local _last_uid="$(dscacheutil -q user | grep uid | awk '{print $2}' | sort -n | tail -n 1)" - local _last_gid="$(dscacheutil -q group | grep gid | awk '{print $2}' | sort -n | tail -n 1)" - local _uid="$((_last_uid+1))" - local _gid="$((_last_gid+1))" - - ensure dscl /Local/Default -create Groups/_stalwart-mail - ensure dscl /Local/Default -create Groups/_stalwart-mail Password \* - ensure dscl /Local/Default -create Groups/_stalwart-mail PrimaryGroupID $_gid - ensure dscl /Local/Default -create Groups/_stalwart-mail RealName "Stalwart Mail service" - ensure dscl /Local/Default -create Groups/_stalwart-mail RecordName _stalwart-mail stalwart-mail - - ensure dscl /Local/Default -create Users/_stalwart-mail - ensure dscl /Local/Default -create Users/_stalwart-mail NFSHomeDirectory /Users/_stalwart-mail - ensure dscl /Local/Default -create Users/_stalwart-mail Password \* - ensure dscl /Local/Default -create Users/_stalwart-mail PrimaryGroupID $_gid - ensure dscl /Local/Default -create Users/_stalwart-mail RealName "Stalwart Mail service" - ensure dscl /Local/Default -create Users/_stalwart-mail RecordName _stalwart-mail stalwart-mail - ensure dscl /Local/Default -create Users/_stalwart-mail UniqueID $_uid - ensure dscl /Local/Default -create Users/_stalwart-mail UserShell /bin/bash - - ensure dscl /Local/Default -delete /Users/_stalwart-mail AuthenticationAuthority - ensure dscl /Local/Default -delete /Users/_stalwart-mail PasswordPolicyOptions - else - ensure useradd ${_account} -s /sbin/nologin -M - fi - fi - - # Run init - ignore $_dir/bin/stalwart-mail --init "$_dir" - - # Set permissions - say "🔐 Setting permissions..." - ensure chown -R ${_account}:${_account} "$_dir" - ensure chmod -R 755 "$_dir" - ensure chmod 700 "$_dir/etc/config.toml" - - # Create service file - say "🚀 Starting service..." - if [ "${_os}" = "linux" ]; then - printf "\n[server.run-as]\nuser = \"stalwart-mail\"\ngroup = \"stalwart-mail\"\n" >> "$_dir/etc/config.toml" - create_service_linux "$_dir" - elif [ "${_os}" = "macos" ]; then - create_service_macos "$_dir" - fi - - # Installation complete - local _host=$(hostname) - say "🎉 Installation complete! Continue the setup at http://$_host:8080/login" - - return 0 -} - -# Functions to create service files -create_service_linux() { - local _dir="$1" - cat < /etc/systemd/system/stalwart-mail.service -[Unit] -Description=Stalwart Mail Server Server -Conflicts=postfix.service sendmail.service exim4.service -ConditionPathExists=__PATH__/etc/config.toml -After=network-online.target - -[Service] -Type=simple -LimitNOFILE=65536 -KillMode=process -KillSignal=SIGINT -Restart=on-failure -RestartSec=5 -ExecStart=__PATH__/bin/stalwart-mail --config=__PATH__/etc/config.toml -PermissionsStartOnly=true -StandardOutput=syslog -StandardError=syslog -SyslogIdentifier=stalwart-mail - -[Install] -WantedBy=multi-user.target -EOF - systemctl daemon-reload - systemctl enable stalwart-mail.service - systemctl restart stalwart-mail.service -} - -create_service_macos() { - local _dir="$1" - cat < /Library/LaunchAgents/stalwart.mail.plist - - - - - Label - stalwart.mail - ServiceDescription - Stalwart Mail Server - ProgramArguments - - __PATH__/bin/stalwart-mail - --config=__PATH__/etc/config.toml - - RunAtLoad - - KeepAlive - - - -EOF - launchctl load /Library/LaunchAgents/stalwart.mail.plist - launchctl enable system/stalwart.mail - launchctl start system/stalwart.mail -} - - -get_architecture() { - local _ostype _cputype _bitness _arch _clibtype - _ostype="$(uname -s)" - _cputype="$(uname -m)" - _clibtype="gnu" - - if [ "$_ostype" = Linux ]; then - if [ "$(uname -o)" = Android ]; then - _ostype=Android - fi - if ldd --version 2>&1 | grep -q 'musl'; then - _clibtype="musl" - fi - fi - - if [ "$_ostype" = Darwin ] && [ "$_cputype" = i386 ]; then - # Darwin `uname -m` lies - if sysctl hw.optional.x86_64 | grep -q ': 1'; then - _cputype=x86_64 - fi - fi - - if [ "$_ostype" = SunOS ]; then - # Both Solaris and illumos presently announce as "SunOS" in "uname -s" - # so use "uname -o" to disambiguate. We use the full path to the - # system uname in case the user has coreutils uname first in PATH, - # which has historically sometimes printed the wrong value here. - if [ "$(/usr/bin/uname -o)" = illumos ]; then - _ostype=illumos - fi - - # illumos systems have multi-arch userlands, and "uname -m" reports the - # machine hardware name; e.g., "i86pc" on both 32- and 64-bit x86 - # systems. Check for the native (widest) instruction set on the - # running kernel: - if [ "$_cputype" = i86pc ]; then - _cputype="$(isainfo -n)" - fi - fi - - case "$_ostype" in - - Android) - _ostype=linux-android - ;; - - Linux) - check_proc - _ostype=unknown-linux-$_clibtype - _bitness=$(get_bitness) - ;; - - FreeBSD) - _ostype=unknown-freebsd - ;; - - NetBSD) - _ostype=unknown-netbsd - ;; - - DragonFly) - _ostype=unknown-dragonfly - ;; - - Darwin) - _ostype=apple-darwin - ;; - - illumos) - _ostype=unknown-illumos - ;; - - MINGW* | MSYS* | CYGWIN* | Windows_NT) - _ostype=pc-windows-gnu - ;; - - *) - err "unrecognized OS type: $_ostype" - ;; - - esac - - case "$_cputype" in - - i386 | i486 | i686 | i786 | x86) - _cputype=i686 - ;; - - xscale | arm) - _cputype=arm - if [ "$_ostype" = "linux-android" ]; then - _ostype=linux-androideabi - fi - ;; - - armv6l) - _cputype=arm - if [ "$_ostype" = "linux-android" ]; then - _ostype=linux-androideabi - else - _ostype="${_ostype}eabihf" - fi - ;; - - armv7l | armv8l) - _cputype=armv7 - if [ "$_ostype" = "linux-android" ]; then - _ostype=linux-androideabi - else - _ostype="${_ostype}eabihf" - fi - ;; - - aarch64 | arm64) - _cputype=aarch64 - ;; - - x86_64 | x86-64 | x64 | amd64) - _cputype=x86_64 - ;; - - mips) - _cputype=$(get_endianness mips '' el) - ;; - - mips64) - if [ "$_bitness" -eq 64 ]; then - # only n64 ABI is supported for now - _ostype="${_ostype}abi64" - _cputype=$(get_endianness mips64 '' el) - fi - ;; - - ppc) - _cputype=powerpc - ;; - - ppc64) - _cputype=powerpc64 - ;; - - ppc64le) - _cputype=powerpc64le - ;; - - s390x) - _cputype=s390x - ;; - riscv64) - _cputype=riscv64gc - ;; - *) - err "unknown CPU type: $_cputype" - - esac - - # Detect 64-bit linux with 32-bit userland - if [ "${_ostype}" = unknown-linux-gnu ] && [ "${_bitness}" -eq 32 ]; then - case $_cputype in - x86_64) - if [ -n "${RUSTUP_CPUTYPE:-}" ]; then - _cputype="$RUSTUP_CPUTYPE" - else { - # 32-bit executable for amd64 = x32 - if is_host_amd64_elf; then { - echo "This host is running an x32 userland; as it stands, x32 support is poor," 1>&2 - echo "and there isn't a native toolchain -- you will have to install" 1>&2 - echo "multiarch compatibility with i686 and/or amd64, then select one" 1>&2 - echo "by re-running this script with the RUSTUP_CPUTYPE environment variable" 1>&2 - echo "set to i686 or x86_64, respectively." 1>&2 - echo 1>&2 - echo "You will be able to add an x32 target after installation by running" 1>&2 - echo " rustup target add x86_64-unknown-linux-gnux32" 1>&2 - exit 1 - }; else - _cputype=i686 - fi - }; fi - ;; - mips64) - _cputype=$(get_endianness mips '' el) - ;; - powerpc64) - _cputype=powerpc - ;; - aarch64) - _cputype=armv7 - if [ "$_ostype" = "linux-android" ]; then - _ostype=linux-androideabi - else - _ostype="${_ostype}eabihf" - fi - ;; - riscv64gc) - err "riscv64 with 32-bit userland unsupported" - ;; - esac - fi - - # Detect armv7 but without the CPU features Rust needs in that build, - # and fall back to arm. - # See https://github.com/rust-lang/rustup.rs/issues/587. - if [ "$_ostype" = "unknown-linux-gnueabihf" ] && [ "$_cputype" = armv7 ]; then - if ensure grep '^Features' /proc/cpuinfo | grep -q -v neon; then - # At least one processor does not have NEON. - _cputype=arm - fi - fi - - _arch="${_cputype}-${_ostype}" - - RETVAL="$_arch" -} - -check_proc() { - # Check for /proc by looking for the /proc/self/exe link - # This is only run on Linux - if ! test -L /proc/self/exe ; then - err "fatal: Unable to find /proc/self/exe. Is /proc mounted? Installation cannot proceed without /proc." - fi -} - -get_bitness() { - need_cmd head - # Architecture detection without dependencies beyond coreutils. - # ELF files start out "\x7fELF", and the following byte is - # 0x01 for 32-bit and - # 0x02 for 64-bit. - # The printf builtin on some shells like dash only supports octal - # escape sequences, so we use those. - local _current_exe_head - _current_exe_head=$(head -c 5 /proc/self/exe ) - if [ "$_current_exe_head" = "$(printf '\177ELF\001')" ]; then - echo 32 - elif [ "$_current_exe_head" = "$(printf '\177ELF\002')" ]; then - echo 64 - else - err "unknown platform bitness" - fi -} - -is_host_amd64_elf() { - need_cmd head - need_cmd tail - # ELF e_machine detection without dependencies beyond coreutils. - # Two-byte field at offset 0x12 indicates the CPU, - # but we're interested in it being 0x3E to indicate amd64, or not that. - local _current_exe_machine - _current_exe_machine=$(head -c 19 /proc/self/exe | tail -c 1) - [ "$_current_exe_machine" = "$(printf '\076')" ] -} - -get_endianness() { - local cputype=$1 - local suffix_eb=$2 - local suffix_el=$3 - - # detect endianness without od/hexdump, like get_bitness() does. - need_cmd head - need_cmd tail - - local _current_exe_endianness - _current_exe_endianness="$(head -c 6 /proc/self/exe | tail -c 1)" - if [ "$_current_exe_endianness" = "$(printf '\001')" ]; then - echo "${cputype}${suffix_el}" - elif [ "$_current_exe_endianness" = "$(printf '\002')" ]; then - echo "${cputype}${suffix_eb}" - else - err "unknown platform endianness" - fi -} - -say() { - printf '%s\n' "$1" -} - -err() { - say "$1" >&2 - exit 1 -} - -need_cmd() { - if ! check_cmd "$1"; then - err "need '$1' (command not found)" - fi -} - -check_cmd() { - command -v "$1" > /dev/null 2>&1 -} - -assert_nz() { - if [ -z "$1" ]; then err "assert_nz $2"; fi -} - -# Run a command that should never fail. If the command fails execution -# will immediately terminate with an error showing the failing -# command. -ensure() { - if ! "$@"; then err "command failed: $*"; fi -} - -# This wraps curl or wget. Try curl first, if not installed, -# use wget instead. -downloader() { - local _dld - local _ciphersuites - local _err - local _status - local _retry - if check_cmd curl; then - _dld=curl - elif check_cmd wget; then - _dld=wget - else - _dld='curl or wget' # to be used in error message of need_cmd - fi - - if [ "$1" = --check ]; then - need_cmd "$_dld" - elif [ "$_dld" = curl ]; then - check_curl_for_retry_support - _retry="$RETVAL" - get_ciphersuites_for_curl - _ciphersuites="$RETVAL" - if [ -n "$_ciphersuites" ]; then - _err=$(curl $_retry --proto '=https' --tlsv1.2 --ciphers "$_ciphersuites" --silent --show-error --fail --location "$1" --output "$2" 2>&1) - _status=$? - else - echo "Warning: Not enforcing strong cipher suites for TLS, this is potentially less secure" - if ! check_help_for "$3" curl --proto --tlsv1.2; then - echo "Warning: Not enforcing TLS v1.2, this is potentially less secure" - _err=$(curl $_retry --silent --show-error --fail --location "$1" --output "$2" 2>&1) - _status=$? - else - _err=$(curl $_retry --proto '=https' --tlsv1.2 --silent --show-error --fail --location "$1" --output "$2" 2>&1) - _status=$? - fi - fi - if [ -n "$_err" ]; then - if echo "$_err" | grep -q 404; then - err "❌ Binary for platform '$3' not found, this platform may be unsupported." - else - echo "$_err" >&2 - fi - fi - return $_status - elif [ "$_dld" = wget ]; then - if [ "$(wget -V 2>&1|head -2|tail -1|cut -f1 -d" ")" = "BusyBox" ]; then - echo "Warning: using the BusyBox version of wget. Not enforcing strong cipher suites for TLS or TLS v1.2, this is potentially less secure" - _err=$(wget "$1" -O "$2" 2>&1) - _status=$? - else - get_ciphersuites_for_wget - _ciphersuites="$RETVAL" - if [ -n "$_ciphersuites" ]; then - _err=$(wget --https-only --secure-protocol=TLSv1_2 --ciphers "$_ciphersuites" "$1" -O "$2" 2>&1) - _status=$? - else - echo "Warning: Not enforcing strong cipher suites for TLS, this is potentially less secure" - if ! check_help_for "$3" wget --https-only --secure-protocol; then - echo "Warning: Not enforcing TLS v1.2, this is potentially less secure" - _err=$(wget "$1" -O "$2" 2>&1) - _status=$? - else - _err=$(wget --https-only --secure-protocol=TLSv1_2 "$1" -O "$2" 2>&1) - _status=$? - fi - fi - fi - if [ -n "$_err" ]; then - if echo "$_err" | grep -q ' 404 Not Found'; then - err "❌ Binary for platform '$3' not found, this platform may be unsupported." - else - echo "$_err" >&2 - fi - fi - return $_status - else - err "Unknown downloader" # should not reach here - fi -} - -# Check if curl supports the --retry flag, then pass it to the curl invocation. -check_curl_for_retry_support() { - local _retry_supported="" - # "unspecified" is for arch, allows for possibility old OS using macports, homebrew, etc. - if check_help_for "notspecified" "curl" "--retry"; then - _retry_supported="--retry 3" - fi - - RETVAL="$_retry_supported" - -} - -check_help_for() { - local _arch - local _cmd - local _arg - _arch="$1" - shift - _cmd="$1" - shift - - local _category - if "$_cmd" --help | grep -q 'For all options use the manual or "--help all".'; then - _category="all" - else - _category="" - fi - - case "$_arch" in - - *darwin*) - if check_cmd sw_vers; then - case $(sw_vers -productVersion) in - 10.*) - # If we're running on macOS, older than 10.13, then we always - # fail to find these options to force fallback - if [ "$(sw_vers -productVersion | cut -d. -f2)" -lt 13 ]; then - # Older than 10.13 - echo "Warning: Detected macOS platform older than 10.13" - return 1 - fi - ;; - 11.*) - # We assume Big Sur will be OK for now - ;; - *) - # Unknown product version, warn and continue - echo "Warning: Detected unknown macOS major version: $(sw_vers -productVersion)" - echo "Warning TLS capabilities detection may fail" - ;; - esac - fi - ;; - - esac - - for _arg in "$@"; do - if ! "$_cmd" --help $_category | grep -q -- "$_arg"; then - return 1 - fi - done - - true # not strictly needed -} - -# Return cipher suite string specified by user, otherwise return strong TLS 1.2-1.3 cipher suites -# if support by local tools is detected. Detection currently supports these curl backends: -# GnuTLS and OpenSSL (possibly also LibreSSL and BoringSSL). Return value can be empty. -get_ciphersuites_for_curl() { - if [ -n "${RUSTUP_TLS_CIPHERSUITES-}" ]; then - # user specified custom cipher suites, assume they know what they're doing - RETVAL="$RUSTUP_TLS_CIPHERSUITES" - return - fi - - local _openssl_syntax="no" - local _gnutls_syntax="no" - local _backend_supported="yes" - if curl -V | grep -q ' OpenSSL/'; then - _openssl_syntax="yes" - elif curl -V | grep -iq ' LibreSSL/'; then - _openssl_syntax="yes" - elif curl -V | grep -iq ' BoringSSL/'; then - _openssl_syntax="yes" - elif curl -V | grep -iq ' GnuTLS/'; then - _gnutls_syntax="yes" - else - _backend_supported="no" - fi - - local _args_supported="no" - if [ "$_backend_supported" = "yes" ]; then - # "unspecified" is for arch, allows for possibility old OS using macports, homebrew, etc. - if check_help_for "notspecified" "curl" "--tlsv1.2" "--ciphers" "--proto"; then - _args_supported="yes" - fi - fi - - local _cs="" - if [ "$_args_supported" = "yes" ]; then - if [ "$_openssl_syntax" = "yes" ]; then - _cs=$(get_strong_ciphersuites_for "openssl") - elif [ "$_gnutls_syntax" = "yes" ]; then - _cs=$(get_strong_ciphersuites_for "gnutls") - fi - fi - - RETVAL="$_cs" -} - -# Return cipher suite string specified by user, otherwise return strong TLS 1.2-1.3 cipher suites -# if support by local tools is detected. Detection currently supports these wget backends: -# GnuTLS and OpenSSL (possibly also LibreSSL and BoringSSL). Return value can be empty. -get_ciphersuites_for_wget() { - if [ -n "${RUSTUP_TLS_CIPHERSUITES-}" ]; then - # user specified custom cipher suites, assume they know what they're doing - RETVAL="$RUSTUP_TLS_CIPHERSUITES" - return - fi - - local _cs="" - if wget -V | grep -q '\-DHAVE_LIBSSL'; then - # "unspecified" is for arch, allows for possibility old OS using macports, homebrew, etc. - if check_help_for "notspecified" "wget" "TLSv1_2" "--ciphers" "--https-only" "--secure-protocol"; then - _cs=$(get_strong_ciphersuites_for "openssl") - fi - elif wget -V | grep -q '\-DHAVE_LIBGNUTLS'; then - # "unspecified" is for arch, allows for possibility old OS using macports, homebrew, etc. - if check_help_for "notspecified" "wget" "TLSv1_2" "--ciphers" "--https-only" "--secure-protocol"; then - _cs=$(get_strong_ciphersuites_for "gnutls") - fi - fi - - RETVAL="$_cs" -} - -# Return strong TLS 1.2-1.3 cipher suites in OpenSSL or GnuTLS syntax. TLS 1.2 -# excludes non-ECDHE and non-AEAD cipher suites. DHE is excluded due to bad -# DH params often found on servers (see RFC 7919). Sequence matches or is -# similar to Firefox 68 ESR with weak cipher suites disabled via about:config. -# $1 must be openssl or gnutls. -get_strong_ciphersuites_for() { - if [ "$1" = "openssl" ]; then - # OpenSSL is forgiving of unknown values, no problems with TLS 1.3 values on versions that don't support it yet. - echo "TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384" - elif [ "$1" = "gnutls" ]; then - # GnuTLS isn't forgiving of unknown values, so this may require a GnuTLS version that supports TLS 1.3 even if wget doesn't. - # Begin with SECURE128 (and higher) then remove/add to build cipher suites. Produces same 9 cipher suites as OpenSSL but in slightly different order. - echo "SECURE128:-VERS-SSL3.0:-VERS-TLS1.0:-VERS-TLS1.1:-VERS-DTLS-ALL:-CIPHER-ALL:-MAC-ALL:-KX-ALL:+AEAD:+ECDHE-ECDSA:+ECDHE-RSA:+AES-128-GCM:+CHACHA20-POLY1305:+AES-256-GCM" - fi -} - -# This is just for indicating that commands' results are being -# intentionally ignored. Usually, because it's being executed -# as part of error handling. -ignore() { - "$@" -} - -main "$@" || exit 1 diff --git a/resources/config.zip b/resources/config.zip deleted file mode 100644 index c134fc48..00000000 Binary files a/resources/config.zip and /dev/null differ diff --git a/tests/Cargo.toml b/tests/Cargo.toml index 4fe524cf..127a5f57 100644 --- a/tests/Cargo.toml +++ b/tests/Cargo.toml @@ -5,9 +5,7 @@ edition = "2021" resolver = "2" [features] -default = ["sqlite", "foundationdb", "postgres", "mysql", "rocks", "elastic", "s3", "redis"] -#default = ["sqlite", "postgres", "mysql", "rocks", "elastic", "s3", "redis"] -#default = ["foundationdb", "postgres"] +default = ["sqlite", "postgres", "mysql", "rocks", "elastic", "s3", "redis"] sqlite = ["store/sqlite"] foundationdb = ["store/foundation"] postgres = ["store/postgres"]