This commit adds a new API `POST /api/tx` that sends an ad-hoc message
to a subscriber based on a pre-defined transactional template. This is
a large commit that adds the following:
- New campaign / tx template types on the UI. tx templates have an
additional subject field.
- New fields `type` and `subject` to the templates table.
- Refactor template CRUD operations and models.
- Refactor template func assignment in manager.
- Add pre-compiled template caching to manager runtime.
- Pre-compile all tx templates into memory on program boot to avoid
expensive template compilation on ad-hoc tx messages.
This is a long pending refactor. All the DB, query, CRUD, and related
logic scattered across HTTP handlers are now moved into a central
`core` package with clean, abstracted methods, decoupling HTTP
handlers from executing direct DB queries and other business logic.
eg: `core.CreateList()`, `core.GetLists()` etc.
- Remove obsolete subscriber methods.
- Move optin hook queries to core.
- Move campaign methods to `core`.
- Move all campaign methods to `core`.
- Move public page functions to `core`.
- Move all template functions to `core`.
- Move media and settings function to `core`.
- Move handler middleware functions to `core`.
- Move all bounce functions to `core`.
- Move all dashboard functions to `core`.
- Fix GetLists() not honouring type
- Fix unwrapped JSON responses.
- Clean up obsolete pre-core util function.
- Replace SQL array null check with cardinality check.
- Fix missing validations in `core` queries.
- Remove superfluous deps on internal `subimporter`.
- Add dashboard functions to `core`.
- Fix broken domain ban check.
- Fix broken subscriber check middleware.
- Remove redundant error handling.
- Remove obsolete functions.
- Remove obsolete structs.
- Remove obsolete queries and DB functions.
- Document the `core` package.
The analytics page showed non-unique counts for views and clicks which
was misleading and source of confusion: #522, #561, #571, #676, #680
This commit changes this behaviour to pull unique views and clicks when
individual subscriber tracking is turned on in settings, and non-unique
counts when it is turned off (as `subscriber_id` in `campaign_views`
and `link_clicks` will be NULL, rendering unique queries dysfunctional).
This commit changes the stats SQL queries to use string interpolation
to either to SELECT `*` or `DISTINCT subscriber_id` on app boot based
on the setting in the DB. This involves significant changes to how
queries are read and prepared on init.
- Refactor `initQueries()` to `readQueries()` and `prepareQueries()`.
- Read queries first before preparing.
- Load settings from the DB using the read settings query.
- Prepare queries next. Use the privacy setting from the DB to apply
string interpolation to the analytics queries to pull
unique/non-unique before preparing the queries.
On the UI:
- Show a note on the analytics page about unique/non-unique counts.
- Hide the % donut charts on the analytics page in non-unique mode.
Closes#676, closes#680
This feature was originally authored by @sweetppro in PR #438.
However, since the PR ended up in an unclean state with
multiple master merges (instead of rebase) from the upstream, there are
several commits that are out of order and can can no longer be be
squashed for a clean feature merge.
This commit aggregates the changes from the original PR and applies the
following fixes on top of it.
- Add custom admin JS box to appearance UI.
- Refactor i18n language strings.
- Add handlers and migrations for the new `appearance.admin.custom_js`
field.
- Fix migration version to `v2.1.0`
- Load custom appearance CSS/JS bytes into global constants during boot
instead of making a DB call on every request.
- Fix and canonicalize URIs from `/api/custom*` to `/public/*.css`
and `/admin/*.css`. Add proxy paths to yarn proxy config.
- Remove redundant HTTP handlers for different custom appearance files
and refactor into a single handler `serveCustomApperance()`
- Fix content-type and UTF8 encoding headers for different file types.
- Fix incorrect registration of public facing custom CSS/JS handlers
in the authenticated admin URI group.
- Fix merge conflicts in `Settings.vue`.
- Minor HTML and style fixes.
- Remove the `AppearanceEditor` component and use the existing
`HTMLEditor` component instead.
- Add `language` prop to the `HTMLEditor` component.
Co-authored-by: SweetPPro <sweetppro@users.noreply.github.com>
- echo is now on v4 with major changes including a few breaking changes
- bind() behaviour is now strict. JSON / form etc. unmarshalling of
request data need appropriate `json`, `form` tags. Missing tags for
the public subscription page is added in this commit.
- This also closes#602.
Every listmonk instance scans the DB periodically to look for
running campaigns to process. This made running multiple instances of
listmonk impractical as they would all pick up the same running
campaign and process them, resulting in duplicate e-mails.
This commit adds a `--passive` flag to the binary that runs listmonk
in a "passive" mode where campaign processing is disabled. This allows
multiple instances of listmonk to be run to handle different kinds of
requests if there is a requirement (scale/traffic?). It is important
to note that there should only be one non-passive instance running at
any given time. If distributed campaign processing is ever considered,
this will change.
If `<!doctype html>` is not found in static/email-templates/base.html,
all system e-mail templates are assumed to be plaintext and go out
as content-type: plaintext e-mails. With this, all HTML tags can
be stripped out of the system e-mail templates (while maintaining
Go template tags and logic) to have plaintext system e-mail templates.
Closes#546
E-mails in the domain blocklist are disallowed on the admin UI, public
subscription forms, API, and in the bulk importer.
- Add blocklist setting that takes a list of multi-line domains on the
Settings -> Privacy UI.
- Refactor e-mail validation in subimporter to add blocklist checking
centrally.
- Add Cypress testr testing domain blocklist behaviour on admin
and non-admin views.
Closes#336.
- Namespace all admin UI URLs behind `/admin/*`.
This breaks the current admin UI URLs.
- Make Vue output build assets to `frontend/dist/*` instead of
`frontend/dist/frontend`.
- Namespace Vue static assets to `/admin/static/*`.
This commit reduces the cofusing and convoluted Vue+WebPack build URI
and static path schemes. In addition, it removes ambiguity in URLs
where non-UI URLs like `/public`, `/api`, `/webhooks` etc. were in the
same name space as UI URLs like `/campaigns`, `/lists` etc. Now all UI
URLs are behind `/admin/`, also simplifying security rules for proxies.
- Introduce a new S3 backend URL on the settings UI
- Add DB migration to populate S3 URL for existing S3 settings
- Refactor and fix URL formatting
Closes#139
- Blocklist or unsubscribe subscribers based on a bounce threshold
- Add /bounces UI for viewing bounces and in the subscriber view
- Add settings UI for managing bounce settings
- Add support for scanning POP3 bounce mailboxes
- Add a generic webhook for posting custom bounces at /webhooks/bounce
- Add SES bounce webhook support at /webhooks/services/ses
- Add Sendgrid bounce webhook support at /webhooks/services/sendgrid
Ref: https://github.com/knadh/listmonk/issues/409
- Introduce `main.appDir` and `main.fronendDir` Go compile-time flags
to hardcode custom paths for loading frontend assets
(frontend/dist/frontend in the repo after build) and app assets
(queries.sql, schema.sql, config.toml.sample) in environments where
embedding files in the binary is not feasible.
These default to CWD unless explicitly set during compilation.
- Fix the Vue favicon path oddity by copying the icon into the built
frontend dir in the `make-frontend` step.
When no static assets are found on init, i.e., when a binary without
stuffbin assets are loaded, the app looks for all necessary static
files in the working dir, including the `./static/*` path which renders
the `--static-dir` flag irrelevant.
This patch gives `--static-dir`, if set, precedence over `./static/*`
when loading assets from the working dir when a binary is not stuffed
with static files.
Closes#340.
The new `--i18n-dir` directory allows the loading of an external
directory of i18n JSON files, milar to have `--static-dir`
works. New languages can be added and existing language files
can be customized this way.
This commit changes file loading behaviour so that invalid or
non-existent don't halt the execution of the app completely but
merely throw a warning and continue with the default (en) lang.
The earlier approach of loading `/api/config.js` as a script on
initial page load with the necessary variables to init the UI is
ditched. Instead, it's now `/api/config` and `/api/settings` like
all other API calls. On load of the frontend, these two resources
are fetched and the frontend is initialised.
In addition to generating HTML forms for selected public lists,
the form page now shows a URL (/subscription/form) that can be
publicly shared to solicit subscriptions. The page lists all
public lists in the database. This page can be disabled on the
Settings UI.
The default language (en) is loaded first and the selected
language is loaded on top of it so that missing translation keys
in the selected language will have the original English strings
available on the UI.
Certain SMTP hosts limit the total number of messages that can be
sent within a window, for instance, X / 24 hours. The concurrency
and message rate controls can only limit that to a max of
1 messages / second, without a global cap.
This commit introduces a simple sliding window rate limit feature
that counts the number of messages sent in a specific window, and
upon reaching that limit, waits for the window to reset before
any more messages are pushed out globally across any number of
campaigns.
Context: https://github.com/knadh/listmonk/issues/119
Lists, campaigns, and subscribers tables now support server-side
sorting from the UI. This significantly changes the internal
queries from prepared to string interpolated to support dynamic
sort params.
A new toggle switch in Settings -> Privacy, which is off by
default, allows campaign views (pixel) and link clicks to function
without registering the subscriber ID against view and click
events, anonymising tracking. When off, the subscriber UUIDs in
view and link tracking URLs are removed, anonymising subscriber
information from HTTP logs as well.
This is a major feature that builds upon the `Messenger` interface
that has been in listmonk since its inception (with SMTP as the only
messenger). This commit introduces a new Messenger implementation, an
HTTP "postback", that can post campaign messages as a standard JSON
payload to arbitrary HTTP servers. These servers can in turn push them
to FCM, SMS, or any or any such upstream, enabling listmonk to be a
generic campaign messenger for any type of communication, not just
e-mails.
Postback HTTP endpoints can be defined in settings and they can be
selected on campaigns.
