- Add materialized views for list -> subscriber counts, dashboard chart,
and dashboard aggregate stats that slow down significantly on large
databases (with millions or tens of millions of subscribers). These
slow queries involve full table scan COUNTS().
- Add a toggle to enable caching slow results in Settings -> Performance.
- Add support for setting a cron string that crons and periodically
refreshes aggregated stats in materialized views.
Closes#1019.
- Add new 'Subscriptions' table on the subscriber list form that shows subs,
IP, and other data.
- Add new `meta` JSONB field to `subscriber_lsts` table.
Closes#1329.
- Change tiled UI to table UI.
- Add support for search and pagination.
- Important: This breaks the `GET /api/media` API to introduce pagination
fields. Media items are now moved into `{ data: results[] }`.
- Adds support for arbitrary file uploads with an admin setting to select allowed file extensions.
- Adds support for attaching media (files) to campaigns.
Remove `messenger.go` and move the interface definition to `manager`
and the `Message` struct to the `models` package, removing superflous
and redundant message structs used in multiple places.
- Add support for `complaint` to the SES bounce processor.
- Add support for `hard/soft` to Sendgrid bounce processor.
- Add new bounce actions `None` and `Unsubscribe`.
- Add per type (`soft/hard/complaint`) bounce rule configuration to
admin settings UI.
- Refactor Cypress bounce tests.
- Introduces a new option on the settings UI to optionally publish the full campaign body in
public archive RSS feeds.
Closes#1033
Co-authored-by: Kailash Nadh <kailash@nadh.in>
The original PR accepts files to the `/tx` endpoints as Base64 encoded
strings in the JSON payload. This isn't ideal as the payload size
increase caused by Base64 for larger files can be significant,
in addition to the added clientside API complexity.
This PR adds supports for multipart form posts to `/tx` where the
JSON data (name: `data`) and multiple files can be posted simultaenously
(one or more `file` fields).
--- PR: #1166
* Attachment model for TxMessage
* Don't reassign values, just pass the manager.Messgage
* Read attachment info from API; create attachment Header
* Refactor tx attachments to use multipart form files. Closes#1166.
---
Co-authored-by: MatiSSL <matiss.lidaka@nic.lv>
Bots easily bypass the simple `nonce` hack. This commit adds support
for the hcaptcha.com widget.
- New `Security` tab in the admin settings UI.
- Enable/disable CAPTCHA.
- Render CAPTCHA on the public subscription form.
Closes#1116.
This patch adds new array fields on `POST /tx`: `subscriber_emails[]`, `subscriber_ids[]`.
Either of these array fields can be sent with multiple subscribers.
The individual non-array fields `subscriber_id` and `subscriber_email` are deprecated.
Closes#994.
This commit adds a new API `POST /api/tx` that sends an ad-hoc message
to a subscriber based on a pre-defined transactional template. This is
a large commit that adds the following:
- New campaign / tx template types on the UI. tx templates have an
additional subject field.
- New fields `type` and `subject` to the templates table.
- Refactor template CRUD operations and models.
- Refactor template func assignment in manager.
- Add pre-compiled template caching to manager runtime.
- Pre-compile all tx templates into memory on program boot to avoid
expensive template compilation on ad-hoc tx messages.
This is a long pending refactor. All the DB, query, CRUD, and related
logic scattered across HTTP handlers are now moved into a central
`core` package with clean, abstracted methods, decoupling HTTP
handlers from executing direct DB queries and other business logic.
eg: `core.CreateList()`, `core.GetLists()` etc.
- Remove obsolete subscriber methods.
- Move optin hook queries to core.
- Move campaign methods to `core`.
- Move all campaign methods to `core`.
- Move public page functions to `core`.
- Move all template functions to `core`.
- Move media and settings function to `core`.
- Move handler middleware functions to `core`.
- Move all bounce functions to `core`.
- Move all dashboard functions to `core`.
- Fix GetLists() not honouring type
- Fix unwrapped JSON responses.
- Clean up obsolete pre-core util function.
- Replace SQL array null check with cardinality check.
- Fix missing validations in `core` queries.
- Remove superfluous deps on internal `subimporter`.
- Add dashboard functions to `core`.
- Fix broken domain ban check.
- Fix broken subscriber check middleware.
- Remove redundant error handling.
- Remove obsolete functions.
- Remove obsolete structs.
- Remove obsolete queries and DB functions.
- Document the `core` package.
This was originally authored by @stevesavanna in #707. This commit
contains changes and refactors that could not be pushed to the original PR.
Changes from #707
- Don't ignore bounce mails missing campaign / subscriber UUIDs. The
original behaviour falls back to looking up subscribers by e-mail.
- Refactor repetetive header.get + regexp conditions per header into
a simpler lookup map.
- Trim e-mail header values of `\r`.
Closes#707, #763
Co-authored-by: stevesavanna <steven@savannacorp.com>
- Change `query-lists` query to aggregate the subscriber count by
status (confirmed, unsubscribed etc.) and expose them under a new
`subscriber_statuses: {}` field in the `GET /lists` API.
- Display the statuses and counts in the lists table on the UI.
Closes#616
- Add new `headers[]` column to the campain table.
- Add new headers box to the campaign UI that takes a JSON array of
custom headers like the headers on the SMTP settings UI.
- Headers are added to e-mails and messenger postback webhooks.
- Add cypress tests.
Closes#514.
- echo is now on v4 with major changes including a few breaking changes
- bind() behaviour is now strict. JSON / form etc. unmarshalling of
request data need appropriate `json`, `form` tags. Missing tags for
the public subscription page is added in this commit.
- This also closes#602.
The default `{{ TrackLink "https://listmonk.app" }}` template function
is clumsy to write and does breaks WYSIWYG editors and HTML syntax
highlighting because of the quotes. The new syntax doesn't break HTML
and is easier to write.
Eg: `<a href="https://listmonk.app@TrackLink">Link</a>`
- Introduce @TrackLink shorthand.
- Add first-class support for tracking links in the WYSIWYG (TinyMCE)
editor by introducing an on/off checkbox on the link dialog.
- Improve default dummy campaign content to highlight this.
- Add indexes.
- Refactor dashboard charts and view/click count queries.
(~10x speed bump on a setup of 7mn subscribers and 80mn views)
- Refactor get subscriber queries.
(~10x speed bump on 7mn subscribers)
- Make subscriber UI issue an equality query for email seach strings.
- Blocklist or unsubscribe subscribers based on a bounce threshold
- Add /bounces UI for viewing bounces and in the subscriber view
- Add settings UI for managing bounce settings
- Add support for scanning POP3 bounce mailboxes
- Add a generic webhook for posting custom bounces at /webhooks/bounce
- Add SES bounce webhook support at /webhooks/services/ses
- Add Sendgrid bounce webhook support at /webhooks/services/sendgrid
Previously, converting between formats simply copied over raw content.
This update does actual conversion between different formats. While
lossy, this seems to a good enough approximation for even reasonbly
rich HTML content. Closes#348.
- richtext, html => plain
Strips HTML and converts content to plain text.
- richtext, html => markdown
Uses turndown (JS) lib to convert HTML to Markdown.
- plain => richtext, html
Converts line breaks in plain text to HTML breaks.
- richtext => html
"Beautifies" the HTML generated by the WYSIWYG editor unlike the
earlier behaviour of dumping one long line of HTML.
- markdown => richtext, html
Makes an API call to the backend to use the Goldmark lib to convert
Markdown to HTML.
This commit removes the Go html2text lib that would automatically
convert all HTML messages to plaintext and add them as the alt
text body to outgoing e-mails. This lib also had memory leak
issues with certain kinds of HTML templates.
A new UI field for optionally adding an alt plaintext body to
a campaign is added. On enabling, it converts the HTML message in
the campaign editor into plaintext (using the textversionjs lib).
This introduces breaking changes in the campaigns table schema,
model, and template compilation.
This is a major feature that builds upon the `Messenger` interface
that has been in listmonk since its inception (with SMTP as the only
messenger). This commit introduces a new Messenger implementation, an
HTTP "postback", that can post campaign messages as a standard JSON
payload to arbitrary HTTP servers. These servers can in turn push them
to FCM, SMS, or any or any such upstream, enabling listmonk to be a
generic campaign messenger for any type of communication, not just
e-mails.
Postback HTTP endpoints can be defined in settings and they can be
selected on campaigns.
