- Sent count is no longer the batch size fetched from the DB but is
the actual count of messages sent.
- Pausing and resuming now accurately tracks the last subscriber that
was processed and resumes from there.
- Fix multiple concurrent campaigns blocking.
Closes#1616. Closes#905. Closes#1496. Closes#1250. Closes#1010.
This commit fully refactors the core campaign manager logic.
It applies a whole new approach to campaign state and lifecycle management.
- Create a new "pipeline" abstraction on top of campaign
for state management.
- Account for every message processed and end campaigns
based on the actual count.
- Discard in-queue messages in the pipeline of a paused
or cancelled campaign.
- `GET /api/events?type=error` opens a long-lived HTTP server side
event connection that streams error messages.
- async (typically SMTP) errors are now streamed to the frontend and
disaplyed as an error toast on the admin UI.
- Adds support for arbitrary file uploads with an admin setting to select allowed file extensions.
- Adds support for attaching media (files) to campaigns.
Remove `messenger.go` and move the interface definition to `manager`
and the `Message` struct to the `models` package, removing superflous
and redundant message structs used in multiple places.
The original PR accepts files to the `/tx` endpoints as Base64 encoded
strings in the JSON payload. This isn't ideal as the payload size
increase caused by Base64 for larger files can be significant,
in addition to the added clientside API complexity.
This PR adds supports for multipart form posts to `/tx` where the
JSON data (name: `data`) and multiple files can be posted simultaenously
(one or more `file` fields).
--- PR: #1166
* Attachment model for TxMessage
* Don't reassign values, just pass the manager.Messgage
* Read attachment info from API; create attachment Header
* Refactor tx attachments to use multipart form files. Closes#1166.
---
Co-authored-by: MatiSSL <matiss.lidaka@nic.lv>
This commit adds a new API `POST /api/tx` that sends an ad-hoc message
to a subscriber based on a pre-defined transactional template. This is
a large commit that adds the following:
- New campaign / tx template types on the UI. tx templates have an
additional subject field.
- New fields `type` and `subject` to the templates table.
- Refactor template CRUD operations and models.
- Refactor template func assignment in manager.
- Add pre-compiled template caching to manager runtime.
- Pre-compile all tx templates into memory on program boot to avoid
expensive template compilation on ad-hoc tx messages.
The `rate` field `/api/campaigns/running/stats` returned was computed
based on the total time spent from the start of the campaign to the
current time. This meant that for large campaigns, if there were
pauses or slowdowns in between, the rate would be skewed heavily
making it useless to figure out the current send rate.
This commit introduces a realtime running rate counter in the campaign
manager that returns accurate (running) send rates for the last minute.
The `rate` field in the API now shows the live running rate and a
new `net_rate` field shows the rate from the beginning of the campaign.
- Add new `headers[]` column to the campain table.
- Add new headers box to the campaign UI that takes a JSON array of
custom headers like the headers on the SMTP settings UI.
- Headers are added to e-mails and messenger postback webhooks.
- Add cypress tests.
Closes#514.
Every listmonk instance scans the DB periodically to look for
running campaigns to process. This made running multiple instances of
listmonk impractical as they would all pick up the same running
campaign and process them, resulting in duplicate e-mails.
This commit adds a `--passive` flag to the binary that runs listmonk
in a "passive" mode where campaign processing is disabled. This allows
multiple instances of listmonk to be run to handle different kinds of
requests if there is a requirement (scale/traffic?). It is important
to note that there should only be one non-passive instance running at
any given time. If distributed campaign processing is ever considered,
this will change.
- Blocklist or unsubscribe subscribers based on a bounce threshold
- Add /bounces UI for viewing bounces and in the subscriber view
- Add settings UI for managing bounce settings
- Add support for scanning POP3 bounce mailboxes
- Add a generic webhook for posting custom bounces at /webhooks/bounce
- Add SES bounce webhook support at /webhooks/services/ses
- Add Sendgrid bounce webhook support at /webhooks/services/sendgrid
Campaign messages are handled by `manager` whereas test messages
were being pushed directly into a messenger skipping some campaign
related routines such as the addition of list unsub headers.
This commit exposes a new function `manager.PushCampaignMessage()`
that accepts arbitrary campaign messages that then pass through
the standard campaign message workers, thus getting the missing unsub
headers. This closes#360.
In addition, this removes the superfluous `CampaignMessage.Render()`
function which had to be mandatorily called always and makes it
implicit in `manager.NewCampaignMessage()`.
This commit removes the Go html2text lib that would automatically
convert all HTML messages to plaintext and add them as the alt
text body to outgoing e-mails. This lib also had memory leak
issues with certain kinds of HTML templates.
A new UI field for optionally adding an alt plaintext body to
a campaign is added. On enabling, it converts the HTML message in
the campaign editor into plaintext (using the textversionjs lib).
This introduces breaking changes in the campaigns table schema,
model, and template compilation.
Certain SMTP hosts limit the total number of messages that can be
sent within a window, for instance, X / 24 hours. The concurrency
and message rate controls can only limit that to a max of
1 messages / second, without a global cap.
This commit introduces a simple sliding window rate limit feature
that counts the number of messages sent in a specific window, and
upon reaching that limit, waits for the window to reset before
any more messages are pushed out globally across any number of
campaigns.
Context: https://github.com/knadh/listmonk/issues/119
A new toggle switch in Settings -> Privacy, which is off by
default, allows campaign views (pixel) and link clicks to function
without registering the subscriber ID against view and click
events, anonymising tracking. When off, the subscriber UUIDs in
view and link tracking URLs are removed, anonymising subscriber
information from HTTP logs as well.
This is a major feature that builds upon the `Messenger` interface
that has been in listmonk since its inception (with SMTP as the only
messenger). This commit introduces a new Messenger implementation, an
HTTP "postback", that can post campaign messages as a standard JSON
payload to arbitrary HTTP servers. These servers can in turn push them
to FCM, SMS, or any or any such upstream, enabling listmonk to be a
generic campaign messenger for any type of communication, not just
e-mails.
Postback HTTP endpoints can be defined in settings and they can be
selected on campaigns.
- Added as a setting in the settings UI.
- Refactor Messenger.Push() method to accept messenger.Message{}
instead of a growing number of positional arguments.
This is a major breaking change that moves away from having the
entire app configuration in external TOML files to settings being
in the database with a UI to update them dynamically.
The app loads all config into memory (app settings, SMTP conf)
on boot. "Hot" replacing them is complex and it's a fair tradeoff
to instead just restart the application as it is practically
instant.
A new `settings` table stores arbitrary string keys with a JSONB
value field which happens to support arbitrary types. After every
settings update, the app gracefully releases all resources
(HTTP server, DB pool, SMTP pool etc.) and restarts itself,
occupying the same PID. If there are any running campaigns, the
auto-restart doesn't happen and the user is prompted to invoke
it manually with a one-click button once all running campaigns
have been paused.
- Refactor campaign.Message into campaign.Message and
campaign.CampaignMessage
- Remove ad-hoc goroutines (flawed approach) that were used to push
admin and optin notifications.
- Provision for largscale pushing of ad-hoc, non-campaign messages
such as transactional messages (in the future).
