- Add materialized views for list -> subscriber counts, dashboard chart,
and dashboard aggregate stats that slow down significantly on large
databases (with millions or tens of millions of subscribers). These
slow queries involve full table scan COUNTS().
- Add a toggle to enable caching slow results in Settings -> Performance.
- Add support for setting a cron string that crons and periodically
refreshes aggregated stats in materialized views.
Closes#1019.
- Sent count is no longer the batch size fetched from the DB but is
the actual count of messages sent.
- Pausing and resuming now accurately tracks the last subscriber that
was processed and resumes from there.
- Fix multiple concurrent campaigns blocking.
Closes#1616. Closes#905. Closes#1496. Closes#1250. Closes#1010.
- Add new 'Subscriptions' table on the subscriber list form that shows subs,
IP, and other data.
- Add new `meta` JSONB field to `subscriber_lsts` table.
Closes#1329.
- Change tiled UI to table UI.
- Add support for search and pagination.
- Important: This breaks the `GET /api/media` API to introduce pagination
fields. Media items are now moved into `{ data: results[] }`.
- Adds support for arbitrary file uploads with an admin setting to select allowed file extensions.
- Adds support for attaching media (files) to campaigns.
- Add support for `complaint` to the SES bounce processor.
- Add support for `hard/soft` to Sendgrid bounce processor.
- Add new bounce actions `None` and `Unsubscribe`.
- Add per type (`soft/hard/complaint`) bounce rule configuration to
admin settings UI.
- Refactor Cypress bounce tests.
- Introduces a new option on the settings UI to optionally publish the full campaign body in
public archive RSS feeds.
Closes#1033
Co-authored-by: Kailash Nadh <kailash@nadh.in>
This commit adds a new API `POST /api/tx` that sends an ad-hoc message
to a subscriber based on a pre-defined transactional template. This is
a large commit that adds the following:
- New campaign / tx template types on the UI. tx templates have an
additional subject field.
- New fields `type` and `subject` to the templates table.
- Refactor template CRUD operations and models.
- Refactor template func assignment in manager.
- Add pre-compiled template caching to manager runtime.
- Pre-compile all tx templates into memory on program boot to avoid
expensive template compilation on ad-hoc tx messages.
This is a long pending refactor. All the DB, query, CRUD, and related
logic scattered across HTTP handlers are now moved into a central
`core` package with clean, abstracted methods, decoupling HTTP
handlers from executing direct DB queries and other business logic.
eg: `core.CreateList()`, `core.GetLists()` etc.
- Remove obsolete subscriber methods.
- Move optin hook queries to core.
- Move campaign methods to `core`.
- Move all campaign methods to `core`.
- Move public page functions to `core`.
- Move all template functions to `core`.
- Move media and settings function to `core`.
- Move handler middleware functions to `core`.
- Move all bounce functions to `core`.
- Move all dashboard functions to `core`.
- Fix GetLists() not honouring type
- Fix unwrapped JSON responses.
- Clean up obsolete pre-core util function.
- Replace SQL array null check with cardinality check.
- Fix missing validations in `core` queries.
- Remove superfluous deps on internal `subimporter`.
- Add dashboard functions to `core`.
- Fix broken domain ban check.
- Fix broken subscriber check middleware.
- Remove redundant error handling.
- Remove obsolete functions.
- Remove obsolete structs.
- Remove obsolete queries and DB functions.
- Document the `core` package.
It was observed that the next-campaign-subscribers query on an instance
with ~9 million subscribers had slowed down significantly. Fetching
a batch of 5k subscribers was taking around ~25 seconds.
After multiple hours of debugging and trial and errors, it turned out
that Postgres was doing very poor query planning on JOINs with CTEs
because of the dynamic cardinality of some CTEs (even with just 1 row).
Afer rewriting the query and adding a hack to overcome the CTE
cardinality issue, the same query now takes a few milliseconds,
a speed up of several orders of magnitude.
