- Make the beginning of handlers consistent with uniform variable declaration
and grouping.
- Add missing comments.
- Fix staticcheck/vet warnings and idiom issues.
This patch removes the forced suffixing of all media upload filenames with
random strings. The upload handler now checks the `media` table to ensure that
the filename being uploaded doesn't exist before forcing a suffix.
Outright rejecting duplicate filenames cannot be done to maintain backwards
compatibility with the old behaviour.
Closes#2277.
This has been a hair-pulling rabbit hole of an issue. #1931 and others.
When the `next-campaign-subscribers` query that fetches $n subscribers
per batch for a campaign returns no results, the manager assumes
that the campaign is done and marks as finished.
Marathon debugging revealed fundamental flaws in qyery's logic that
would incorrectly return 0 rows under certain conditions.
- Based on the "layout" of subscribers for eg: a series of blocklisted
subscribers between confirmed subscribers.
A series of unconfirmed subscribers in a batch belonging to a double
opt-in list.
- Bulk import blocklisting users, but not marking their subscriptions
as 'unsubscribed'.
- Conditions spread across multiple CTEs resulted in returning an
arbitrary number of rows and $N per batch as the selected $N rows
would get filtered out elsewhere, possibly even becoming 0.
After fixing this and testing it on our prod instance that has
15 million subscribers and ~70 million subscriptions in the
`subscriber_lists` table, ended up discovered significant inefficiences
in Postgres query planning. When `subscriber_lists` and campaign list IDs
are joined dynamically (CTE or ANY() or any kind of JOIN that involves)
a query, the Postgres query planner is unable to use the right indexes.
After testing dozens of approaches, discovered that statically passing
the values to join on (hardcoding or passing via parametrized $1 vars),
the query uses the right indexes. The difference is staggering.
For the particular scenario on our large prod DB to pull a batch,
~15 seconds vs. ~50ms, a whopping 300x improvement!
This patch splits `next-campaign-subscribers` into two separate queries,
one which fetches campaign metadata and list_ids, whose values are then
passed statically to the next query to fetch subscribers by batch.
In addition, it fixes and refactors broken filtering and counting logic
in `create-campaign` and `next-campaign` queries.
Closes#1931, #1993, #1986.
- Sent count is no longer the batch size fetched from the DB but is
the actual count of messages sent.
- Pausing and resuming now accurately tracks the last subscriber that
was processed and resumes from there.
- Fix multiple concurrent campaigns blocking.
Closes#1616. Closes#905. Closes#1496. Closes#1250. Closes#1010.
- Adds support for arbitrary file uploads with an admin setting to select allowed file extensions.
- Adds support for attaching media (files) to campaigns.
This is a long pending refactor. All the DB, query, CRUD, and related
logic scattered across HTTP handlers are now moved into a central
`core` package with clean, abstracted methods, decoupling HTTP
handlers from executing direct DB queries and other business logic.
eg: `core.CreateList()`, `core.GetLists()` etc.
- Remove obsolete subscriber methods.
- Move optin hook queries to core.
- Move campaign methods to `core`.
- Move all campaign methods to `core`.
- Move public page functions to `core`.
- Move all template functions to `core`.
- Move media and settings function to `core`.
- Move handler middleware functions to `core`.
- Move all bounce functions to `core`.
- Move all dashboard functions to `core`.
- Fix GetLists() not honouring type
- Fix unwrapped JSON responses.
- Clean up obsolete pre-core util function.
- Replace SQL array null check with cardinality check.
- Fix missing validations in `core` queries.
- Remove superfluous deps on internal `subimporter`.
- Add dashboard functions to `core`.
- Fix broken domain ban check.
- Fix broken subscriber check middleware.
- Remove redundant error handling.
- Remove obsolete functions.
- Remove obsolete structs.
- Remove obsolete queries and DB functions.
- Document the `core` package.
- Blocklist or unsubscribe subscribers based on a bounce threshold
- Add /bounces UI for viewing bounces and in the subscriber view
- Add settings UI for managing bounce settings
- Add support for scanning POP3 bounce mailboxes
- Add a generic webhook for posting custom bounces at /webhooks/bounce
- Add SES bounce webhook support at /webhooks/services/ses
- Add Sendgrid bounce webhook support at /webhooks/services/sendgrid
