The Alert System

Alert Creation & Storage

Alert creation is handled via the /alert/create endpoint (see api/alerts.py). It creates entries in the alerts table in the database, which can be manipulated via the miscellaneous other endpoints under the /alert path. This is mentioned because tracking sent alerts is not done with this table.

Triggering Alerts

The special endpoint /alert/trigger is used to trigger sending alert emails. It takes an optional parameter frequency which is used by the cronjobs to only trigger subsets of the alerts. For example: /alert/trigger?frequency=weekly triggers weekly alerts only. Omitting the parameter triggers all alerts.

The /alert/trigger endpoint will return 401 unless the requester's IP address is in the comma-separated whitelist found in keys.conf. By default, this contains only 127.0.0.1---meaning that only triggers from the server itself (usually via cron) are accepted.

Note: this uses the TCP address, so while hosting within AWS or another PaaS it is tremendously unlikely that you will be able to use any meaningful address besides localhost due to the proxies placed between the server and the rest of the web.

Periodic Triggers

The script scripts/setup-cron.py creates 3 cronjobs to trigger periodic alerts by calling the /alert/trigger endpoint:

• Weekly (7 days): call /alert/trigger?frequency=weekly
• Semi-Weekly (10 days): call /alert/trigger?frequency=semi-weekly
• Monthly (30 days): call /alert/trigger?frequency=monthly

The frequency is used to prevent new alerts from becoming misaligned with their cron job. For example: if a weekly alert were created between the Weekly and Semi-Weekly triggers, it would otherwise be sent by the Semi-Weekly trigger and then skipped over by the Weekly trigger because it'd been sent only 4 days prior---resulting in an effective send period of 10 days until the periods naturally realigned over time. The frequency flag prevents this.

When Triggered...

When a permitted client triggers an alert, the server proceeds as follows:

1. Obtain a list of all alerts that (1) have confirmed emails and (2; optional) match the specified frequency.
2. Iterates through the list. For each alert:
   1. Determine the correct window for the alert, based on its frequency setting. The possible windows are the past 7, 10, or 30 days.
   2. If the most recent send_date from sent_alerts occurred within the previous window, skip this alert.
   3. Run the stored filter against the leads / annotated_leads tables. If there are no matching items published within the window, skip this alert.
   4. At this point, the alert will be sent. Insert information about the alert into sent_alerts, including a full copy of the alert itself (to preserve its status at time of sending) and the deep link to the DB (db_link). The leads that matched the filter for this alert are additionally stored in sent_alert_contents.
   5. Render & send the alert email (see below for details on the templates).
3. If no errors occurred, respond to the client with {"status": "ok"}.

Alert Email Templates

There are two templates for the alert email: an HTML template (api/templates/alert.html) that most users will see, and a text template (api/templates/alert.txt) that is the fallback for old browsers and some low-end mobile devices.

The following variables are exposed to each template via Jinja2:

Variable	Content
filter_text	Keyword Filter
source_text	Pre-formatted Text Source Filter. To adjust the formatting for this, see format_source in api/mail.py
leads	The first 3 matching leads. Each lead is exactly the same as returned by the API. The maximum number rendered can be adjusted in the call to render_alert in api/alerts.py.
links.alert	The link to the alert results.
links.delete	The link to delete the alert using a secure token.
links.unsubscribe	The link to unsubscribe from all alerts using a secure token. Specifically: this link deletes all alerts associated with the recipient's email address.

Some important things to keep in mind:

• Rendering for HTML emails tends to be much more restricted than for full web pages. Using <table>-based layouts is considered best practice to ensure the email renders correctly across devices (especially Outlook).
• Including both Text and HTML contents, as well as including clear Unsubscribe links helps prevent the email from being flagged as spam.

Updating Tests

Snapshot tests are used to ensure that a change does not accidentally break the rendered emails. If you change the email templates, then you also need to update the snapshots for the tests to pass. This can be done by running:

pytest api --snapshot-update

What is Tracked

When an alert email is sent, the following information is recorded in sent_alerts in the database:

• The ID of the triggering alert (alert_id). Note that this is not specified as a foreign key, so that even if the user deletes the alert later we can still see how many emails were generated for each alert.
• A complete copy of the alert that was matched. If a user modifies the alert later, this copy is not changed.
• The timestamp at which the email was sent (send_date).
• The link sent to the recipient (db_link; the same as links.alert in the template).

Colloquially, I call these entries "sends".

Additionally, the ID of every matched lead is recorded in sent_alert_contents. This allows tracking the interest in each lead relatively easily, as well as interest in kinds of leads by joining onto the leads or annotated_leads tables.