feat: enhance notification system and migrate to pubsub/v2 to fix CI

docs/_sidebar.md

@@ -7,6 +7,7 @@
     - [Debug Console](flagr_debugging.md)
 - Server Configuration
     - [Env](flagr_env.md)
+    - [Notifications](flagr_notifications.md)
 - Client SDKs
     - [Ruby SDK 🔗](https://github.com/openflagr/rbflagr)
     - [Go SDK 🔗](https://github.com/openflagr/goflagr)

docs/flagr_notifications.md

@@ -0,0 +1,98 @@
+# Notifications
+
+Flagr provides an integrated notification system that allows you to monitor changes and updates to your operational resources in real-time. You can configure Flagr to automatically send notifications regarding CRUD (Create, Read, Update, Delete) operations over several distinct channels: **Email**, **Slack**, or generic **Webhooks**.
+
+## Tracked Operations
+
+Flagr monitors changes to **flags** and their related configuration. All notifications have `EntityType: "flag"` in the payload.
+
+The following operations trigger notifications:
+
+| Operation | Description |
+|-----------|-------------|
+| `create` | A new flag is created |
+| `update` | Any change to a flag's metadata, enabled state, or any of its associated entities (segments, variants, constraints, distributions, tags) |
+| `delete` | A flag is soft-deleted |
+| `restore` | A soft-deleted flag is restored |
+
+**Note**: Operations such as adding/removing tags, updating segment rollout percentages, modifying constraints, or changing variant attachments all trigger an `update` notification for the parent flag. Enabling or disabling a flag is also considered an update.
+
+## Global Configuration
+
+Notifications are enabled automatically when at least one provider is configured. No global toggle is required.
+
+- `FLAGR_NOTIFICATION_PROVIDER=slack` (Options: `slack`, `email`, `webhook`) - Determines the active transport channel.
+- `FLAGR_NOTIFICATION_DETAILED_DIFF_ENABLED=true` (Default: `false`) - When enabled, Flagr will embed the precise visual JSON diff of the modified entity within the notification payload.
+- `FLAGR_NOTIFICATION_TIMEOUT=10s` (Default: `10s`) - Configures the timeout window for dialing external notification webhooks and email APIs.
+
+### Provider Configuration
+
+Enable at least one provider to activate the notification system:
+
+- `FLAGR_NOTIFICATION_SLACK_ENABLED=true` (Default: `false`) - Enable Slack notifications
+- `FLAGR_NOTIFICATION_EMAIL_ENABLED=true` (Default: `false`) - Enable email notifications  
+- `FLAGR_NOTIFICATION_WEBHOOK_ENABLED=true` (Default: `false`) - Enable generic webhook notifications
+
+### Retry Configuration (HTTP providers only)
+
+- `FLAGR_NOTIFICATION_MAX_RETRIES=3` (Default: `3`) - Maximum number of retry attempts for transient HTTP failures (5xx errors). Set to `0` to disable retries.
+- `FLAGR_NOTIFICATION_RETRY_BASE=1s` (Default: `1s`) - Base delay for exponential backoff between retries.
+- `FLAGR_NOTIFICATION_RETRY_MAX=10s` (Default: `10s`) - Maximum delay between retries.
+
+### Concurrency & Observability
+
+- Notifications are sent asynchronously with a default concurrency limit of 100 to prevent resource exhaustion under load.
+- Metric `notification.sent` is emitted when statsd is enabled, tagged with `provider`, `operation`, `entity_type`, and `status` (`success`/`failure`).
+
+### Important Notes
+
+- **Asynchronous delivery**: Notifications are sent in background goroutines. Failures are logged but **do not affect the API response**.
+- **Startup validation**: Flagr validates the notification configuration at startup and logs warnings if required settings are missing for enabled providers.
+- **Silent fallback**: If a provider is enabled (e.g., `FLAGR_NOTIFICATION_SLACK_ENABLED=true`) but its required settings are missing (e.g., `FLAGR_NOTIFICATION_SLACK_WEBHOOK_URL`), notifications for that provider will be silently dropped. A warning is logged at startup to help diagnose misconfiguration.
+
+## Provider Settings
+
+### 1. Slack
+
+When using Slack, the notification is delivered as a formatted `Mrkdwn` message directly to your channel block.
+
+- `FLAGR_NOTIFICATION_SLACK_WEBHOOK_URL=...` - The Incoming Webhook URL provided by your Slack Workspace.
+- `FLAGR_NOTIFICATION_SLACK_CHANNEL=#engineering` - (Optional) Overrides the destination Slack channel.
+
+### 2. Email
+
+The Email provider sends beautifully formatted HTML summaries of modifications to a target inbox leveraging the SendGrid REST APIs.
+
+- `FLAGR_NOTIFICATION_EMAIL_URL=https://api.sendgrid.com/v3/mail/send` - HTTP email delivery API endpoint.
+- `FLAGR_NOTIFICATION_EMAIL_TO=alerts@your-org.com` - The recipient's email address.
+- `FLAGR_NOTIFICATION_EMAIL_FROM=flagr-ops@your-org.com` - The designated sender address.
+- `FLAGR_NOTIFICATION_EMAIL_API_KEY=...` - The authorization key for evaluating HTTP API calls.
+
+### 3. Generic Webhook
+
+If you wish to consume these events programmatically, the generic `webhook` provider sends HTTP `POST` requests directly to an arbitrary URL containing a serialized JSON `Notification` object representing the change.
+
+- `FLAGR_NOTIFICATION_WEBHOOK_URL=https://api.your-org.com/webhooks/flagr` - HTTP destination endpoint for generic webhook POST requests.
+- `FLAGR_NOTIFICATION_WEBHOOK_HEADERS=Authorization: Bearer secret-token, X-Custom-Header: value` - (Optional) Custom comma-separated HTTP headers, often utilized for securing your webhook receiver with an API token.
+
+---
+
+## The JSON Webhook Payload Format
+
+If `FLAGR_NOTIFICATION_PROVIDER` is set to `webhook`, the target endpoint will receive a structured payload similar to the following:
+
+```json
+{
+  "Operation": "update",
+  "EntityType": "flag",
+  "EntityID": 123,
+  "EntityKey": "my-feature-flag",
+  "Description": "Optional description of the update",
+  "PreValue": "{\"key\": \"value\"}",
+  "PostValue": "{\"key\": \"new_value\"}",
+  "Diff": "--- Previous\n+++ Current\n@@ -1 +1 @@\n-{\"key\": \"value\"}\n+{\"key\": \"new_value\"}",
+  "User": "admin@example.com"
+}
+```
+
+> **Note**: The `Diff` key is visually rendered in Markdown format for rendering natively across internal dashboards or chat systems, but is only populated if `FLAGR_NOTIFICATION_DETAILED_DIFF_ENABLED=true` is set on the server.

go.mod

@@ -13,7 +13,7 @@ require (
 	github.com/bsm/ratelimit v2.0.0+incompatible
 	github.com/caarlos0/env v3.5.0+incompatible
 	github.com/certifi/gocertifi v0.0.0-20210507211836-431795d63e8d // indirect
-	github.com/davecgh/go-spew v1.1.1
+	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
 	github.com/dchest/uniuri v1.2.0
 	github.com/evalphobia/logrus_sentry v0.8.2
 	github.com/form3tech-oss/jwt-go v3.2.5+incompatible
@@ -46,14 +46,13 @@ require (
 	github.com/yadvendar/negroni-newrelic-go-agent v0.0.0-20160803090806-3dc58758cb67
 	github.com/zhouzhuojie/conditions v0.2.3
 	github.com/zhouzhuojie/withtimeout v0.0.0-20190405051827-12b39eb2edd5
-	golang.org/x/net v0.47.0
-	google.golang.org/api v0.247.0
-	google.golang.org/grpc v1.74.2
+	golang.org/x/net v0.48.0
+	google.golang.org/api v0.257.0
+	google.golang.org/grpc v1.77.0
 	gopkg.in/DataDog/dd-trace-go.v1 v1.46.0
 )
 
 require (
-	cloud.google.com/go/pubsub v1.49.0
 	github.com/glebarez/sqlite v1.6.0
 	github.com/newrelic/go-agent v2.1.0+incompatible
 	gorm.io/driver/mysql v1.4.5
@@ -62,35 +61,39 @@ require (
 )
 
 require (
-	github.com/aws/aws-sdk-go-v2/config v1.31.20
+	cloud.google.com/go/pubsub/v2 v2.0.0
+	github.com/aws/aws-sdk-go-v2/config v1.32.5
 	github.com/aws/aws-sdk-go-v2/service/kinesis v1.42.3
+	github.com/nikoksr/notify v1.5.0
+	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
 )
 
 require (
-	cloud.google.com/go/auth v0.16.4 // indirect
+	cloud.google.com/go/auth v0.17.0 // indirect
 	cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
-	cloud.google.com/go/compute/metadata v0.8.0 // indirect
-	cloud.google.com/go/iam v1.5.2 // indirect
+	cloud.google.com/go/compute/metadata v0.9.0 // indirect
+	cloud.google.com/go/iam v1.5.3 // indirect
 	github.com/DataDog/datadog-agent/pkg/obfuscate v0.41.1 // indirect
 	github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.42.0-rc.5 // indirect
 	github.com/DataDog/datadog-go/v5 v5.2.0 // indirect
 	github.com/DataDog/go-tuf v0.3.0--fix-localmeta-fork // indirect
 	github.com/DataDog/sketches-go v1.4.1 // indirect
 	github.com/Microsoft/go-winio v0.6.0 // indirect
 	github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
-	github.com/aws/aws-sdk-go-v2 v1.39.6 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.41.0 // indirect
 	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.3 // indirect
-	github.com/aws/aws-sdk-go-v2/credentials v1.18.24 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.13 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.13 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.13 // indirect
+	github.com/aws/aws-sdk-go-v2/credentials v1.19.5 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 // indirect
 	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.3 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.13 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.30.3 // indirect
-	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.7 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.40.2 // indirect
-	github.com/aws/smithy-go v1.23.2 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 // indirect
+	github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.30.7 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 // indirect
+	github.com/aws/smithy-go v1.24.0 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/dgraph-io/ristretto v0.1.1 // indirect
@@ -122,8 +125,9 @@ require (
 	github.com/google/go-cmp v0.7.0 // indirect
 	github.com/google/s2a-go v0.1.9 // indirect
 	github.com/google/uuid v1.6.0 // indirect
-	github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect
+	github.com/googleapis/enterprise-certificate-proxy v0.3.7 // indirect
 	github.com/googleapis/gax-go/v2 v2.15.0 // indirect
+	github.com/gorilla/websocket v1.5.3 // indirect
 	github.com/hashicorp/errwrap v1.1.0 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
 	github.com/hashicorp/go-uuid v1.0.3 // indirect
@@ -139,22 +143,23 @@ require (
 	github.com/jinzhu/inflection v1.0.0 // indirect
 	github.com/jinzhu/now v1.1.5 // indirect
 	github.com/klauspost/compress v1.18.0 // indirect
-	github.com/mattn/go-isatty v0.0.17 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/oklog/ulid v1.3.1 // indirect
 	github.com/pierrec/lz4/v4 v4.1.17 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
-	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/prometheus/common v0.66.1 // indirect
 	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/secure-systems-lab/go-securesystemslib v0.4.0 // indirect
-	go.einride.tech/aip v0.68.1 // indirect
+	github.com/slack-go/slack v0.17.3 // indirect
+	github.com/stretchr/objx v0.5.3 // indirect
+	go.einride.tech/aip v0.73.0 // indirect
 	go.mongodb.org/mongo-driver v1.17.4 // indirect
 	go.opencensus.io v0.24.0 // indirect
 	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
-	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.61.0 // indirect
-	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.64.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 // indirect
 	go.opentelemetry.io/otel v1.40.0 // indirect
 	go.opentelemetry.io/otel/metric v1.40.0 // indirect
 	go.opentelemetry.io/otel/sdk v1.40.0 // indirect
@@ -164,19 +169,19 @@ require (
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
 	go4.org/intern v0.0.0-20220617035311-6925f38cc365 // indirect
 	go4.org/unsafe/assume-no-moving-gc v0.0.0-20220617031537-928513b29760 // indirect
-	golang.org/x/crypto v0.45.0 // indirect
-	golang.org/x/mod v0.29.0 // indirect
-	golang.org/x/oauth2 v0.30.0 // indirect
-	golang.org/x/sync v0.18.0 // indirect
+	golang.org/x/crypto v0.46.0 // indirect
+	golang.org/x/mod v0.30.0 // indirect
+	golang.org/x/oauth2 v0.34.0 // indirect
+	golang.org/x/sync v0.19.0 // indirect
 	golang.org/x/sys v0.40.0 // indirect
-	golang.org/x/text v0.31.0 // indirect
-	golang.org/x/time v0.12.0 // indirect
-	golang.org/x/tools v0.38.0 // indirect
+	golang.org/x/text v0.32.0 // indirect
+	golang.org/x/time v0.14.0 // indirect
+	golang.org/x/tools v0.39.0 // indirect
 	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
-	google.golang.org/genproto v0.0.0-20250603155806-513f23925822 // indirect
-	google.golang.org/genproto/googleapis/api v0.0.0-20250721164621-a45f3dfb1074 // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20250818200422-3122310a409c // indirect
-	google.golang.org/protobuf v1.36.8 // indirect
+	google.golang.org/genproto v0.0.0-20251202230838-ff82c1b0f217 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20251202230838-ff82c1b0f217 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
+	google.golang.org/protobuf v1.36.10 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	inet.af/netaddr v0.0.0-20220811202034-502d2d690317 // indirect
 	modernc.org/libc v1.22.5 // indirect