Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 132 additions & 0 deletions specs/notifications.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Notification System for Scheduled Triggers

## Problem

Warden's scheduled triggers analyze the full codebase on a cron. Today, findings go to a single tracking issue per skill that gets overwritten each run. There's no way to:
- Get notified in Slack when new findings appear
- Mark findings as false positives so they don't recur
- Track individual findings across runs

## Goals

1. Per-finding GitHub issues with semantic dedup across runs
2. Slack webhook notifications for new findings
3. Suppression file for false positives (rule-based pre-filtering)
4. Replace the single-tracking-issue approach with a provider-based notification layer

## Configuration

### `warden.toml`

```toml
[[notifications]]
type = "github-issues"
labels = ["warden"]

[[notifications]]
type = "slack"
webhookUrl = "$SLACK_WEBHOOK_URL"
```

Top-level `[[notifications]]` array. Each provider receives all non-suppressed findings independently. Environment variables expanded at runtime via `$VAR` syntax.

### Migration

The `[[notifications]]` section replaces the existing `schedule.issueTitle` tracking-issue approach. The `createOrUpdateIssue` code path and `schedule.issueTitle` config are removed. `schedule.createFixPR` and `schedule.fixBranchPrefix` remain (fix PRs are orthogonal to notifications).

## Suppression File

Located at `.agents/warden/suppressions.yaml`:

```yaml
suppressions:
- skill: "security-audit"
paths: ["src/legacy/**"]
reason: "Legacy code, not worth fixing"

- skill: "security-audit"
paths: ["src/admin/query.ts"]
title: "SQL injection"
reason: "Uses parameterized queries, false positive"
```

Rules match on:
- **skill** (required): Exact skill name
- **paths** (required): Glob patterns matched against finding location path
- **title** (optional): Substring match against finding title
- **reason** (required): Human-readable justification

Loaded once per workflow run, applied before any provider receives findings.

## Provider Interface

```typescript
interface NotificationProvider {
readonly name: string;
notify(context: NotificationContext): Promise<NotificationResult>;
}

interface NotificationContext {
findings: Finding[];
reports: SkillReport[];
repository: { owner: string; name: string };
commitSha: string;
}

interface NotificationResult {
provider: string;
sent: number;
skipped: number;
errors: string[];
}
```

## Provider Flow

```
Skill Report -> Apply Suppressions -> All Providers (each gets same findings)
|-- github-issues (semantic dedup, creates/skips per finding)
|-- slack (sends all findings it receives)
```

## GitHub Issues Provider

- Creates one issue per unique finding
- Dedup: two-tier
1. Hash match via `<!-- warden:SHA256 -->` marker in issue body (cheap, catches identical text)
2. Semantic match via Haiku for same-file findings with no hash match (handles LLM variation)
- Also checks closed issues with `warden:false-positive` label (treated as suppressed)
- Issue title: `[Warden] {finding.title}`
- Labels: configurable base labels + `warden:{skillName}`
- Body: severity, description, location with code link, suggested fix, hash marker
- Config: `labels` (string array, default `["warden"]`)

## Slack Provider

- Posts to incoming webhook URL using Block Kit formatting
- Sends all non-suppressed findings it receives
- Message: repo, commit, severity summary, up to 10 findings with details
- Skips notification if findings array is empty
- Config: `webhookUrl` (string, supports `$ENV_VAR` expansion)

## Schedule Workflow Integration

In `src/action/workflow/schedule.ts`, the `createOrUpdateIssue` call is replaced with:

```typescript
const dispatcher = new NotificationDispatcher(providers, suppressions);
const result = await dispatcher.dispatch({
findings: report.findings,
reports: [report],
repository: { owner, name: repo },
commitSha: headSha,
skillName: resolved.name,
});
```

## Future Work

- Persistent storage for "previously seen" finding tracking across all providers
- `autoClose` config flag on GitHub Issues provider (close issues when findings disappear)
- CLI command for managing suppressions (`warden suppress add`)
- PR trigger integration (generic enough, but PR comments already have sophisticated dedup)
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ paths = ["src/**/*.ts"]
type = "schedule"

[skills.triggers.schedule]
issueTitle = "Custom Issue Title"
createFixPR = false
61 changes: 23 additions & 38 deletions src/action/workflow/schedule.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,12 +52,24 @@ vi.mock('../../event/schedule-context.js', () => ({
buildScheduleEventContext: vi.fn(),
}));

// Mock GitHub issue/PR creation
// Mock GitHub PR creation (createOrUpdateIssue removed; notifications handle issues now)
vi.mock('../../output/github-issues.js', () => ({
createOrUpdateIssue: vi.fn(),
createFixPR: vi.fn(),
}));

// Mock suppressions loader
vi.mock('../../suppressions/loader.js', () => ({
loadSuppressions: vi.fn(() => []),
}));

// Mock notification system
vi.mock('../../notifications/index.js', () => ({
NotificationDispatcher: vi.fn().mockImplementation(() => ({
dispatch: vi.fn(() => Promise.resolve({ suppressed: 0, results: [] })),
})),
buildProviders: vi.fn(() => []),
}));

// Mock skill loader — filesystem reads; keep clearSkillsCache real
vi.mock('../../skills/loader.js', async () => {
const actual = await vi.importActual('../../skills/loader.js');
Expand All @@ -76,7 +88,7 @@ vi.mock('../../skills/loader.js', async () => {
// Import after mocks
import { runSkill } from '../../sdk/runner.js';
import { buildScheduleEventContext } from '../../event/schedule-context.js';
import { createOrUpdateIssue, createFixPR } from '../../output/github-issues.js';
import { createFixPR } from '../../output/github-issues.js';
import { resolveSkillAsync } from '../../skills/loader.js';
import { setFailed } from './base.js';
import { runScheduleWorkflow } from './schedule.js';
Expand All @@ -85,7 +97,6 @@ import { clearSkillsCache } from '../../skills/loader.js';
// Type the mocks
const mockRunSkill = vi.mocked(runSkill);
const mockBuildContext = vi.mocked(buildScheduleEventContext);
const mockCreateOrUpdateIssue = vi.mocked(createOrUpdateIssue);
const mockCreateFixPR = vi.mocked(createFixPR);
const mockResolveSkillAsync = vi.mocked(resolveSkillAsync);
const mockSetFailed = vi.mocked(setFailed);
Expand Down Expand Up @@ -195,11 +206,6 @@ describe('runScheduleWorkflow', () => {
// Default mock: context with files, no findings
mockBuildContext.mockResolvedValue(createScheduleContext());
mockRunSkill.mockResolvedValue(createSkillReport());
mockCreateOrUpdateIssue.mockResolvedValue({
issueNumber: 1,
issueUrl: 'https://github.com/test-owner/test-repo/issues/1',
created: true,
});
mockResolveSkillAsync.mockResolvedValue({
name: 'test-skill',
description: 'Test skill',
Expand Down Expand Up @@ -227,7 +233,6 @@ describe('runScheduleWorkflow', () => {
await runScheduleWorkflow(mockOctokit, createDefaultInputs(), PR_ONLY_FIXTURES);

expect(mockRunSkill).not.toHaveBeenCalled();
expect(mockCreateOrUpdateIssue).not.toHaveBeenCalled();
});

it('fails when GITHUB_REPOSITORY is not set', async () => {
Expand Down Expand Up @@ -270,33 +275,22 @@ describe('runScheduleWorkflow', () => {
// ---------------------------------------------------------------------------

describe('happy path', () => {
it('runs skill and creates issue when findings exist', async () => {
it('runs skill when findings exist', async () => {
const finding = createFinding({ severity: 'high' });
const report = createSkillReport({ findings: [finding] });
mockRunSkill.mockResolvedValue(report);

await runScheduleWorkflow(mockOctokit, createDefaultInputs(), SCHEDULE_FIXTURES);

expect(mockRunSkill).toHaveBeenCalledTimes(1);
expect(mockCreateOrUpdateIssue).toHaveBeenCalledWith(
mockOctokit,
'test-owner',
'test-repo',
[report],
expect.objectContaining({
title: 'Warden: test-skill',
commitSha: 'abc123',
})
);
});

it('creates issue even when no findings', async () => {
it('runs skill even when no findings', async () => {
mockRunSkill.mockResolvedValue(createSkillReport({ findings: [] }));

await runScheduleWorkflow(mockOctokit, createDefaultInputs(), SCHEDULE_FIXTURES);

expect(mockRunSkill).toHaveBeenCalledTimes(1);
expect(mockCreateOrUpdateIssue).toHaveBeenCalledTimes(1);
});

it('skips skill run when no files match trigger', async () => {
Expand All @@ -319,15 +313,14 @@ describe('runScheduleWorkflow', () => {
await runScheduleWorkflow(mockOctokit, createDefaultInputs(), SCHEDULE_FIXTURES);

expect(mockRunSkill).not.toHaveBeenCalled();
expect(mockCreateOrUpdateIssue).not.toHaveBeenCalled();
});
});

// ---------------------------------------------------------------------------
// Issue & PR Creation
// Fix PR Creation
// ---------------------------------------------------------------------------

describe('issue and PR creation', () => {
describe('fix PR creation', () => {
it('creates fix PR when schedule.createFixPR is enabled', async () => {
const finding = createFinding({
suggestedFix: { description: 'Fix it', diff: '--- a\n+++ b\n' },
Expand Down Expand Up @@ -369,25 +362,17 @@ describe('runScheduleWorkflow', () => {
expect(mockCreateFixPR).not.toHaveBeenCalled();
});

it('uses custom issue title from schedule config', async () => {
const report = createSkillReport({ findings: [] });
mockRunSkill.mockResolvedValue(report);
it('does not create fix PR for schedule-title fixture', async () => {
const finding = createFinding();
mockRunSkill.mockResolvedValue(createSkillReport({ findings: [finding] }));

await runScheduleWorkflow(
mockOctokit,
createDefaultInputs(),
SCHEDULE_TITLE_FIXTURES
);

expect(mockCreateOrUpdateIssue).toHaveBeenCalledWith(
mockOctokit,
'test-owner',
'test-repo',
[report],
expect.objectContaining({
title: 'Custom Issue Title',
})
);
expect(mockCreateFixPR).not.toHaveBeenCalled();
});
});

Expand Down
37 changes: 21 additions & 16 deletions src/action/workflow/schedule.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@
import { dirname, join } from 'node:path';
import type { Octokit } from '@octokit/rest';
import { loadWardenConfig, resolveSkillConfigs } from '../../config/loader.js';
import type { ScheduleConfig } from '../../config/schema.js';
import { buildScheduleEventContext } from '../../event/schedule-context.js';
import { runSkill } from '../../sdk/runner.js';
import { createOrUpdateIssue, createFixPR } from '../../output/github-issues.js';
import { createFixPR } from '../../output/github-issues.js';
import { shouldFail, countFindingsAtOrAbove, countSeverity } from '../../triggers/matcher.js';
import { resolveSkillAsync } from '../../skills/loader.js';
import { loadSuppressions } from '../../suppressions/loader.js';
import { NotificationDispatcher, buildProviders } from '../../notifications/index.js';
import type { SkillReport } from '../../types/index.js';
import type { ActionInputs } from '../inputs.js';
import {
Expand Down Expand Up @@ -70,6 +71,13 @@ export async function runScheduleWorkflow(

const defaultBranch = await getDefaultBranchFromAPI(octokit, owner, repo);

// Load suppressions and build notification providers
const suppressions = loadSuppressions(repoPath);
const providers = config.notifications
? buildProviders({ configs: config.notifications, octokit, apiKey: inputs.anthropicApiKey })
: [];
const dispatcher = new NotificationDispatcher(providers, suppressions);

logGroup('Processing schedule triggers');
for (const trigger of scheduleTriggers) {
console.log(`- ${trigger.name}: ${trigger.skill}`);
Expand Down Expand Up @@ -127,24 +135,21 @@ export async function runScheduleWorkflow(
allReports.push(report);
totalFindings += report.findings.length;

// Create/update issue with findings
const scheduleConfig: Partial<ScheduleConfig> = resolved.schedule ?? {};
const issueTitle = scheduleConfig.issueTitle ?? `Warden: ${resolved.name}`;

const issueResult = await createOrUpdateIssue(octokit, owner, repo, [report], {
title: issueTitle,
commitSha: headSha,
});

if (issueResult) {
console.log(`${issueResult.created ? 'Created' : 'Updated'} issue #${issueResult.issueNumber}`);
console.log(`Issue URL: ${issueResult.issueUrl}`);
// Dispatch notifications for findings
if (providers.length > 0) {
await dispatcher.dispatch({
findings: report.findings,
reports: [report],
repository: { owner, name: repo },
commitSha: headSha,
skillName: resolved.name,
});
}

// Create fix PR if enabled and there are fixable findings
if (scheduleConfig.createFixPR) {
if (resolved.schedule?.createFixPR) {
const fixResult = await createFixPR(octokit, owner, repo, report.findings, {
branchPrefix: scheduleConfig.fixBranchPrefix ?? 'warden-fix',
branchPrefix: resolved.schedule.fixBranchPrefix ?? 'warden-fix',
baseBranch: defaultBranch,
baseSha: headSha,
repoPath,
Expand Down
25 changes: 23 additions & 2 deletions src/config/schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,34 @@ export type SkillDefinition = z.infer<typeof SkillDefinitionSchema>;

// Schedule-specific configuration
export const ScheduleConfigSchema = z.object({
/** Title for the tracking issue (default: "Warden: {skillName}") */
issueTitle: z.string().optional(),
/** Create PR with fixes when suggestedFix is available */
createFixPR: z.boolean().default(false),
/** Branch prefix for fix PRs (default: "warden-fix") */
fixBranchPrefix: z.string().default('warden-fix'),
});
export type ScheduleConfig = z.infer<typeof ScheduleConfigSchema>;

// Notification provider configurations
export const GitHubIssuesNotificationSchema = z.object({
type: z.literal('github-issues'),
/** Labels to apply to created issues (default: ["warden"]) */
labels: z.array(z.string()).default(['warden']),
});
export type GitHubIssuesNotificationConfig = z.infer<typeof GitHubIssuesNotificationSchema>;

export const SlackNotificationSchema = z.object({
type: z.literal('slack'),
/** Slack incoming webhook URL. Supports $ENV_VAR expansion. */
webhookUrl: z.string().min(1),
});
export type SlackNotificationConfig = z.infer<typeof SlackNotificationSchema>;

export const NotificationConfigSchema = z.discriminatedUnion('type', [
GitHubIssuesNotificationSchema,
SlackNotificationSchema,
]);
export type NotificationConfig = z.infer<typeof NotificationConfigSchema>;

// Trigger type: where the trigger runs
export const TriggerTypeSchema = z.enum(['pull_request', 'local', 'schedule']);
export type TriggerType = z.infer<typeof TriggerTypeSchema>;
Expand Down Expand Up @@ -183,6 +202,8 @@ export const WardenConfigSchema = z
defaults: DefaultsSchema.optional(),
skills: z.array(SkillConfigSchema).default([]),
runner: RunnerConfigSchema.optional(),
/** Notification providers for scheduled trigger findings */
notifications: z.array(NotificationConfigSchema).optional(),
})
.superRefine((config, ctx) => {
const names = config.skills.map((s) => s.name);
Expand Down
Loading
Loading