diff --git a/docs/pages/incident-management/communication-strategies.mdx b/docs/pages/incident-management/communication-strategies.mdx index c52885f7..1df5a1a6 100644 --- a/docs/pages/incident-management/communication-strategies.mdx +++ b/docs/pages/incident-management/communication-strategies.mdx @@ -4,6 +4,10 @@ description: "Establish secure communication channels for incident response. App tags: - Security Specialist - Operations & Strategy + +contributors: + - role: wrote + users: [dickson] --- import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../components' @@ -17,7 +21,7 @@ import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } fr Communication during an incident can be very hard, as people are often scrambling to fix the issue at hand. Nonetheless, -from aa team member, outsider or observer's point of view, communication is very important to be able to understand +from a team member, outsider or observer's point of view, communication is very important to be able to understand what's happening, and it also provide some time to reflect and think about what is going on. With that said, providing information before confirming that it's accurate, can often be very negative and cause uncertainty. It is recommended to have a person designated for communication during an incident, and that updates are sent out on a fixed schedule, and @@ -37,6 +41,9 @@ responsibilities. 6. Be transparent with external stakeholders about the incident, the impact, and the steps being taken to address it. Avoid speculation and provide factual information. +For message templates and example public updates, see +[Incident Response Template: Communications](/incident-management/incident-response-template/communications). + --- diff --git a/docs/pages/incident-management/incident-detection-and-response.mdx b/docs/pages/incident-management/incident-detection-and-response.mdx index b1e1884b..cb50c695 100644 --- a/docs/pages/incident-management/incident-detection-and-response.mdx +++ b/docs/pages/incident-management/incident-detection-and-response.mdx @@ -4,6 +4,10 @@ description: "Detect security incidents early with continuous on-chain monitorin tags: - Security Specialist - Operations & Strategy + +contributors: + - role: wrote + users: [dickson] --- import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../components' @@ -37,6 +41,11 @@ incidents. - **Post-Incident Review**: Conduct a thorough review of the incident to identify lessons learned and improve future response efforts. +For a complete incident response policy template covering roles, severity, documentation, and response flow, see +[Incident Response Template: Incident Response Policy](/incident-management/incident-response-template/incident-response-policy) +and +[Incident Response Template: Roles and Staffing](/incident-management/incident-response-template/roles-and-staffing). + --- diff --git a/docs/pages/incident-management/incident-response-template/communications.mdx b/docs/pages/incident-management/incident-response-template/communications.mdx new file mode 100644 index 00000000..76e3df9b --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/communications.mdx @@ -0,0 +1,212 @@ +--- +title: "Communication Templates | Security Alliance" +description: "Templates and building blocks for incident communications. Adapt these to your situation and tone." +tags: + - Security Specialist + - Operations & Strategy + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../components' + + + + +# Communication Templates + + + + +Templates and building blocks for incident communications. Adapt these to your situation and tone. + +## Before You Post + +### Checklist + +- [ ] Get approval from Incident Leader or Decision Maker +- [ ] Verify facts are accurate +- [ ] Avoid speculation about root cause (until confirmed) +- [ ] Include what users should do (or not do) +- [ ] State when you'll provide the next update + +### What to Include + +| Element | When to Include | +|---------|-----------------| +| What happened (high level) | Always | +| User funds are safe | If true | +| What users should do | If action needed | +| What users should NOT do | If relevant (e.g., don't interact with X) | +| What still works | If partial outage | +| When you'll update next | Always | +| Link to status page or thread | If available | + +--- + +## Building Blocks + +Use these as modular pieces. Combine as needed for your situation. + +### Acknowledgment +> We're aware of an issue affecting [service/feature] and are actively investigating. + +### Funds Safe +> User funds are safe and have not been affected. + +### Funds at Risk (be careful) +> We are investigating a potential security issue. Out of caution, we recommend users [specific action]. + +### Action Required +> If you [specific condition], please [specific action]. + +### Do Not Interact +> Do not interact with [specific thing] until further notice. + +### Service Paused +> We have temporarily paused [service/feature] while we investigate. + +### Partial Outage +> [Feature X] is currently unavailable. [Feature Y] and [Feature Z] continue to work normally. + +### Timeline Unknown +> We don't have an ETA for resolution yet. We'll provide updates as we learn more. + +### Next Update +> We'll provide an update within [timeframe] or sooner if the situation changes. + +### Resolution +> The issue has been resolved. [Brief description of what happened and fix]. + +### Post-Mortem Coming +> We'll publish a detailed post-mortem within [timeframe]. + +--- + +## Example Templates + +### Protocol Paused + +**For: Twitter/X, Discord announcement** + +> We have temporarily paused [protocol/feature] while we investigate a potential issue. +> +> User funds are safe. [OR: We are still assessing the situation.] +> +> Do not interact with [specific contracts/UI] until we confirm the issue is resolved. +> +> We'll provide an update within [1 hour / as soon as we know more]. + +--- + +### Website/Frontend Down + +**For: Twitter/X, Discord announcement** + +> Our website is currently unavailable. We've taken it offline while we investigate [a potential security issue / technical problems]. +> +> Your funds in the protocol are not affected. Do not approve any transactions from sites claiming to be [protocol name] until we confirm service is restored. +> +> Follow this thread for updates. + +--- + +### Social Account Compromised + +**For: Alternate channel (Discord if Twitter compromised, etc.)** + +> The [Twitter/Discord/Telegram] account of [person/official account] has been compromised. +> +> Do NOT click any links or interact with messages from that account. +> +> We are working to recover the account. Any legitimate announcements will come from [list alternate verified channels]. +> +> If you interacted with any links, revoke token approvals immediately at [revoke.cash or similar]. + +--- + +### Active Exploit (P1) + +**For: Initial announcement, keep brief** + +> We are aware of a security incident affecting [protocol/feature]. +> +> We are actively responding and will share more information as soon as possible. +> +> [If applicable: We have paused affected contracts.] +> +> Do not interact with [specific thing] until further notice. + +**For: Follow-up once stabilized** + +> Update on the security incident: +> +> [What happened - high level] +> [Current status] +> [What users should do] +> [Funds status - be precise about what was/wasn't affected] +> +> We'll publish a full post-mortem within [timeframe]. + +--- + +### Third-Party Outage + +**For: When the issue is not your fault** + +> [Feature] is currently unavailable due to an outage at [provider/third-party]. +> +> Your funds are safe. This is affecting [what's broken] but [what still works] continues to function normally. +> +> We're monitoring the situation and will restore service when [provider] resolves the issue. + +--- + +### Issue Resolved + +**For: Closing out an incident** + +> The issue affecting [service/feature] has been resolved. +> +> [One sentence on what happened] +> [One sentence on the fix] +> +> Thank you for your patience. We'll share a post-mortem with more details within [timeframe]. + +--- + +## Channel-Specific Notes + +### Twitter/X +- Keep initial post short +- Use thread for updates +- Pin important updates + +### Discord +- Use @everyone or @here sparingly (P1 only) +- Create dedicated thread for ongoing updates +- Lock thread after resolution to preserve record + +### Telegram +- Pin critical messages +- Consider disabling chat during active incident to reduce noise + +--- + +## Tone Guidelines + +- Be direct and factual +- Avoid jargon users won't understand +- Don't speculate on root cause until confirmed +- Don't blame (individuals, third parties, users) +- Acknowledge impact on users +- Avoid excessive apologies (one is enough) + +--- + +*See [Incident-Response-Policy](./incident-response-policy) for the overall response process.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/contacts.mdx b/docs/pages/incident-management/incident-response-template/contacts.mdx new file mode 100644 index 00000000..6d6d3c67 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/contacts.mdx @@ -0,0 +1,163 @@ +--- +title: "Critical Contacts | Security Alliance" +description: "Keep this updated. Stale contacts during an incident waste critical time." +tags: + - Security Specialist + - Operations & Strategy + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../components' + + + + +# Critical Contacts + + + + +Keep this updated. Stale contacts during an incident waste critical time. + +**Last verified:** [DATE] + +--- + +## Emergency Contacts + +### SEAL 911 + +Industry-wide emergency response coordination for Web3 security incidents. + +| Field | Value | +|-------|-------| +| **Website** | https://securityalliance.org/seal-911 | +| **When to contact** | Active exploits, cross-protocol incidents, need for industry coordination | +| **How to contact** | [Add contact method from their site] | + +> SEAL 911 can help coordinate response across the industry and connect you with security researchers and other affected protocols during major incidents. + +--- + +## Internal Team + +> **Tip:** Use team/group emails (e.g., `team-security@company.com`) instead of personal contacts where possible. Group emails are more resilient to team changes and reduce maintenance burden. + +### Decision Makers + +| Role | Name | Phone | Telegram/Signal | Email | +|------|------|-------|-----------------|-------| +| | | | | | +| | | | | | +| | | | | | + +### Subject Matter Experts + +| Domain | Primary | Backup | Contact | +|--------|---------|--------|---------| +| Smart Contracts | | | | +| Infrastructure | | | | +| Frontend | | | | +| Security | | | | + +### On-Call + +| Schedule | Current On-Call | Rotation Link | +|----------|-----------------|---------------| +| | | | + +--- + +## Security Partners + +### Auditors + +| Firm | Primary Contact | Email | When to Contact | +|------|-----------------|-------|-----------------| +| | | | Code review, vulnerability assessment | +| | | | | + +### Incident Response Retainer + +| Firm | Primary Contact | Phone | Email | +|------|-----------------|-------|-------| +| | | | | + +### Bug Bounty Platform + +| Platform | Program Link | Contact | +|----------|--------------|---------| +| | | | + +--- + +## Infrastructure Vendors + +| Service | Provider | Support Contact | Escalation | +|---------|----------|-----------------|------------| +| Hosting | | | | +| CDN | | | | +| DNS | | | | +| RPC Provider | | | | +| Monitoring | | | | + +--- + +## Legal & Communications + +| Role | Name/Firm | Contact | +|------|-----------|---------| +| Legal Counsel | | | +| PR/Communications | | | + +--- + +## Key Partners + +Protocols, exchanges, or other partners to notify during major incidents. + +| Partner | Relationship | Contact | When to Notify | +|---------|--------------|---------|----------------| +| | | | | +| | | | | + +--- + +## Communication Channels + +| Purpose | Platform | Channel/Link | +|---------|----------|--------------| +| Incident coordination | | | +| Public updates | | | +| Team alerts | | | + +--- + +## Quick Reference - P1 Escalation + +For P1 (critical) incidents, contact in this order: + +1. [ ] SEAL 911 (if active exploit) +2. [ ] Decision Makers (internal) +3. [ ] Security Partners (if needed) +4. [ ] Legal (if fund loss or regulatory implications) + +--- + +## Maintenance + +- [ ] Review and update quarterly +- [ ] Verify contact info after team changes +- [ ] Test escalation paths periodically + +**Owner:** [Assign someone to maintain this document] + +--- + +*See [Incident-Response-Policy](./incident-response-policy) and [Roles-and-Staffing](./roles-and-staffing) for how these contacts fit into the response process.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/incident-logs/2024-01-15-example-api-outage.mdx b/docs/pages/incident-management/incident-response-template/incident-logs/2024-01-15-example-api-outage.mdx new file mode 100644 index 00000000..e7796fb6 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/incident-logs/2024-01-15-example-api-outage.mdx @@ -0,0 +1,158 @@ +--- +title: "Incident: API Outage - Rate Limiter Misconfiguration | SEAL" +description: "This is an example incident log. Delete this file or use it as reference when creating real incident logs." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Incident: API Outage - Rate Limiter Misconfiguration + + + + +> **This is an example incident log.** Delete this file or use it as reference when creating real incident logs. + +## Summary + +| Field | Value | +|-------|-------| +| **Status** | Resolved | +| **Severity** | P2 | +| **Start Time** | 2024-01-15 14:32 UTC | +| **Resolution Time** | 2024-01-15 16:45 UTC | +| **Affected Services** | API, Frontend (degraded) | + +## Roles + +| Role | Person | +|------|--------| +| Detector | Monitoring (DataDog alert) | +| Incident Leader | Alice | +| Scribe | Bob | +| Communication Manager | Carol | +| Responders | Alice, Bob, Dave (infra SME) | + +## Communication Channels + +- **Call:** https://meet.google.com/xxx-xxxx-xxx +- **Chat:** #incident-2024-01-15-api + +--- + +## Timeline + +``` +14:32 UTC - DataDog alert fired: API error rate >5% +14:35 UTC - Bob acknowledged alert, started investigating +14:38 UTC - Bob escalated to #incident channel, started call +14:42 UTC - Alice joined, assigned as Incident Leader +14:45 UTC - Bob assigned as Scribe +14:48 UTC - Dave (infra SME) joined +14:52 UTC - Identified: rate limiter rejecting legitimate requests +14:55 UTC - Carol joined as Communication Manager +15:05 UTC - Root cause identified: bad config deployed at 14:15 UTC +15:12 UTC - Decision: rollback config +15:18 UTC - Rollback initiated +15:25 UTC - Rollback complete, monitoring +15:40 UTC - Error rates back to normal +15:45 UTC - Carol posted update to Discord +16:00 UTC - Continued monitoring, no issues +16:45 UTC - Incident declared resolved +``` + +--- + +## Investigation + +### What We Know + +- API started returning 429 errors at 14:32 UTC +- Rate limiter was updated at 14:15 UTC as part of routine config change +- New config had threshold set to 10 req/min instead of 1000 req/min (typo) +- Affected all API endpoints + +### Affected Services + +| Service | Impact | Status | +|---------|--------|--------| +| API | 429 errors for most requests | Resolved | +| Frontend | Degraded (API calls failing) | Resolved | +| Smart Contracts | No impact | N/A | + +### Root Cause (initial assessment) + +Typo in rate limiter configuration: `10` instead of `1000` for requests per minute threshold. + +--- + +## Actions + +### Immediate + +- [x] Identify source of errors @Dave +- [x] Find recent changes @Dave +- [x] Prepare rollback @Dave + +### Resolution + +- [x] Rollback config change @Dave +- [x] Verify API functioning @Bob +- [x] Post community update @Carol + +--- + +## Resolution Summary + +### Mitigation Applied + +Rolled back rate limiter configuration to previous known-good version. + +### Verification + +- [x] API error rate back to baseline (<0.1%) +- [x] Sample API calls succeeding +- [x] No user reports of issues post-fix + +--- + +## Communications Sent + +| Time | Channel | Summary | +|------|---------|---------| +| 15:00 UTC | Discord #announcements | "Investigating API issues" | +| 15:45 UTC | Discord #announcements | "API issues resolved" | +| 16:00 UTC | Twitter | Brief update (optional) | + +--- + +## Post-Incident + +- [x] Post-mortem scheduled for: 2024-01-17 15:00 UTC +- [x] [Post-mortem document](../post-mortems/2024-01-15-example-api-outage) created +- [ ] Action items assigned (pending post-mortem) + +--- + +## Links & Evidence + +- DataDog dashboard: [link] +- Config PR that caused issue: [link] +- Rollback PR: [link] + +--- + +*See [Incident-Response-Policy](../incident-response-policy) for severity definitions and process.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/incident-logs/overview.mdx b/docs/pages/incident-management/incident-response-template/incident-logs/overview.mdx new file mode 100644 index 00000000..4dbfc924 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/incident-logs/overview.mdx @@ -0,0 +1,45 @@ +--- +title: "Incident Logs | Security Alliance" +description: "Store incident logs here. Create a new file for each incident." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Incident Logs + + + + +Store incident logs here. Create a new file for each incident. + +## Creating a New Log + +1. Copy [Incident Log Template](../templates/incident-log-template) +2. Name it: `YYYY-MM-DD-brief-description.md` +3. Fill in details as the incident progresses + +## Example + +See [2024-01-15-example-api-outage](./2024-01-15-example-api-outage) for a sample filled-out incident log. + +## Past Incidents + +{/* Add links to your incident logs as they're created */} + +--- + +*See [Incident-Response-Policy](../incident-response-policy) for the full response process.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/incident-response-policy.mdx b/docs/pages/incident-management/incident-response-template/incident-response-policy.mdx new file mode 100644 index 00000000..630b342c --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/incident-response-policy.mdx @@ -0,0 +1,212 @@ +--- +title: "Incident Response Policy | Security Alliance" +description: "This policy defines how we respond to security and operational incidents. It covers roles, severity classification, and the steps from detection through post..." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../components' + + + + +# Incident Response Policy + + + + +## Overview + +This policy defines how we respond to security and operational incidents. It covers roles, severity classification, and the steps from detection through post-incident review. + +For role structures and on-call options, see [Roles-and-Staffing](./roles-and-staffing). + +--- + +## Roles + +> One person can hold multiple roles. Roles can be reassigned during an incident as needed. + +### Detector +The person who identifies the incident. Their job is to notify responders and hand off. They don't need to fix it. + +### Incident Leader +Coordinates the response. Assigns tasks, makes decisions, ensures the process is followed. Escalates to Decision Makers when needed. + +### Scribe +Documents everything in the [Incident Log](./templates/incident-log-template). Maintains timestamps (UTC), captures decisions and rationale. This is a focused role. Don't also assign the Scribe to fix things. + +### Communication Manager +Handles internal and external communications. Drafts updates, coordinates with PR if needed, manages community channels. + +### Subject Matter Experts (SMEs) +Technical specialists called in based on incident type (smart contracts, infrastructure, security, etc.). + +### Decision Makers +Senior leadership for high-stakes decisions. Define who these are for your protocol (founders, security lead, legal, etc.). + +--- + +## Severity Levels + +> **When in doubt, choose the higher severity.** A false P1 creates noise. A missed P1 costs funds. + +### P1 - Critical + +| Aspect | Details | +|--------|---------| +| **Impact** | Loss of funds, critical systems down, active exploit | +| **Response** | Immediate. Core team. Scale as needed. Decision Makers involved. | +| **Examples** | Active exploit, private key compromise, critical smart contract vulnerability, production down | + +### P2 - High + +| Aspect | Details | +|--------|---------| +| **Impact** | High impact to production, potential fund loss under specific conditions | +| **Response** | Immediate | +| **Examples** | Major vulnerability (not actively exploited), significant outage, DDoS on core services | + +### P3 - Moderate + +| Aspect | Details | +|--------|---------| +| **Impact** | Medium impact, no fund loss likely | +| **Response** | Within hours | +| **Examples** | Minor vulnerability, degraded performance, non-critical service down | + +### P4 - Low + +| Aspect | Details | +|--------|---------| +| **Impact** | Low impact, no fund loss | +| **Response** | Can be scheduled | +| **Examples** | Minor bugs, display issues, non-urgent fixes | + +### P5 - Info + +| Aspect | Details | +|--------|---------| +| **Impact** | Informational, often from automated systems | +| **Response** | No immediate action | +| **Examples** | Expiring certificates, resource spikes, maintenance notices | + +--- + +## Response Process + +### Step 1: Detection + +Incidents can be detected via: +- **Monitoring alerts** (Grafana, DataDog, on-chain monitors, etc.) +- **Community reports** (Discord, Twitter, Telegram) +- **Team members** noticing something wrong +- **Bug bounty reports** +- **Security audits** +- **Partner notifications** + +The Detector's job: get the right people involved, fast. + +> **Don't know who to call?** Contact your incident response on-call team (e.g., DevOps or SecOps) via on-call system or team-wide email (e.g., `team-security@company.com`). These serve as fallbacks when the detector is unfamiliar with escalation paths. + +### Step 2: Coordination + +**Detector responsibilities:** +1. Start a call (Zoom/Meet/Huddle) +2. Create a private channel (`#incident-[brief-description]`) +3. Alert responders via your alerting system or direct contact +4. Provide all known information +5. Hand off to an Incident Leader (get explicit acknowledgment) + +**For P1 incidents:** Keep the group minimal initially. Alert Decision Makers immediately. + +**Incident Leader responsibilities:** +1. Pull in relevant SMEs +2. Assign a Scribe → they create an [Incident Log](./templates/incident-log-template) +3. Assign Communication Manager(s) + +### Step 3: Investigation + +**Goal:** Understand what's happening and assess impact. + +- Collect logs, error messages, reproduction steps +- Identify affected services and scope of impact +- Confirm or adjust severity level +- Determine mitigation options + +**The Incident Leader assigns specific tasks to individuals.** One task per person at a time keeps things focused. + +### Step 4: Resolution + +**Goal:** Stop the bleeding first, permanent fix later. + +If a temporary fix (rollback, pause, disable feature) is faster than a full fix and reduces damage, do that first. + +**Checklist:** +- [ ] Apply temporary mitigation +- [ ] Verify it's working +- [ ] Notify stakeholders +- [ ] Plan permanent fix with owner and timeline + +See [Runbooks](./runbooks/overview) for step-by-step guides for specific incident types. + +### Step 5: Monitoring + +**Goal:** Confirm the fix actually worked. + +- Verify immediately after deployment +- Monitor for at least a week +- Consider adding new alerts or test cases +- Document what monitoring is now in place + +### Step 6: Post-Incident Review + +**Goal:** Learn and prevent recurrence. + +1. Incident Leader schedules post-mortem (within a week of resolution) +2. Scribe prepares [Post-Mortem](./templates/post-mortem-template) draft +3. Team reviews timeline, identifies root causes, captures lessons +4. Define action items with owners and deadlines +5. Share with team (and community if appropriate) + +> All action items must have owners and deadlines. Track them to completion. + +--- + +## Communication Guidelines + +- **Internal:** Regular updates in the incident channel. Frequency depends on severity. +- **External:** Communication Manager drafts, gets approval before posting. See [Communications](./communications) for examples. +- **Transparency:** Default to sharing post-mortems publicly (redacting sensitive details). + +--- + +## Related Documents + +- [Roles-and-Staffing](./roles-and-staffing) - Team structure options +- [Communications](./communications) - Public announcement guidance +- [Incident Log Template](./templates/incident-log-template) - Fill out during incidents +- [Post-Mortem Template](./templates/post-mortem-template) - Complete after incidents +- [Contacts](./contacts) - Critical contacts +- [Runbooks](./runbooks/overview) - Step-by-step incident guides + +--- + +## Document Control + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | Initial release | + +--- + +*Customize this policy for your protocol's tools, team structure, and communication channels.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/overview.mdx b/docs/pages/incident-management/incident-response-template/overview.mdx new file mode 100644 index 00000000..a0a47562 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/overview.mdx @@ -0,0 +1,163 @@ +--- +title: "Incident Response Template for Web3 Protocols | SEAL" +description: "A practical template for building incident response capabilities at cryptocurrency protocols and DAOs." +tags: + - Security Specialist + - Operations & Strategy + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../components' + + + + +# Incident Response Template for Web3 Protocols + + + + +A practical template for building incident response capabilities at cryptocurrency protocols and DAOs. + +> **This is a template, not a turnkey solution.** Every document, runbook, and checklist must be reviewed and customized for your specific protocol, team, and infrastructure. The example runbooks contain generic guidance. +> Do not rely on them without filling in your actual commands, contacts, and procedures. Test your IR capability through tabletop exercises before you need it for real. + +## Why Incident Response Matters + +Web3 protocols face unique challenges: +- **Immutability**: On-chain actions often can't be reversed +- **24/7 Operations**: Blockchain networks never sleep +- **High Stakes**: Incidents can mean immediate, irrecoverable fund loss +- **Adversarial Environment**: Active attackers constantly probing + +Without a plan, you'll waste critical time figuring out *who does what* while an exploit drains funds. You can't prevent all incidents. Focus on detecting quickly, responding effectively, and learning continuously. + +## How to Build Your IR Capability + +### Start Simple + +Don't over-engineer. A basic IR capability needs: + +1. **Severity levels** - How bad is this? (P1-P5 scale) +2. **Roles** - Who does what during an incident? +3. **Communication channels** - Where do we coordinate? +4. **Documentation** - How do we capture what happened? + +That's it to start. Add complexity only as you need it. + +### Scale With Your Team + +**Small teams (2-5 people):** +- Everyone wears multiple hats +- Define who leads incidents vs. who executes fixes +- Establish a single communication channel +- Keep one person documenting + +**Medium teams (5-15 people):** +- Designate subject matter experts by domain +- Consider a simple on-call rotation +- Separate the "fix it" people from "communicate about it" people + +**Larger teams (15+ people):** +- Formal First Responder program with training +- Parallel on-call schedules (infra vs. smart contracts) +- Dedicated communication/PR coordination +- Regular tabletop exercises + +### Practice Before You Need It + +- Read through your policy when there's no emergency +- Run a tabletop exercise quarterly (even 30 minutes helps) +- Review past incidents (yours or others via [Rekt News](https://rekt.news/)) +- Update docs when you find gaps + +## Using This Template + +This is an [Obsidian](https://obsidian.md/) vault you can clone and customize. + +### What's Included + +| Document | Purpose | +| ------------------------------------------------------- | --------------------------------------------------------------- | +| [Incident-Response-Policy](./incident-response-policy) | Core policy defining roles, severity levels, and response steps | +| [Roles-and-Staffing](./roles-and-staffing) | Options for structuring your response team | +| [Communications](./communications) | Guidance and examples for public announcements | +| [Contacts](./contacts) | Critical partner and vendor contact sheet | +| [Templates](./templates/overview) | Incident log and post-mortem templates | +| [Incident Logs](./incident-logs/overview) | Store active and past incident logs | +| [Post-Mortems](./post-mortems/overview) | Store completed post-mortems | +| [Runbooks](./runbooks/overview) | Step-by-step guides for specific incident types | + +### Customization Checklist + +Before using this template, customize these sections: + +- [ ] **Incident-Response-Policy** + - [ ] Update severity level examples for your protocol + - [ ] Add your communication tools (Slack/Discord/Telegram) + - [ ] Add your alerting tools (PagerDuty/etc.) + - [ ] Define your Decision Makers +- [ ] **Contacts** + - [ ] Fill in your team contacts + - [ ] Add security partners (auditors, IR firms) + - [ ] Add infrastructure vendors + - [ ] Add legal/PR contacts +- [ ] **Roles-and-Staffing** + - [ ] Choose the model that fits your team size + - [ ] Assign initial role holders +- [ ] **Runbooks** + - [ ] Customize for your specific infrastructure + - [ ] Add protocol-specific scenarios + - [ ] Fill in actual commands and dashboards + +### Quick Start + +1. Clone this repo +2. Open in Obsidian (or your preferred markdown editor) +3. Work through the customization checklist above +4. Share with your team and walk through together +5. Run a tabletop exercise to test it + +## Related Documentation + +These documents support effective incident response but typically live elsewhere in your protocol's documentation: + +| Document | Why It Helps | +|----------|--------------| +| **Access Control Inventory** | List of privileged roles, multisigs, admin keys, and what each controls. Critical for understanding blast radius during key compromise. | +| **Protocol Architecture** | Diagrams and descriptions of how components interact. Helps responders understand impact and dependencies. | +| **Threat Model** | Documented risks, attack vectors, and mitigations. Speeds up investigation when you can reference known threats. | +| **Audit Reports** | Findings from security audits. Known issues and accepted risks inform incident triage. | +| **Tabletop Exercise Reports** | Learnings from practice incidents. Reveals gaps before real incidents expose them. | +| **Dependency Inventory** | Third-party services, oracles, bridges your protocol relies on. Useful for third-party outage response. | +| **Deployment Procedures** | How to deploy, pause, or upgrade contracts. Essential for resolution steps in runbooks. | +| **Monitoring Documentation** | What's being monitored, alerting thresholds, and infrastructure used. Alerts in your paging system should link directly to relevant runbooks. | + +You don't need all of these to start. Build them over time as your protocol matures. + +## Key Principles + +**When in doubt, escalate.** Treating a P2 as P1 creates some noise. Treating a P1 as P2 can cost millions. + +**Document as you go.** You won't remember details later. The Scribe role exists for a reason. + +**Blameless post-mortems.** Focus on systems, not individuals. "Why did the system allow this?" not "Who screwed up?" + +**Keep it current.** Stale contacts and outdated procedures create false confidence. + +## Resources + +- [PagerDuty Incident Response Guide](https://response.pagerduty.com/) +- [Google Cloud Incident Response](https://cloud.google.com/docs/security/incident-response) +- [Rekt News](https://rekt.news/) - Learn from others' incidents +- [SEAL 911](https://securityalliance.org/seal-911) - Emergency response coordination + +--- + +*This template is open source. Adapt it to your needs.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/post-mortems/2024-01-15-example-api-outage.mdx b/docs/pages/incident-management/incident-response-template/post-mortems/2024-01-15-example-api-outage.mdx new file mode 100644 index 00000000..af0a8c7e --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/post-mortems/2024-01-15-example-api-outage.mdx @@ -0,0 +1,184 @@ +--- +title: "Post-Mortem: API Outage - Rate Limiter Misconfiguration | SEAL" +description: "This is an example post-mortem. Delete this file or use it as reference when creating real post-mortems." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Post-Mortem: API Outage - Rate Limiter Misconfiguration + + + + +> **This is an example post-mortem.** Delete this file or use it as reference when creating real post-mortems. + +## Metadata + +| Field | Value | +|-------|-------| +| **Incident Date** | 2024-01-15 | +| **Severity** | P2 | +| **Authors** | Alice, Bob | +| **Status** | Final | +| **Incident Log** | [2024-01-15-example-api-outage](../incident-logs/2024-01-15-example-api-outage) | + +--- + +## Summary + +On January 15, 2024, at 14:32 UTC, our API began rejecting most requests with 429 (rate limited) errors. The incident lasted approximately 2 hours and 13 minutes. + +The root cause was a typo in a rate limiter configuration change deployed at 14:15 UTC. The threshold was set to 10 requests per minute instead of the intended 1000. This caused legitimate user requests to be rejected. + +The incident was detected via automated monitoring and resolved by rolling back the configuration change. No funds were at risk, but approximately 3,000 users experienced failed transactions during the outage window. + +--- + +## Impact + +### Users +- Users affected: ~3,000 +- Duration: 2h 13m +- Services unavailable: API (full), Frontend (degraded) + +### Financial +- Funds at risk: None +- Actual losses: None +- Estimated costs: ~2 engineering hours for response + +### Reputation +- Public visibility: Low (brief Discord posts) +- Media coverage: None +- Community response: Minor frustration, resolved quickly + +--- + +## Timeline + +| Time (UTC) | Event | +|------------|-------| +| 14:15 | Config change deployed | +| 14:32 | Incident began (errors started) | +| 14:32 | Detected via DataDog alert | +| 14:38 | Response started | +| 14:42 | Incident Leader assigned | +| 15:05 | Root cause identified | +| 15:25 | Rollback complete | +| 16:45 | Incident resolved | + +See [Incident Log](../incident-logs/2024-01-15-example-api-outage) for detailed timeline. + +--- + +## Root Cause + +### Primary Cause + +Human error during configuration change. The rate limit threshold was typed as `10` instead of `1000`. + +### Contributing Factors + +1. No validation in config deployment process for obviously wrong values +2. Config change was not tested in staging environment +3. No gradual rollout for config changes +4. Alert threshold (5% error rate) took 17 minutes to trigger + +### 5 Whys + +| Question | Answer | +|----------|--------| +| Why did the API reject requests? | Rate limiter threshold was too low | +| Why was the threshold wrong? | Typo in configuration file | +| Why wasn't the typo caught? | No review process for config changes | +| Why is there no review process? | Config changes considered "low risk" | +| Why are they considered low risk? | Never had an incident before (survivorship bias) | + +--- + +## What Went Well + +1. Monitoring detected the issue within 17 minutes +2. Team mobilized quickly once alert fired +3. Root cause identified within 30 minutes +4. Rollback was straightforward +5. Communication to users was timely + +## What Went Wrong + +1. Config change deployed without testing +2. No peer review for config changes +3. Alert took 17 minutes to fire (threshold too high) +4. No validation for obviously wrong values (10 vs 1000) + +## Where We Got Lucky + +1. This happened during business hours when the team was available +2. The fix (rollback) was simple. If the old config was also broken, recovery would have been harder +3. No financial impact because this was the API layer, not smart contracts + +--- + +## Action Items + +| Action | Owner | Deadline | Status | +|--------|-------|----------|--------| +| Add peer review requirement for config changes | Dave | 2024-01-22 | Done | +| Add config validation for rate limit thresholds | Dave | 2024-01-29 | Done | +| Lower alert threshold to 1% error rate | Bob | 2024-01-19 | Done | +| Add staging environment testing for config changes | Dave | 2024-02-15 | In Progress | +| Document config change process | Alice | 2024-01-31 | Not Started | + +--- + +## Lessons for Runbooks + +- [x] Existing runbook sufficient: [Third-Party-Outage](../runbooks/third-party-outage) (config rollback section applicable) +- [ ] No new runbook needed + +--- + +## Detection + +| Aspect | Details | +|--------|---------| +| How detected | Monitoring alert (DataDog) | +| Time to detection | 17 minutes | +| Could we detect faster? | Yes - lower alert threshold to 1% | + +--- + +## Links + +- Incident Log: [2024-01-15-example-api-outage](../incident-logs/2024-01-15-example-api-outage) +- Config PR (bad): [link] +- Rollback PR: [link] +- Monitoring dashboard: [link] + +--- + +## Meeting Notes + +**Attendees:** Alice, Bob, Carol, Dave + +**Discussion points:** +- Agreed config changes need same rigor as code changes +- Discussed whether to require staging for all changes (decided yes) +- Dave to implement validation this week + +--- + +*See [Incident-Response-Policy](../incident-response-policy#step-6-post-incident-review) for post-mortem process.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/post-mortems/overview.mdx b/docs/pages/incident-management/incident-response-template/post-mortems/overview.mdx new file mode 100644 index 00000000..462f738e --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/post-mortems/overview.mdx @@ -0,0 +1,45 @@ +--- +title: "Post-Mortems | Security Alliance" +description: "Store post-mortem documents here. Create one after every significant incident (P1-P3)." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Post-Mortems + + + + +Store post-mortem documents here. Create one after every significant incident (P1-P3). + +## Creating a New Post-Mortem + +1. Copy [Post-Mortem Template](../templates/post-mortem-template) +2. Name it: `YYYY-MM-DD-brief-description.md` +3. Complete before/during the post-mortem meeting + +## Example + +See [2024-01-15-example-api-outage](./2024-01-15-example-api-outage) for a sample filled-out post-mortem. + +## Past Post-Mortems + +{/* Add links to your post-mortems as they're created */} + +--- + +*See [Incident-Response-Policy](../incident-response-policy) for post-mortem guidance.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/roles-and-staffing.mdx b/docs/pages/incident-management/incident-response-template/roles-and-staffing.mdx new file mode 100644 index 00000000..229fb303 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/roles-and-staffing.mdx @@ -0,0 +1,219 @@ +--- +title: "Roles and Staffing | Security Alliance" +description: "How you staff incident response depends on your team size and structure. This document covers options from small teams to larger organizations." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../components' + + + + +# Roles and Staffing + + + + +How you staff incident response depends on your team size and structure. This document covers options from small teams to larger organizations. + +See [Incident-Response-Policy](./incident-response-policy#roles) for role definitions. + +--- + +## Core Roles (All Team Sizes) + +Every incident needs these roles filled, even if one person wears multiple hats: + +| Role | Responsibility | +|------|----------------| +| **Incident Leader** | Coordinates response, assigns tasks, makes decisions | +| **Scribe** | Documents everything in [Incident Log](./templates/incident-log-template) | +| **Responders** | Execute fixes, investigate, implement mitigations | + +For larger incidents, add: +- **Communication Manager** - Handles internal/external comms +- **Subject Matter Experts** - Specialists for specific domains + +--- + +## Small Teams (2-5 people) + +### Approach + +Everyone knows everything. Just make sure someone is always reachable. + +> Whether you need formal on-call depends more on your commitments (SLAs, assets held, user expectations) than team size alone. Small teams with high-value assets may still need structured coverage. + +### Structure + +- Designate 1-2 people as default Incident Leaders (only one leads any given incident) +- Everyone else responds as needed +- Leader also serves as Scribe for minor incidents +- Separate Scribe for P1/P2 incidents + +### Expectations + +- Keep a shared contact list (see [Contacts](./contacts)) +- Establish one communication channel for incidents +- Someone should always be reachable (informal coverage) + +### What You Might Not Need + +- Formal on-call rotation (unless your commitments require it) +- Separate First Responder program +- Multiple communication managers + +--- + +## Medium Teams (5-15 people) + +### Approach + +Define subject matter experts. Consider a simple on-call rotation. + +### Structure + +**Subject Matter Experts (SMEs)** + +| Domain | Primary | Backup | +|--------|---------|--------| +| Smart Contracts | | | +| Infrastructure | | | +| Frontend | | | +| Security | | | + +**On-Call Options** + +*Option A: Informal* +- No formal schedule, but SMEs are expected to be reachable during their working hours +- Clear escalation for after-hours: who to call first + +*Option B: Simple Rotation* +- Weekly rotation among willing team members +- One person on-call, responsible for initial triage +- They pull in SMEs as needed + +### Expectations + +- SMEs respond quickly when paged for their domain +- On-call person handles initial assessment and escalation +- Separate Scribe and Incident Leader for P1/P2 incidents + +--- + +## Larger Teams (15+ people) + +### Approach + +Formal First Responder program with trained personnel and scheduled on-call. + +### First Responder Program + +**What First Responders Do:** +- Initial triage when an incident is detected +- Assess severity +- Kick off the incident response process +- Pull in the right people +- Hand off to Incident Leader + +**What First Responders Don't Do:** +- Fix the issue themselves (unless they're also the SME) +- Make major decisions without escalation + +**Why This Model:** +- Distributes knowledge across the organization +- Reduces burden on any single team +- Ensures someone is always ready to start the process +- Doesn't require deep expertise in all domains + +### On-Call Structure + +Consider parallel schedules for different domains: + +| Schedule | Coverage | Rotation | +|----------|----------|----------| +| Infrastructure | 24/7 | Weekly among 6-8 people | +| Smart Contracts | 24/7 | Weekly among 6-8 people | + +ex. With 8 people per rotation, each person is on-call one week every two months. + +### First Responder Training + +Before going on-call, complete: +- [ ] Review [Incident-Response-Policy](./incident-response-policy) +- [ ] Review [Incident Log](./templates/incident-log-template) and [Post-Mortem](./templates/post-mortem-template) templates +- [ ] Read 2-3 past post-mortems +- [ ] Understand basic architecture (infra and smart contracts) +- [ ] Know how to reach SMEs and Decision Makers +- [ ] Test alerting system access + +### On-Call Expectations + +**During your shift:** +- Keep alerting device accessible +- Respond to pages within 15 minutes +- Triage and escalate appropriately +- You don't need to fix everything. Get the right people involved + +**Off-shift:** +- Stay current on documentation +- Review new post-mortems +- Participate in tabletop exercises + +--- + +## Decision Makers + +Regardless of team size, define who can make high-stakes decisions during P1 incidents: + +| Role | Name | Contact | +|------|------|---------| +| | | | +| | | | +| | | | + +These people should be reachable 24/7 for critical incidents. Consider: +- Founders / C-level +- Security Lead +- Engineering Lead +- Legal (for incidents with legal implications) + +--- + +## Tools Checklist + +Ensure your on-call personnel have access to: + +- [ ] Alerting system (PagerDuty, etc.) +- [ ] Communication platform (Slack, Discord, etc.) +- [ ] Video conferencing +- [ ] Monitoring dashboards +- [ ] On-call schedule +- [ ] [Contacts](./contacts) list + +--- + +## Choosing Your Model + +| Team Size | Recommended Approach | +|-----------|---------------------| +| 2-5 | Informal coverage, designated leaders | +| 5-10 | SME structure, optional simple rotation | +| 10-15 | Simple rotation with SMEs | +| 15+ | First Responder program with parallel schedules | + +**Start simple and add structure as you grow.** A lightweight process that people follow beats a heavyweight process that gets ignored. + +--- + +*See [Incident-Response-Policy](./incident-response-policy) for how these roles work during an actual incident.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/build-pipeline-compromise.mdx b/docs/pages/incident-management/incident-response-template/runbooks/build-pipeline-compromise.mdx new file mode 100644 index 00000000..694195c9 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/build-pipeline-compromise.mdx @@ -0,0 +1,90 @@ +--- +title: "Runbook: Build Pipeline Compromise | Security Alliance" +description: "Stub runbook. Customize with your CI/CD platform and procedures." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: Build Pipeline Compromise + + + + +> **Stub runbook.** Customize with your CI/CD platform and procedures. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P1 | +| **Primary Responder** | DevOps / Infrastructure SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Unexpected code in deployed artifacts +- [ ] CI/CD configuration changed without approval +- [ ] Secrets accessed or exfiltrated +- [ ] Unauthorized workflow runs + +### Confirm Compromise + +- Review CI/CD audit logs +- Compare build artifacts to source +- Check for config changes in CI/CD platform + +--- + +## Immediate Actions + +1. [ ] Disable compromised pipelines +2. [ ] Rotate all secrets and tokens +3. [ ] Take down potentially compromised deployments +4. [ ] Audit recent builds and deployments + +--- + +## Mitigation + +1. [ ] Audit CI/CD configuration for unauthorized changes +2. [ ] Rebuild from trusted commit using clean pipeline +3. [ ] Implement additional approval requirements +4. [ ] Review and restrict pipeline permissions + +--- + +## Prevention + +- [ ] Require approval for CI/CD config changes +- [ ] Use short-lived credentials +- [ ] Implement branch protection +- [ ] Audit pipeline access regularly +- [ ] Use signed commits +- [ ] Separate build and deploy permissions + +--- + +## Related + +- [Frontend-Compromise](./frontend-compromise) +- [Dependency-Attack](./dependency-attack) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/cdn-hosting-compromise.mdx b/docs/pages/incident-management/incident-response-template/runbooks/cdn-hosting-compromise.mdx new file mode 100644 index 00000000..864bb846 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/cdn-hosting-compromise.mdx @@ -0,0 +1,86 @@ +--- +title: "Runbook: CDN/Hosting Compromise | Security Alliance" +description: "Stub runbook. Customize with your CDN and hosting provider details." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: CDN/Hosting Compromise + + + + +> **Stub runbook.** Customize with your CDN and hosting provider details. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P1 | +| **Primary Responder** | Infrastructure SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Malicious files being served +- [ ] File hashes don't match expected +- [ ] Unauthorized access in provider logs + +### Confirm Compromise + +- Compare served files to known good source +- Check CDN/hosting access logs + +--- + +## Immediate Actions + +1. [ ] Purge CDN cache +2. [ ] Take down site or redirect to maintenance page +3. [ ] Rotate all access credentials +4. [ ] Review access logs for unauthorized activity + +--- + +## Mitigation + +1. [ ] Redeploy from verified source (git, not existing infra) +2. [ ] Verify deployment matches expected +3. [ ] Enable additional access controls +4. [ ] Set up file integrity monitoring + +--- + +## Prevention + +- [ ] Limit hosting/CDN admin access +- [ ] Enable 2FA on all accounts +- [ ] Use subresource integrity (SRI) +- [ ] Implement Content Security Policy (CSP) +- [ ] Regular access audits + +--- + +## Related + +- [Frontend-Compromise](./frontend-compromise) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/ddos-attack.mdx b/docs/pages/incident-management/incident-response-template/runbooks/ddos-attack.mdx new file mode 100644 index 00000000..e9e694aa --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/ddos-attack.mdx @@ -0,0 +1,195 @@ +--- +title: "Runbook: DDoS Attack | Security Alliance" +description: "This is an example runbook. Review and customize for your protocol before use. Add your specific CDN/WAF provider commands and escalation contacts." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: DDoS Attack + + + + +> **This is an example runbook.** Review and customize for your protocol before use. Add your specific CDN/WAF provider commands and escalation contacts. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P2-P3 | +| **Primary Responder** | Infrastructure SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Website/API unresponsive or slow +- [ ] Monitoring shows traffic spike +- [ ] CDN/hosting alerts +- [ ] Error rate increase +- [ ] Legitimate requests timing out + +### Differentiation + +| Symptom | Likely Cause | +|---------|--------------| +| Traffic spike + slow response | DDoS | +| Traffic normal + slow response | Application issue | +| Single endpoint affected | Targeted attack or bug | +| All traffic from few IPs | Simple attack, easy to block | +| Distributed traffic | Sophisticated DDoS | + +--- + +## Immediate Actions + +### Step 1: Confirm DDoS + +**Why:** Distinguish from application issues + +- [ ] Check CDN/WAF dashboards +- [ ] Review traffic patterns +- [ ] Check if specific endpoints targeted + +### Step 2: Enable DDoS Protection + +**Why:** Use provider-level mitigation + +For Cloudflare: +``` +[Document your Cloudflare mitigation steps] +``` + +For AWS: +``` +[Document your AWS Shield steps] +``` + +### Step 3: Assess Impact + +- [ ] Which services affected? +- [ ] Are critical functions available? +- [ ] User impact level? + +--- + +## Investigation + +### Key Questions + +- [ ] Attack type (volumetric, protocol, application layer)? +- [ ] Targeted endpoints? +- [ ] Attack source patterns? +- [ ] Why now? (retaliation, extortion, distraction?) + +### Data to Collect + +| Data | Source | +|------|--------| +| Traffic volume | CDN/hosting dashboard | +| Request patterns | Access logs | +| Source IPs | WAF/firewall logs | +| Targeted paths | Application logs | + +--- + +## Mitigation + +### CDN/WAF Level + +1. Enable "Under Attack" mode if available +2. Increase security level +3. Add rate limiting rules +4. Block obvious attack patterns +5. Enable bot protection + +### Application Level + +1. Enable rate limiting +2. Add CAPTCHA to affected endpoints +3. Cache aggressively +4. Disable non-essential features temporarily + +### DNS Level + +1. Reduce TTL for flexibility +2. Consider geo-blocking if attack is regional +3. Failover to backup if available + +--- + +## Escalation + +- [ ] [CDN/hosting provider](../contacts#infrastructure-vendors) - for attack mitigation support +- [ ] [Decision Makers](../contacts#decision-makers) - if extended outage + +--- + +## Recovery + +### Attack Subsiding + +1. Monitor traffic patterns +2. Gradually reduce protection levels +3. Watch for resurgence +4. Keep enhanced monitoring for 24-48 hours + +### Post-Attack + +- [ ] Review logs for attack details +- [ ] Update protection rules +- [ ] Document attack patterns +- [ ] Consider permanent mitigations + +--- + +## Communication + +### Internal + +- Status updates every 30 min during active attack +- Clear when degraded vs. fully down + +### External (if needed) + +> We're experiencing elevated traffic that's affecting site performance. We're actively mitigating and will provide updates. + +Usually not needed for brief DDoS attacks. Avoid drawing attention. + +--- + +## Prevention + +After resolving, consider: + +- [ ] Always-on DDoS protection +- [ ] Rate limiting by default +- [ ] Geographic restrictions if appropriate +- [ ] Improved caching +- [ ] Backup/failover infrastructure + +--- + +## Related + +- [Third-Party-Outage](./third-party-outage) +- [Frontend-Compromise](./frontend-compromise) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/dependency-attack.mdx b/docs/pages/incident-management/incident-response-template/runbooks/dependency-attack.mdx new file mode 100644 index 00000000..952c10a4 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/dependency-attack.mdx @@ -0,0 +1,94 @@ +--- +title: "Runbook: Dependency Attack | Security Alliance" +description: "Stub runbook. Customize with your package management and build procedures." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: Dependency Attack + + + + +> **Stub runbook.** Customize with your package management and build procedures. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P1 | +| **Primary Responder** | Frontend SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Unexpected behavior after dependency update +- [ ] Security advisory for a package you use +- [ ] Malicious code found in node_modules or similar +- [ ] Lockfile changes you didn't make + +### Confirm Dependency Attack + +``` +npm audit +# or +yarn audit +``` + +Check for recent lockfile changes in git history. + +--- + +## Immediate Actions + +1. [ ] Take down site to stop serving malicious code +2. [ ] Identify the malicious package +3. [ ] Pin dependencies to last known good version +4. [ ] Rebuild from clean environment + +--- + +## Mitigation + +1. [ ] Remove or replace malicious package +2. [ ] Update lockfile with known good versions +3. [ ] Rebuild using `npm ci` or `yarn --frozen-lockfile` +4. [ ] Redeploy verified build + +--- + +## Prevention + +- [ ] Use lockfiles and commit them +- [ ] Use `npm ci` / `yarn --frozen-lockfile` in CI +- [ ] Regularly audit dependencies +- [ ] Consider using a private registry +- [ ] Pin exact versions for critical packages +- [ ] Review dependency changes in PRs + +--- + +## Related + +- [Frontend-Compromise](./frontend-compromise) +- [Build-Pipeline-Compromise](./build-pipeline-compromise) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/dns-hijack.mdx b/docs/pages/incident-management/incident-response-template/runbooks/dns-hijack.mdx new file mode 100644 index 00000000..e2bb16a2 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/dns-hijack.mdx @@ -0,0 +1,85 @@ +--- +title: "Runbook: DNS Hijack | Security Alliance" +description: "Stub runbook. Customize with your DNS provider details and procedures." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: DNS Hijack + + + + +> **Stub runbook.** Customize with your DNS provider details and procedures. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P1 | +| **Primary Responder** | Infrastructure SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Domain pointing to wrong IP +- [ ] Users redirected to malicious site +- [ ] SSL certificate errors (attacker using different cert) + +### Confirm DNS Hijack + +``` +dig yourdomain.com +# Compare output to expected IP +``` + +--- + +## Immediate Actions + +1. [ ] Regain access to DNS provider account +2. [ ] Enable 2FA if not already enabled +3. [ ] Point DNS to known good infrastructure or maintenance page +4. [ ] Enable DNS lock / registrar lock + +--- + +## Mitigation + +[Document your specific DNS provider procedures here] + +--- + +## Prevention + +- [ ] Enable registrar lock +- [ ] Use DNSSEC +- [ ] Enable 2FA on DNS provider +- [ ] Limit DNS admin access +- [ ] Monitor DNS records for changes + +--- + +## Related + +- [Frontend-Compromise](./frontend-compromise) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/frontend-compromise.mdx b/docs/pages/incident-management/incident-response-template/runbooks/frontend-compromise.mdx new file mode 100644 index 00000000..938c7aaa --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/frontend-compromise.mdx @@ -0,0 +1,200 @@ +--- +title: "Runbook: Frontend Compromise | Security Alliance" +description: "This is an example runbook. Review and customize for your protocol before use. Add your specific DNS provider, hosting platform, and deployment procedures." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: Frontend Compromise + + + + +> **This is an example runbook.** Review and customize for your protocol before use. Add your specific DNS provider, hosting platform, and deployment procedures. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P1 | +| **Primary Responder** | Frontend SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Users report unexpected transaction requests +- [ ] UI behaving differently than expected +- [ ] Wallet drainer detected +- [ ] Injected scripts in page source +- [ ] DNS/domain issues +- [ ] Community reports of phishing via official domain + +### Attack Vectors + +| Vector | Signs | +|--------|-------| +| DNS hijack | Domain pointing to wrong IP | +| CDN compromise | Malicious files served | +| Dependency attack | npm/yarn package compromised | +| Build pipeline | CI/CD compromise | +| Hosting compromise | Files modified on server | + +--- + +## Immediate Actions + +### Step 1: Take Down or Redirect + +**Why:** Stop users from interacting with compromised site + +Options: +- [ ] Point DNS to maintenance page +- [ ] Disable CDN distribution +- [ ] Remove site from hosting +- [ ] Add banner/warning if partial control + +### Step 2: Warn Users + +**Why:** Prevent further damage + +- [ ] Post on Twitter/X +- [ ] Post in Discord/Telegram +- [ ] Update status page + +Message template: +> Our website may be compromised. Do NOT interact with [domain] or approve any transactions until further notice. Your funds in the protocol are safe if you don't sign new transactions. + +### Step 3: Assess Scope + +- [ ] What was changed? +- [ ] How long was it compromised? +- [ ] How many users potentially affected? +- [ ] Were any transactions signed? + +--- + +## Investigation + +### Priority: Identify Attack Vector + +The first priority is understanding how the attacker gained access so you can close that vector: + +- [ ] **How did attacker gain access?** (DNS, CDN, dependencies, CI/CD, hosting) + +Once identified, go to the specific runbook: +- DNS hijack → [DNS-Hijack](./dns-hijack) +- CDN/hosting compromise → [CDN-Hosting-Compromise](./cdn-hosting-compromise) +- Dependency attack → [Dependency-Attack](./dependency-attack) +- Build pipeline compromise → [Build-Pipeline-Compromise](./build-pipeline-compromise) + +### Post-Mitigation Investigation + +After the threat is contained, investigate impact: + +- [ ] What was injected/changed? +- [ ] Which users interacted during compromise window? +- [ ] What were users tricked into signing? + +### Check These + +| Component | How to Check | +|-----------|--------------| +| DNS records | `dig` or DNS provider console | +| CDN files | Compare to known good | +| Build artifacts | Check CI/CD logs | +| Dependencies | `npm audit`, lockfile changes | +| Hosting files | Compare to repo | + +--- + +## Mitigation + +See the specific runbook for detailed mitigation steps: + +- [DNS-Hijack](./dns-hijack) +- [CDN-Hosting-Compromise](./cdn-hosting-compromise) +- [Dependency-Attack](./dependency-attack) +- [Build-Pipeline-Compromise](./build-pipeline-compromise) + +--- + +## Recovery + +### Before Restoring Service + +- [ ] Root cause identified +- [ ] Vulnerability fixed +- [ ] Fresh deployment from verified source +- [ ] All credentials rotated +- [ ] Additional security measures in place + +### Restoring Service + +1. Deploy verified build +2. Verify deployment matches expected +3. Test critical user flows +4. Monitor for anomalies +5. Announce service restored + +--- + +## Affected User Support + +If users signed malicious transactions: + +- [ ] Compile list of affected addresses (from chain data) +- [ ] Provide guidance on revoking approvals +- [ ] Consider compensation if protocol at fault +- [ ] Document for post-mortem + +--- + +## Escalation + +- [ ] [Decision Makers](../contacts#decision-makers) - immediately +- [ ] [Infrastructure Vendors](../contacts#infrastructure-vendors) - if hosting/CDN involved +- [ ] [Legal & Communications](../contacts#legal--communications) - for user communication + +--- + +## Prevention Checklist + +After resolving, review: + +- [ ] DNS provider security (2FA, lock) +- [ ] Hosting access controls +- [ ] CI/CD security +- [ ] Dependency management +- [ ] Subresource integrity +- [ ] Content Security Policy + +--- + +## Related + +- [DNS-Hijack](./dns-hijack) +- [CDN-Hosting-Compromise](./cdn-hosting-compromise) +- [Dependency-Attack](./dependency-attack) +- [Build-Pipeline-Compromise](./build-pipeline-compromise) +- [DDoS-Attack](./ddos-attack) +- [Third-Party-Outage](./third-party-outage) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/key-compromise.mdx b/docs/pages/incident-management/incident-response-template/runbooks/key-compromise.mdx new file mode 100644 index 00000000..16eecdd9 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/key-compromise.mdx @@ -0,0 +1,174 @@ +--- +title: "Runbook: Key Compromise | Security Alliance" +description: "This is an example runbook. Review and customize for your protocol before use. Fill in your key types, what each controls, and your specific rotation procedu..." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: Key Compromise + + + + +> **This is an example runbook.** Review and customize for your protocol before use. Fill in your key types, what each controls, and your specific rotation procedures. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P1 | +| **Primary Responder** | Security SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Unauthorized transactions from controlled addresses +- [ ] Unexpected multisig proposals +- [ ] Key holder reports compromise +- [ ] Suspicious signing activity +- [ ] Social engineering attempt succeeded + +### Key Types + +| Key Type | Risk Level | Controlled Assets | +|----------|------------|-------------------| +| Deployer | [High/Med] | [What it controls] | +| Admin/Owner | [High/Med] | [What it controls] | +| Multisig signer | [High/Med] | [What it controls] | +| Hot wallet | [High/Med] | [What it controls] | + +--- + +## Immediate Actions + +### Step 1: Confirm Compromise + +**Why:** Avoid false positives, but err on side of caution + +- [ ] Verify with key holder directly (not via potentially compromised channels) +- [ ] Check for unauthorized transactions +- [ ] Review recent signing activity + +### Step 2: Assess Blast Radius + +**Why:** Understand what the attacker can do + +- [ ] What can this key sign? +- [ ] Are there timelocks? +- [ ] What other systems use this key? + +### Step 3: Revoke/Rotate + +**Why:** Remove attacker access + +For multisig signers: +- [ ] Remove compromised signer +- [ ] Add replacement signer + +For admin keys: +- [ ] Transfer ownership to new address +- [ ] Or revoke permissions if possible + +For hot wallets: +- [ ] Move remaining funds to secure address + +### Step 4: Check for Damage + +**Why:** Understand if attacker acted + +- [ ] Review all transactions from compromised key +- [ ] Check pending proposals/timelocks +- [ ] Audit any systems key had access to + +--- + +## Investigation + +### Key Questions + +- [ ] How was the key compromised? (phishing, malware, insider, leaked) +- [ ] When was it compromised? +- [ ] What did the attacker do (if anything)? +- [ ] Are other keys at risk? + +### Evidence to Collect + +| Data | Source | +|------|--------| +| Transaction history | Block explorer | +| Login/access logs | Infrastructure providers | +| Signing requests | Multisig interface | +| Communications | If social engineering | + +--- + +## Mitigation by Key Type + +### Multisig Signer + +1. Propose removal of compromised signer +2. Collect required signatures +3. Execute removal +4. Add new signer +5. Verify new threshold is appropriate + +### Contract Admin Key + +1. Prepare ownership transfer transaction +2. Execute transfer to secure address +3. Verify new owner +4. Update documentation + +### Hot Wallet + +1. Prepare transaction to move all funds +2. Move to cold storage or new hot wallet +3. Update any systems using old address +4. Retire compromised address + +--- + +## Escalation + +- [ ] [Decision Makers](../contacts#decision-makers) - immediately for any confirmed compromise +- [ ] [Security Partners](../contacts#security-partners) - for investigation support +- [ ] [Legal](../contacts#legal--communications) - if funds were stolen + +--- + +## Prevention Checklist + +After resolving, review: + +- [ ] Key storage practices +- [ ] Phishing awareness training +- [ ] Hardware wallet usage +- [ ] Multisig thresholds +- [ ] Access logging +- [ ] Key rotation schedule + +--- + +## Related + +- [Smart-Contract-Exploit](./smart-contract-exploit) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/overview.mdx b/docs/pages/incident-management/incident-response-template/runbooks/overview.mdx new file mode 100644 index 00000000..bbd21ae7 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/overview.mdx @@ -0,0 +1,70 @@ +--- +title: "Runbooks | Security Alliance" +description: "Step-by-step guides for specific incident types. Use these during active incidents to reduce cognitive load and ensure consistent response." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbooks + + + + +Step-by-step guides for specific incident types. Use these during active incidents to reduce cognitive load and ensure consistent response. + +> **These runbooks are examples and starting points.** They contain generic guidance that must be adapted to your specific protocol, infrastructure, and team. Review each runbook carefully and customize the commands, contacts, and procedures before relying on them during an actual incident. Untested runbooks can be worse than no runbook at all. + +## Available Runbooks + +### Critical (P1) + +- [Smart-Contract-Exploit](./smart-contract-exploit) - Active exploit or critical vulnerability +- [Key-Compromise](./key-compromise) - Private key or signer compromise +- [Frontend-Compromise](./frontend-compromise) - Website/UI compromise (routes to specific runbooks below) + - [DNS-Hijack](./dns-hijack) - Domain/DNS compromise + - [CDN-Hosting-Compromise](./cdn-hosting-compromise) - CDN or hosting provider compromise + - [Dependency-Attack](./dependency-attack) - npm/package supply chain attack + - [Build-Pipeline-Compromise](./build-pipeline-compromise) - CI/CD compromise + +### High/Moderate (P2-P3) + +- [DDoS-Attack](./ddos-attack) - Denial of service attacks +- [Third-Party-Outage](./third-party-outage) - External provider issues + +## Creating New Runbooks + +Use [_Runbook-Template](./runbook-template) as your starting point. + +Good runbooks: +- Are concise. Responders need quick answers +- Include actual commands and links +- Get tested in tabletop exercises +- Get updated after real incidents + +## Suggested Runbooks to Add + +Consider creating runbooks for: + +- [ ] Oracle manipulation +- [ ] Governance attack +- [ ] SSL certificate issues +- [ ] Deployment failure/rollback +- [ ] Data inconsistency + +--- + +*See [Incident-Response-Policy](../incident-response-policy) for the overall response process.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/runbook-template.mdx b/docs/pages/incident-management/incident-response-template/runbooks/runbook-template.mdx new file mode 100644 index 00000000..ee58c462 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/runbook-template.mdx @@ -0,0 +1,154 @@ +--- +title: "Runbook Template | Security Alliance" +description: "Copy this template to create runbooks for specific incident types." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook Template + + + + +Copy this template to create runbooks for specific incident types. + +--- + +# Runbook: [INCIDENT TYPE] + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P1 / P2 / P3 | +| **Primary Responder** | [Team/Role] | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Symptom 1 +- [ ] Symptom 2 +- [ ] Symptom 3 + +### Alerts + +- Alert: [name] in [system] +- Dashboard: [link] + +### Differentiation + +If you see [X] but not [Y], it might be [different issue] instead. + +--- + +## Immediate Actions + +### Step 1: [Name] + +**Why:** [Purpose] + +``` +[Commands or steps] +``` + +**Expected:** [Result] + +### Step 2: [Name] + +**Why:** [Purpose] + +``` +[Commands or steps] +``` + +--- + +## Investigation + +### Key Questions + +- [ ] What is the scope? +- [ ] When did it start? +- [ ] What changed recently? + +### Information to Gather + +| Data | How to Get It | +|------|---------------| +| | | + +--- + +## Mitigation + +### Option A: [Name] - Preferred + +**When:** [Conditions] +**Impact:** [Side effects] + +1. Step 1 +2. Step 2 + +**Verify:** +``` +[Command to verify] +``` + +### Option B: [Name] - Fallback + +**When:** [Conditions] + +1. Step 1 +2. Step 2 + +--- + +## Escalation + +Escalate to [Contacts](../contacts) if: +- [ ] Mitigation doesn't work within [time] +- [ ] Impact expands +- [ ] [Condition] + +--- + +## Resolution Checklist + +- [ ] Mitigation verified +- [ ] Stakeholders notified +- [ ] Timeline documented in [Incident Log](../templates/incident-log-template) +- [ ] Post-mortem scheduled if warranted + +--- + +## Common Root Causes + +| Cause | Signs | Fix | +|-------|-------|-----| +| | | | + +--- + +## Related + +- [Other-Runbook](#TODO) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/smart-contract-exploit.mdx b/docs/pages/incident-management/incident-response-template/runbooks/smart-contract-exploit.mdx new file mode 100644 index 00000000..4a36704d --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/smart-contract-exploit.mdx @@ -0,0 +1,185 @@ +--- +title: "Runbook: Smart Contract Exploit | Security Alliance" +description: "This is an example runbook. Review and customize for your protocol before use. Fill in the placeholder commands, add your specific contracts and addresses, a..." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: Smart Contract Exploit + + + + +> **This is an example runbook.** Review and customize for your protocol before use. Fill in the placeholder commands, add your specific contracts and addresses, and test during a tabletop exercise. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P1 | +| **Primary Responder** | Smart Contract SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Unexpected fund movements +- [ ] On-chain monitoring alerts +- [ ] Unusual transaction patterns +- [ ] Community reports of lost funds +- [ ] Abnormal contract state changes + +### Alerts + +- [Your on-chain monitoring tool] +- [Blockchain explorer alerts] + +### Differentiation + +- If funds moving to known addresses (multisig, treasury) → likely authorized +- If single user affected → might be phishing, not protocol exploit +- If UI shows wrong values but chain is correct → frontend issue + +--- + +## Immediate Actions + +### Step 1: Assess Scope + +**Why:** Understand what's at risk before acting + +- [ ] Check total funds at risk +- [ ] Identify affected contracts +- [ ] Determine if exploit is ongoing + +### Step 2: Contact SEAL 911 + +**Why:** Get immediate expert support for active exploits + +See [Contacts - SEAL 911](../contacts#seal-911) + +### Step 3: Pause if Possible + +**Why:** Stop the bleeding + +If your contracts have pause functionality: +``` +[Document your pause procedure here] +``` + +**If no pause function:** Consider other containment options (front-running, rescue operations). + +### Step 4: Preserve Evidence + +**Why:** Needed for investigation and potential recovery + +- [ ] Screenshot/save relevant transactions +- [ ] Note block numbers +- [ ] Save mempool data if available +- [ ] Document attacker addresses + +--- + +## Investigation + +### Key Questions + +- [ ] What vulnerability was exploited? +- [ ] How much was taken/at risk? +- [ ] Is the exploit still possible? +- [ ] Are other contracts affected? +- [ ] Was this reported via bug bounty first? + +### Information to Gather + +| Data | Source | +|------|--------| +| Exploit transactions | Block explorer | +| Attacker addresses | On-chain analysis | +| Vulnerability details | Code review | +| Affected user count | Contract state | + +--- + +## Mitigation + +### Option A: Pause Contracts + +**When:** Pause functionality exists +**Impact:** Users cannot interact until unpaused + +1. Execute pause function +2. Verify pause is effective +3. Communicate to users + +### Option B: Emergency Upgrade + +**When:** Upgrade mechanism exists +**Impact:** Requires governance or multisig action + +1. Prepare patched contract +2. Execute upgrade +3. Verify fix + +### Option C: Rescue Operation + +**When:** Funds can be moved to safety before attacker +**Impact:** Complex, requires coordination + +1. Coordinate with security researchers +2. Prepare rescue transactions +3. Execute with proper sequencing + +--- + +## Escalation + +Contact immediately: +- [ ] [Decision Makers](../contacts#decision-makers) +- [ ] [Security Partners](../contacts#security-partners) +- [ ] [Legal](../contacts#legal--communications) if significant fund loss + +--- + +## Post-Exploit + +- [ ] Determine fix approach +- [ ] Plan for affected users (if any) +- [ ] Prepare public communication +- [ ] Consider disclosure timeline +- [ ] Engage with law enforcement if appropriate + +--- + +## Common Root Causes + +| Cause | Signs | Prevention | +|-------|-------|------------| +| Reentrancy | Recursive calls in tx | CEI pattern, reentrancy guards | +| Access control | Unauthorized caller | Proper modifiers | +| Oracle manipulation | Price deviation before exploit | TWAP, multiple sources | + +--- + +## Related + +- [Key-Compromise](./key-compromise) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/runbooks/third-party-outage.mdx b/docs/pages/incident-management/incident-response-template/runbooks/third-party-outage.mdx new file mode 100644 index 00000000..584cf998 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/runbooks/third-party-outage.mdx @@ -0,0 +1,196 @@ +--- +title: "Runbook: Third-Party Outage | Security Alliance" +description: "This is an example runbook. Review and customize for your protocol before use. Fill in your specific vendors, dependencies, and failover procedures." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Runbook: Third-Party Outage + + + + +> **This is an example runbook.** Review and customize for your protocol before use. Fill in your specific vendors, dependencies, and failover procedures. + +## Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | P2-P3 | +| **Primary Responder** | Infrastructure SME | +| **Last Updated** | [Date] | +| **Owner** | [Name] | + +--- + +## Identification + +### Symptoms + +- [ ] Specific functionality broken +- [ ] Errors referencing external service +- [ ] Status page shows provider outage +- [ ] Timeouts to external endpoints + +### Common Dependencies + +| Service Type | Provider(s) | Criticality | Fallback? | +|--------------|-------------|-------------|-----------| +| RPC | | Critical / High / Med | Yes / No | +| Hosting | | | | +| CDN | | | | +| DNS | | | | +| Monitoring | | | | +| Auth | | | | +| Oracle | | | | + +--- + +## Immediate Actions + +### Step 1: Confirm Third-Party Issue + +**Why:** Verify it's not your own infrastructure + +- [ ] Check provider status page +- [ ] Test provider directly +- [ ] Check if other users reporting (Twitter, Discord) + +### Step 2: Assess Your Impact + +- [ ] Which of your services affected? +- [ ] What functionality is degraded/broken? +- [ ] User-facing impact? + +### Step 3: Enable Fallback (if available) + +- [ ] Switch to backup provider +- [ ] Enable cached/degraded mode +- [ ] Disable affected features gracefully + +--- + +## By Service Type + +### RPC Provider Outage + +**Fallback options:** +1. Switch to secondary RPC +2. Use public endpoints temporarily +3. Run own node if available + +``` +[Document your RPC failover procedure] +``` + +### Hosting/CDN Outage + +**Fallback options:** +1. Failover to secondary hosting +2. Enable maintenance page +3. Redirect to status page + +### Oracle Outage + +**Impact:** Price feeds may be stale or unavailable + +**Options:** +1. Pause affected operations + +> Customize this section for your specific oracle dependencies and what operations depend on them. + +--- + +## Communication + +### Internal + +- Update team on status +- ETA if known +- What's affected + +### External + +If significant user impact: +> We're experiencing issues with [affected feature] due to a third-party service outage. We're monitoring and will restore service when the provider recovers. + +Include: +- What's affected +- What still works +- Where to check for updates + +--- + +## Monitoring the Outage + +- [ ] Subscribe to provider status updates +- [ ] Set reminder to check every 15-30 min +- [ ] Monitor your own metrics for recovery + +--- + +## Recovery + +### When Provider Recovers + +1. Verify your services are working +2. Check for any data inconsistencies +3. Monitor for stability +4. Switch back from fallback if used + +### Post-Outage + +- [ ] Document timeline +- [ ] Review if fallback worked +- [ ] Assess if additional redundancy needed +- [ ] Consider SLA review with provider + +--- + +## Escalation + +- [ ] [Infrastructure Vendors](../contacts#infrastructure-vendors) - contact provider support +- [ ] [Decision Makers](../contacts#decision-makers) - if extended outage or no ETA + +--- + +## Prevention + +Reduce third-party dependency risk: + +- [ ] Multiple RPC providers +- [ ] Multi-region hosting +- [ ] CDN with failover +- [ ] Graceful degradation +- [ ] Cached fallbacks where appropriate +- [ ] Regular dependency review + +--- + +## Provider Contacts + +| Provider | Status Page | Support Contact | +|----------|-------------|-----------------| +| | | | +| | | | + +--- + +## Related + +- [DDoS-Attack](./ddos-attack) +- [Incident-Response-Policy](../incident-response-policy) + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/templates/incident-log-template.mdx b/docs/pages/incident-management/incident-response-template/templates/incident-log-template.mdx new file mode 100644 index 00000000..22aee51f --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/templates/incident-log-template.mdx @@ -0,0 +1,161 @@ +--- +title: "Incident Log Template | Security Alliance" +description: "Use this template during active incidents. The Scribe owns this document." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Incident Log Template + + + + +Use this template during active incidents. The [Scribe](../incident-response-policy#scribe) owns this document. + +## How to Use + +1. Copy this template to [Incident-Logs](../incident-logs/overview) folder +2. Name it: `YYYY-MM-DD-brief-description` (e.g., `2024-03-15-api-outage`) +3. Update in real-time as things happen +4. Use UTC timestamps, 24-hour format +5. More detail is better. You can summarize later + +--- + +# Incident: [TITLE] + +## Summary + +| Field | Value | +|-------|-------| +| **Status** | Active / Mitigated / Resolved | +| **Severity** | P1 / P2 / P3 / P4 / P5 | +| **Start Time** | YYYY-MM-DD HH:MM UTC | +| **Resolution Time** | | +| **Affected Services** | | + +## Roles + +| Role | Person | +|------|--------| +| Detector | | +| Incident Leader | | +| Scribe | | +| Communication Manager | | +| Responders | | + +## Communication Channels + +- **Call:** [link] +- **Chat:** [channel] + +--- + +## Timeline + +``` +HH:MM UTC - Incident detected by [who/what] +HH:MM UTC - [Person] assigned as Incident Leader +HH:MM UTC - [Person] assigned as Scribe +HH:MM UTC - Initial assessment: [description] +HH:MM UTC - ... +``` + +--- + +## Investigation + +### What We Know + +- +- +- + +### Affected Services + +| Service | Impact | Status | +|---------|--------|--------| +| | | | + +### Root Cause (initial assessment) + + + +--- + +## Actions + +### Immediate + +- [ ] [Action] @[Owner] +- [ ] [Action] @[Owner] + +### Resolution + +- [ ] [Action] @[Owner] +- [ ] [Action] @[Owner] + +--- + +## Resolution Summary + +### Mitigation Applied + + + +### Verification + +- [ ] [Check 1] +- [ ] [Check 2] + +--- + +## Communications Sent + +| Time | Channel | Summary | +|------|---------|---------| +| | | | + +--- + +## Post-Incident + +- [ ] Post-mortem scheduled for: [date] +- [ ] [Post-mortem document](./post-mortem-template) created (save to [Post-Mortems](../post-mortems/overview) folder) +- [ ] Action items assigned + +--- + +## Links & Evidence + +- [Relevant dashboard] +- [Relevant PR/commit] +- [Screenshots] + +--- + +## Severity Reference + +| Level | Description | +|-------|-------------| +| P1 | Critical - funds at risk, active exploit | +| P2 | High - major impact, immediate response | +| P3 | Moderate - medium impact | +| P4 | Low - minor issues | +| P5 | Info - no action needed | + +See [Incident-Response-Policy](../incident-response-policy#severity-levels) for full definitions. + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/templates/overview.mdx b/docs/pages/incident-management/incident-response-template/templates/overview.mdx new file mode 100644 index 00000000..1a4fb871 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/templates/overview.mdx @@ -0,0 +1,46 @@ +--- +title: "Templates | Security Alliance" +description: "Templates for incident response documentation. Copy these when needed." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Templates + + + + +Templates for incident response documentation. Copy these when needed. + +> **Review these templates before your first incident.** Familiarize yourself with the structure so you're not learning the format during an emergency. Consider running a tabletop exercise to practice filling them out. + +## Available Templates + +| Template | When to Use | +|----------|-------------| +| [Incident-Log-Template](./incident-log-template) | Copy when declaring an incident | +| [Post-Mortem-Template](./post-mortem-template) | Copy after resolving a significant incident (P1-P3) | + +For communication guidance and examples, see [Communications](../communications). + +## Runbook Template + +See [Runbooks](../runbooks/overview) for the runbook template and existing runbooks. + +--- + +*See [Incident-Response-Policy](../incident-response-policy) for the full response process.* + + +--- + + + diff --git a/docs/pages/incident-management/incident-response-template/templates/post-mortem-template.mdx b/docs/pages/incident-management/incident-response-template/templates/post-mortem-template.mdx new file mode 100644 index 00000000..3a3c1246 --- /dev/null +++ b/docs/pages/incident-management/incident-response-template/templates/post-mortem-template.mdx @@ -0,0 +1,186 @@ +--- +title: "Post-Mortem Template | Security Alliance" +description: "Complete this after significant incidents (P1-P3). Focus on learning, not blame." +tags: + - Security Specialist + - Operations & Strategy + - Devops + +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../../components' + + + + +# Post-Mortem Template + + + + +Complete this after significant incidents (P1-P3). Focus on learning, not blame. + +## How to Use + +1. Copy this template to [Post-Mortems](../post-mortems/overview) folder +2. Scribe creates draft before the post-mortem meeting +3. Hold meeting within a week of resolution +4. All responders contribute +5. Every post-mortem produces action items with owners and deadlines +6. Share with team (and publicly if appropriate) + +--- + +# Post-Mortem: [INCIDENT TITLE] + +## Metadata + +| Field | Value | +|-------|-------| +| **Incident Date** | YYYY-MM-DD | +| **Severity** | P1 / P2 / P3 | +| **Authors** | | +| **Status** | Draft / Final | +| **Incident Log** | [Link to incident log](../incident-logs/overview) | + +--- + +## Summary + +[2-4 paragraphs. What happened, when, how long, how it was resolved. Someone unfamiliar should understand the incident after reading this.] + +--- + +## Impact + +### Users +- Users affected: +- Duration: +- Services unavailable: + +### Financial +- Funds at risk: +- Actual losses: + +### Reputation +- Public visibility: +- Media coverage: + +--- + +## Timeline + +| Time (UTC) | Event | +|------------|-------| +| | Incident began | +| | Detected | +| | Response started | +| | Root cause identified | +| | Mitigation applied | +| | Resolved | + +See linked Incident Log for detailed timeline. + +--- + +## Root Cause + +### Primary Cause + +[What was the fundamental reason this happened?] + +### Contributing Factors + +1. +2. +3. + +### 5 Whys + +| Question | Answer | +|----------|--------| +| Why did [incident] happen? | | +| Why? | | +| Why? | | +| Why? | | +| Why? | | + +--- + +## What Went Well + +1. +2. +3. + +## What Went Wrong + +1. +2. +3. + +## Where We Got Lucky + +[What fortunate circumstances helped that we shouldn't rely on next time?] + +1. +2. + +--- + +## Action Items + +> Every action item needs an owner and deadline. + +| Action | Owner | Deadline | Status | +|--------|-------|----------|--------| +| | | | | +| | | | | +| | | | | + +--- + +## Lessons for Runbooks + +Should we create or update a [runbook](../runbooks/overview) based on this incident? + +- [ ] New runbook needed: [type] +- [ ] Existing runbook to update: [which one] +- [ ] No runbook changes needed + +--- + +## Detection + +| Aspect | Details | +|--------|---------| +| How detected | Monitoring / User report / Team member / Other | +| Time to detection | | +| Could we detect faster? | | + +--- + +## Links + +- Incident Log: [Link to incident log] +- Relevant PRs: +- Dashboards: +- External references: + +--- + +## Meeting Notes + +**Attendees:** + +**Discussion points:** + +--- + +*Template based on [Incident-Response-Policy](../incident-response-policy#step-6-post-incident-review)* + + +--- + + + diff --git a/docs/pages/incident-management/index.mdx b/docs/pages/incident-management/index.mdx index 133cc706..0eef31b4 100644 --- a/docs/pages/incident-management/index.mdx +++ b/docs/pages/incident-management/index.mdx @@ -15,4 +15,5 @@ title: "Incident Management" - [Incident Communication Strategies](/incident-management/communication-strategies) - [Incident Detection And Response](/incident-management/incident-detection-and-response) - [Incident Lessons Learned](/incident-management/lessons-learned) +- [Incident Response Template](/incident-management/incident-response-template) - [Playbooks](/incident-management/playbooks) diff --git a/docs/pages/incident-management/lessons-learned.mdx b/docs/pages/incident-management/lessons-learned.mdx index 32b908a4..8f83fae0 100644 --- a/docs/pages/incident-management/lessons-learned.mdx +++ b/docs/pages/incident-management/lessons-learned.mdx @@ -6,6 +6,10 @@ tags: - Operations & Strategy - Devops - SRE + +contributors: + - role: wrote + users: [dickson] --- import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../components' @@ -33,6 +37,33 @@ implementing improvements. 5. Share the lessons learned with the ecosystem to promote awareness and improve overall security practices. 6. Revise incident response policies and procedures based on the lessons learned to ensure continuous improvement. +## Questions worth asking + +- Was the incident detected as quickly as it should have been? +- Did the severity level reflect actual impact? +- Were the right people involved early enough? +- Did the team have an appropriate runbook or was too much invented during the response? +- Was the external communication cadence appropriate? +- Did logs, dashboards, and evidence collection support investigation effectively? +- What should change in monitoring, alerting, staffing, or access controls? + +## Outputs beyond the write-up + +A good retrospective often drives updates to: + +- playbooks and runbooks +- alert thresholds and monitoring coverage +- access control or break-glass procedures +- communication templates +- training and tabletop exercises + +The review should end with tangible changes, not just a document. + +For a concrete post-mortem structure and example write-up, see +[Incident Response Template: Post-Mortem Template](/incident-management/incident-response-template/templates/post-mortem-template) +and +[Incident Response Template: Example Post-Mortem](/incident-management/incident-response-template/post-mortems/2024-01-15-example-api-outage). + --- diff --git a/docs/pages/incident-management/overview.mdx b/docs/pages/incident-management/overview.mdx index 0bdf2218..cb2244fd 100644 --- a/docs/pages/incident-management/overview.mdx +++ b/docs/pages/incident-management/overview.mdx @@ -6,6 +6,10 @@ tags: - Operations & Strategy - Devops - SRE + +contributors: + - role: wrote + users: [dickson] --- import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../components' @@ -29,6 +33,7 @@ timely recovery. 3. [Lessons Learned](/incident-management/lessons-learned) 4. [Playbooks](/incident-management/playbooks/overview) 5. [SEAL 911 War Room Guidelines](/incident-management/playbooks/seal-911-war-room-guidelines) +6. [Incident Response Template](/incident-management/incident-response-template/overview) --- diff --git a/docs/pages/incident-management/playbooks/overview.mdx b/docs/pages/incident-management/playbooks/overview.mdx index 278521b8..574be795 100644 --- a/docs/pages/incident-management/playbooks/overview.mdx +++ b/docs/pages/incident-management/playbooks/overview.mdx @@ -4,6 +4,10 @@ description: "Create step-by-step incident response playbooks for stolen funds, tags: - Security Specialist - Operations & Strategy + +contributors: + - role: wrote + users: [dickson] --- import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../../components' @@ -20,6 +24,18 @@ Generally speaking, incident response playbooks aim to provide detailed, step-by types of security incidents. Obviously, it's not possible to have thought about every possible scenario ahead of time, but one could create documentation for the most likely or devastating scenarios. +## What good playbooks do + +Good playbooks reduce cognitive load during high-pressure events. They should help responders: + +- recognize the incident pattern quickly +- identify the first containment actions +- gather the right evidence +- escalate to the right people +- recover safely without skipping verification + +A playbook should not be a long essay. It should be easy to follow while an incident is active. + ## Best Practices 1. Define the type of incident the playbook addresses (e.g., stolen funds, data breach, DDoS attack). @@ -30,6 +46,46 @@ to use. 5. Outline procedures for restoring everything affected to normal operation. 6. Detail the steps for conducting a lessons learned review. +## Recommended structure + +Many teams find a simple structure works best: + +- **Quick reference**: severity, owner, and primary responder +- **Identification**: symptoms, indicators, and how to distinguish similar incidents +- **Immediate actions**: first steps to stop further damage +- **Investigation**: scope, evidence sources, and key questions +- **Mitigation**: preferred and fallback containment options +- **Recovery**: what must be verified before normal operations resume +- **Escalation**: when to involve decision makers, vendors, legal, or external responders +- **Post-incident follow-up**: monitoring, documentation, and review + +## Scenario coverage + +Start with the scenarios that are both plausible and high impact for the project. Examples often include: + +- smart contract exploit or critical vulnerability +- key compromise +- frontend compromise +- dependency or build pipeline compromise +- third-party outage +- DDoS or service availability attack + +Not every team needs every playbook on day one, but every team should know which incidents would be hardest to improvise. + +## Playbook maintenance + +Playbooks should be updated: + +- after real incidents +- after drills or tabletop exercises +- when infrastructure, vendors, or ownership changes +- when the team discovers that instructions are too vague to execute quickly + +Untested runbooks can create false confidence, so teams should practice the highest-risk ones periodically. + +For example incident runbooks and a reusable runbook template, see +[Incident Response Template: Runbooks](/incident-management/incident-response-template/runbooks/overview). + --- diff --git a/vocs.config.tsx b/vocs.config.tsx index ae34506d..8f0e5ac9 100644 --- a/vocs.config.tsx +++ b/vocs.config.tsx @@ -259,6 +259,60 @@ const config = { { text: 'Decentralized Incident Response Framework (DeIRF)', link: '/incident-management/playbooks/decentralized-ir' }, ] }, + { + text: 'Incident Response Template', + collapsed: false, + dev: true, + items: [ + { text: 'Overview', link: '/incident-management/incident-response-template/overview', dev: true }, + { text: 'Incident Response Policy', link: '/incident-management/incident-response-template/incident-response-policy', dev: true }, + { text: 'Roles and Staffing', link: '/incident-management/incident-response-template/roles-and-staffing', dev: true }, + { text: 'Communications', link: '/incident-management/incident-response-template/communications', dev: true }, + { text: 'Contacts', link: '/incident-management/incident-response-template/contacts', dev: true }, + { + text: 'Templates', + collapsed: true, + items: [ + { text: 'Overview', link: '/incident-management/incident-response-template/templates/overview', dev: true }, + { text: 'Incident Log Template', link: '/incident-management/incident-response-template/templates/incident-log-template', dev: true }, + { text: 'Post-Mortem Template', link: '/incident-management/incident-response-template/templates/post-mortem-template', dev: true }, + ] + }, + { + text: 'Incident Logs', + collapsed: true, + items: [ + { text: 'Overview', link: '/incident-management/incident-response-template/incident-logs/overview', dev: true }, + { text: 'API Outage Example', link: '/incident-management/incident-response-template/incident-logs/2024-01-15-example-api-outage', dev: true }, + ] + }, + { + text: 'Post-Mortems', + collapsed: true, + items: [ + { text: 'Overview', link: '/incident-management/incident-response-template/post-mortems/overview', dev: true }, + { text: 'API Outage Example', link: '/incident-management/incident-response-template/post-mortems/2024-01-15-example-api-outage', dev: true }, + ] + }, + { + text: 'Runbooks', + collapsed: true, + items: [ + { text: 'Overview', link: '/incident-management/incident-response-template/runbooks/overview', dev: true }, + { text: 'Smart Contract Exploit', link: '/incident-management/incident-response-template/runbooks/smart-contract-exploit', dev: true }, + { text: 'Key Compromise', link: '/incident-management/incident-response-template/runbooks/key-compromise', dev: true }, + { text: 'Frontend Compromise', link: '/incident-management/incident-response-template/runbooks/frontend-compromise', dev: true }, + { text: 'DNS Hijack', link: '/incident-management/incident-response-template/runbooks/dns-hijack', dev: true }, + { text: 'CDN/Hosting Compromise', link: '/incident-management/incident-response-template/runbooks/cdn-hosting-compromise', dev: true }, + { text: 'Dependency Attack', link: '/incident-management/incident-response-template/runbooks/dependency-attack', dev: true }, + { text: 'Build Pipeline Compromise', link: '/incident-management/incident-response-template/runbooks/build-pipeline-compromise', dev: true }, + { text: 'DDoS Attack', link: '/incident-management/incident-response-template/runbooks/ddos-attack', dev: true }, + { text: 'Third-Party Outage', link: '/incident-management/incident-response-template/runbooks/third-party-outage', dev: true }, + { text: 'Runbook Template', link: '/incident-management/incident-response-template/runbooks/runbook-template', dev: true }, + ] + }, + ] + }, ] }, {