diff --git a/docs/pages/multisig-for-protocols/emergency-procedures.mdx b/docs/pages/multisig-for-protocols/emergency-procedures.mdx index fea8c928..70fe4486 100644 --- a/docs/pages/multisig-for-protocols/emergency-procedures.mdx +++ b/docs/pages/multisig-for-protocols/emergency-procedures.mdx @@ -173,6 +173,12 @@ Current multisig status: - **Backup signers** - Have additional signers available for activation - **Communication redundancy** - Multiple ways to reach each signer +## Emergency Pause Runbook + +Use the full [Operational Runbooks](/multisig-for-protocols/operational-runbooks) page as the canonical execution +playbook for emergency pause transactions. Keep this page focused on incident coordination, communication, and recovery, +and use the runbook page for the transaction-specific signing and execution steps. + ### Emergency Drill Procedures #### Regular Testing Schedule @@ -230,6 +236,7 @@ Current multisig status: - [Hardware Wallet Setup](/wallet-security/intermediates-and-medium-funds) - Device replacement procedures - [Seed Phrase Management](/wallet-security/seed-phrase-management) - Key recovery procedures - [Personal Security (OpSec)](/multisig-for-protocols/personal-security-opsec) - Account security measures +- [Operational Runbooks](/multisig-for-protocols/operational-runbooks) - Example runbooks for common operations diff --git a/docs/pages/multisig-for-protocols/index.mdx b/docs/pages/multisig-for-protocols/index.mdx index d6d02cca..a2153573 100644 --- a/docs/pages/multisig-for-protocols/index.mdx +++ b/docs/pages/multisig-for-protocols/index.mdx @@ -17,6 +17,7 @@ title: "Multisig For Protocols" - [Multisig Registration & Documentation](/multisig-for-protocols/registration-and-documentation) - [Multisig Communication Setup](/multisig-for-protocols/communication-setup) - [Multisig Use Case Requirements](/multisig-for-protocols/use-case-specific-requirements) +- [Multisig Operational Runbooks](/multisig-for-protocols/operational-runbooks) - [Joining a Multisig](/multisig-for-protocols/joining-a-multisig) - [Multisig Emergency Procedures](/multisig-for-protocols/emergency-procedures) - [Backup Signing & Infrastructure](/multisig-for-protocols/backup-signing-and-infrastructure) diff --git a/docs/pages/multisig-for-protocols/operational-runbooks.mdx b/docs/pages/multisig-for-protocols/operational-runbooks.mdx new file mode 100644 index 00000000..a6453485 --- /dev/null +++ b/docs/pages/multisig-for-protocols/operational-runbooks.mdx @@ -0,0 +1,488 @@ +--- +title: "Multisig Operational Runbooks | SEAL" +description: "Multisig operational runbooks for token transfers, threshold changes, and emergency pause actions, with verification steps and fallback paths." +tags: + - Engineer/Developer + - Security Specialist + - Multisig Security +contributors: + - role: wrote + users: [isaac, geoffrey, louis, pablo, dickson] +--- + +import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } from '../../../components' + + + + +# Operational Runbooks + + + + +These are example runbooks for common multisig operations. Review and customize them for your protocol before use. + +## Token Transfer Runbook + +### Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | Routine / Time-Sensitive | +| **Required Threshold** | Standard | +| **Owner** | Treasury Admin | +| **Last Updated** | [Date] | + +### When to Use + +Sending tokens (ETH, ERC20, SOL, SPL tokens) from a multisig to another address. + +### Prerequisites + +- [ ] Recipient address verified via independent source (not just chat message) +- [ ] Amount confirmed +- [ ] Reason for transfer documented or approved + +### EVM (Safe) Procedure + +#### 1. Create Transaction + +**Proposer**: + +1. Go to [https://app.safe.global](https://app.safe.global) +2. Select correct Safe and network +3. Click "New Transaction" > "Send tokens" +4. Enter recipient address (use checksummed format) +5. Select token and enter amount +6. Review and create transaction + +#### 2. Verify Transaction + +**Each signer must**: + +```text +[ ] Correct Safe address +[ ] Correct network (Ethereum/Base/etc.) +[ ] Expected nonce +[ ] Recipient address matches independent source +[ ] Token is correct +[ ] Amount is correct +[ ] Simulation shows expected transfer +[ ] Message hash matches hardware wallet +``` + +**Hash verification**: + +```bash +./safe_hashes.sh --network mainnet --address [SAFE_ADDRESS] --nonce [NONCE] +``` + +Or use [OpenZeppelin Safe Utils](https://safe-utils.openzeppelin.com). + +**Simulation check**: + +- Click "Simulate" in Safe UI +- Verify token transfer event shows correct recipient and amount +- No unexpected transfers or approvals + +#### 3. Sign + +- [ ] Review transaction on hardware wallet +- [ ] Verify message hash matches tool output +- [ ] Sign +- [ ] Communicate: "Verified and signed, X more needed" + +#### 4. Execute + +- [ ] Last signer clicks "Execute" (or designated executor) +- [ ] Confirm transaction on-chain via block explorer +- [ ] Communicate: "Executed - [tx hash]" + +### Solana (Squads) Procedure + +#### 1. Create Transaction + +**Proposer**: + +1. Go to [https://squads.xyz/squads-multisig](https://squads.xyz/squads-multisig) +2. Select correct Squad +3. Click "Send" > Select token +4. Enter recipient address and amount +5. Create proposal + +#### 2. Verify Transaction + +**Each signer must**: + +```text +[ ] Correct Squad address +[ ] Recipient address matches independent source +[ ] Token is correct +[ ] Amount is correct +[ ] Simulation shows expected transfer +``` + +**Simulation check**: + +- Click "Simulate" after first signature +- For SOL: Find "System Program Instruction", verify destination and amount +- For tokens: Find "Token Program Instruction", verify destination, mint, and amount + +#### 3. Sign + +- [ ] Click "Approve" +- [ ] Verify hash in Squads UI matches Ledger display +- [ ] Sign on Ledger +- [ ] Communicate: "Verified and signed, X more needed" + +#### 4. Execute + +- [ ] Last signer executes +- [ ] Verify on Solana Explorer +- [ ] Communicate: "Executed - [tx signature]" + +### Verification Details + +#### Recipient Address + +Always verify recipient via independent source: + +- Direct communication with recipient (video call or verified channel) +- Published address in official documentation +- On-chain verification (existing relationship) + +Never trust an address shared only via chat message. + +#### Amount Verification + +For ERC20 or SPL tokens, verify decimals: + +- USDC or USDS: 6 decimals (1,000,000 = 1 token) +- Most ERC20: 18 decimals + +### Common Issues + +| Issue | Solution | +|-------|----------| +| Simulation fails | Check token balance, gas estimation | +| Wrong network | Reject and recreate on correct network | +| Nonce mismatch | Clear pending transactions first | +| Recipient can't receive | Verify address is correct type (EOA vs contract) | + +## Threshold Change Runbook + +### Quick Reference + +| Field | Value | +|-------|-------| +| **Typical Severity** | Routine | +| **Required Threshold** | Current threshold | +| **Owner** | Multisig Admin | +| **Last Updated** | [Date] | + +### When to Use + +- Increasing threshold for higher security +- Decreasing threshold for operational efficiency +- Adjusting after signer count changes + +### Prerequisites + +- [ ] Justification documented +- [ ] New threshold meets policy requirements: + - Minimum 50% of signers + - Cannot exceed total signer count +- [ ] If decreasing: approval from security contact +- [ ] All signers notified of change + +### Policy Requirements + +| Multisig Type | Minimum Threshold | +|--------------|-------------------| +| Standard | 50%+ | +| Treasury ($1M+) | 50%+ with 7+ signers | +| Emergency | Can be lower (2/4 typical) | +| Smart Contract Upgrades | Consider 7/9+ | + +### EVM (Safe) Procedure + +#### 1. Create Transaction + +**Proposer**: + +1. Go to [https://app.safe.global](https://app.safe.global) +2. Settings > Setup > Owners +3. Click "Change threshold" +4. Select new threshold from dropdown +5. Create transaction + +#### 2. Verify Transaction + +**Each signer must**: + +```text +[ ] Transaction is ONLY changing threshold (no other changes) +[ ] New threshold value is correct +[ ] New threshold meets policy requirements +[ ] No signers being added or removed in same transaction +[ ] Message hash matches hardware wallet +``` + +**Simulation check**: + +- Should show `changeThreshold` call to Safe +- New threshold value matches intended + +#### 3. Sign and Execute + +Standard signing process: + +- [ ] Verify hash on hardware wallet +- [ ] Sign +- [ ] Communicate: "Verified and signed, X more needed" +- [ ] Last signer executes + +#### 4. Verify + +- [ ] Check Safe settings show new threshold +- [ ] Test that the next transaction requires new threshold + +### Solana (Squads) Procedure + +#### 1. Create Proposal + +1. Go to [https://squads.xyz/squads-multisig](https://squads.xyz/squads-multisig) +2. Configuration > Change Threshold +3. Select new threshold +4. Create proposal + +#### 2. Verify and Sign + +```text +[ ] Only threshold being changed +[ ] New value is correct +[ ] Meets policy requirements +``` + +#### 3. Execute + +Standard execution after threshold reached. + +### Post-Change + +- [ ] Update registration or internal documentation with new threshold +- [ ] Notify all signers of new threshold +- [ ] Update any documentation referencing old threshold + +### Common Scenarios + +#### Adding Signer + Adjusting Threshold + +When adding a signer, it is often appropriate to increase threshold. + +**Option 1**: Two transactions + +1. Add signer (threshold stays same) +2. Increase threshold + +**Option 2**: Single batch transaction (EVM) + +1. Use Transaction Builder +2. Batch `addOwnerWithThreshold` (sets new threshold automatically) + +#### Removing Signer + Adjusting Threshold + +When removing a signer, threshold cannot exceed new signer count: + +1. If threshold equals signers: reduce threshold first or use batch +2. Remove signer transaction will auto-adjust if needed + +### Common Issues + +| Issue | Solution | +|-------|----------| +| Threshold > signer count | Reduce to valid value | +| Transaction fails | Check current signer count | +| Policy violation | Get approval before proceeding | + +## Emergency Pause Runbook + +This is an example runbook. Review and customize it for your protocol before use. Add your specific contract +addresses, pause functions, emergency contacts, and communication channels. + +### Quick Reference + +| Field | Value | +|-------|-------| +| **Severity** | EMERGENCY | +| **Response Time** | \<2 hours | +| **Required Threshold** | Per multisig config (often lower for emergencies) | +| **Owner** | Security Team | +| **Last Updated** | [Date] | + +### When to Use + +- Active exploit detected +- Suspicious activity on protocol contracts +- Key compromise affecting protocol +- Vulnerability disclosure requiring immediate action + +### Immediate Actions (First 30 Minutes) + +#### 1. Alert Team + +- [ ] Send alert to emergency Signal group +- [ ] Page signers via configured paging system +- [ ] Notify the security contact + +**Alert template**: + +```text +URGENT: [Brief description] +Multisig: [Name] +Action needed: [Pause/Freeze/etc.] +Respond ASAP - <2hr SLA +``` + +#### 2. Assess Situation + +- [ ] Confirm threat is real (not false alarm) +- [ ] Identify affected contracts or assets +- [ ] Determine which pause function(s) to call +- [ ] Estimate urgency + +#### 3. Prepare Transaction + +**Proposer** (can be any signer or delegated proposer): + +1. Go to Safe or Squads UI +2. Use Transaction Builder for contract interaction +3. Select target contract +4. Select `pause()` or appropriate emergency function +5. Create transaction + +### Signing Process (Streamlined) + +Emergency signing follows abbreviated verification. + +#### Minimum Verification + +```text +[ ] Correct multisig address +[ ] Correct network +[ ] Target contract is correct (verify address) +[ ] Function is pause() or expected emergency function +[ ] No unexpected additional calls +[ ] Hash matches hardware wallet +``` + +#### Sign and Communicate + +- [ ] Sign immediately after verification +- [ ] Message: "Signed - [X/Y] - [your name]" +- [ ] Stay available until executed + +#### Execute + +- [ ] Execute as soon as threshold is reached +- [ ] Verify pause took effect (check contract state) +- [ ] Communicate: "EXECUTED - pause confirmed" + +### Contract Reference + +| Contract | Address | Pause Function | Expected Behavior | +|----------|---------|---------------|-------------------| +| [Contract 1] | `0x...` | `pause()` | Stops all operations | +| [Contract 2] | `0x...` | `emergencyStop()` | Stops withdrawals | +| [Add your contracts] | | | | + +### After Pause + +#### Immediate (Within 1 hour) + +- [ ] Confirm pause is effective +- [ ] Document incident using [Incident Reporting](/multisig-for-protocols/incident-reporting) +- [ ] Notify stakeholders + +#### Short-term (Within 24 hours) + +- [ ] Root cause analysis +- [ ] Plan for resolution +- [ ] Draft public communication if needed + +#### Resolution + +- [ ] Fix underlying issue +- [ ] Test fix thoroughly +- [ ] Plan unpause procedure +- [ ] Execute unpause with full verification (not emergency process) + +### If Primary UI is Down + +Use backup infrastructure: + +**EVM**: + +1. [Eternal Safe](https://eternalsafe.eth.limo) +2. Configure with backup RPC +3. Load Safe address +4. Create custom transaction with pause calldata + +**Solana**: + +1. [Squads Backup](https://backup.app.squads.so) +2. Configure RPC +3. Load multisig + +See [Backup Signing & Infrastructure](/multisig-for-protocols/backup-signing-and-infrastructure) for detailed instructions. + +### Key Compromise Variant + +If a signer key is compromised: + +1. Do not use compromised key for any operations +2. Alert team via backup channel (assume primary may be compromised) +3. Assess if attacker has threshold +4. Prepare signer rotation transaction +5. Execute removal of compromised signer +6. Rotate compromised signer to new address + +### Communication Channel Compromise + +If a communication channel is compromised: + +1. Switch to backup channel immediately +2. Verify identity of all participants via secondary method +3. Do not trust any pending requests from compromised channel +4. Assume attacker may attempt social engineering + +### Drill Checklist + +For emergency drills (run bi-annually): + +- [ ] Page all signers +- [ ] Measure time to reach threshold +- [ ] Practice emergency signing flow on testnet +- [ ] Test backup communication channels +- [ ] Test backup UIs +- [ ] Document results and improve procedures + +### Escalation + +If threshold cannot be reached within SLA: + +1. Contact backup signers +2. Escalate to decision makers +3. Consider alternative mitigations +4. Document why threshold was not met + +## Related Documents + +- [Safe Multisig: Step-by-Step Verification](/wallet-security/signing-and-verification/secure-multisig-safe-verification) +- [Squads Multisig: Step-by-Step Verification](/wallet-security/signing-and-verification/secure-multisig-squads-verification) +- [Emergency Procedures](/multisig-for-protocols/emergency-procedures) +- [Backup Signing & Infrastructure](/multisig-for-protocols/backup-signing-and-infrastructure) +- [Incident Reporting](/multisig-for-protocols/incident-reporting) + + + diff --git a/docs/pages/multisig-for-protocols/overview.mdx b/docs/pages/multisig-for-protocols/overview.mdx index 27a753f2..04f18e81 100644 --- a/docs/pages/multisig-for-protocols/overview.mdx +++ b/docs/pages/multisig-for-protocols/overview.mdx @@ -32,7 +32,8 @@ import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } fr - Joining as a signer? → [Joining a Multisig](/multisig-for-protocols/joining-a-multisig) and [Hardware Wallet Setup](/wallet-security/intermediates-and-medium-funds) - Need to sign a transaction? → Signing & Verification: [Safe Multisig](/wallet-security/signing-and-verification/secure-multisig-safe-verification) and -[Squads](/wallet-security/signing-and-verification/secure-multisig-squads-verification) +[Squads](/wallet-security/signing-and-verification/secure-multisig-squads-verification), plus +[Operational Runbooks](/multisig-for-protocols/operational-runbooks) - Emergency situation? → [Emergency Procedures](/multisig-for-protocols/emergency-procedures) ## Core principles @@ -55,6 +56,7 @@ import { TagList, AttributionList, TagProvider, TagFilter, ContributeFooter } fr - [Registration & Documentation](/multisig-for-protocols/registration-and-documentation) - Document and verify setup - [Communication Setup](/multisig-for-protocols/communication-setup) - Establish secure communication channels - [Use Case Specific Requirements](/multisig-for-protocols/use-case-specific-requirements) - Special requirements by type +- [Operational Runbooks](/multisig-for-protocols/operational-runbooks) - Example procedures for common operations ### 3. For Signers diff --git a/docs/pages/multisig-for-protocols/registration-and-documentation.mdx b/docs/pages/multisig-for-protocols/registration-and-documentation.mdx index 6cbd0167..28ea67e6 100644 --- a/docs/pages/multisig-for-protocols/registration-and-documentation.mdx +++ b/docs/pages/multisig-for-protocols/registration-and-documentation.mdx @@ -76,6 +76,73 @@ Detailed steps for collecting this information are provided in [Joining a Multis **Note**: Entity affiliations are acceptable - the goal is accountability, not doxing. +## Roles & Accountability + +### Accountability Structure + +| Role | Responsibilities | +|------|------------------| +| **Multisig Operations Lead** | Policy maintenance, signer coordination, documentation, periodic reviews, incident escalation | +| **Security Contact** | Security incident response, signer verification, emergency coordination | + +### Multisig-Specific Roles + +For each multisig, assign: + +| Role | Responsibility | +|------|----------------| +| **Admin** | Setup, configuration, signer management, documentation | +| **Transaction Proposer** | Prepares and proposes transactions (may be delegated non-signer) | +| **Signers** | Review, verify, and sign transactions | + +### Signer Responsibilities + +Every signer must: + +- Use a hardware wallet for all multisig operations +- Maintain a backup hardware wallet with the same seed +- Store the seed phrase securely +- Verify every transaction before signing +- Respond within SLA based on multisig classification +- Report incidents immediately +- Complete training and participate in drills + +#### Response Time SLAs + +Use the response time requirements from [Planning & Classification](/multisig-for-protocols/planning-and-classification#operational-classification): + +- **Emergency**: \<2 hours +- **Time-Sensitive**: 2-12 hours +- **Routine**: 24-48 hours + +### Admin Responsibilities + +Multisig admins must: + +- Ensure the multisig is properly documented +- Maintain an up-to-date signer list with verified addresses +- Set up primary and backup communication channels +- Coordinate signer onboarding and offboarding +- Schedule and conduct periodic reviews (quarterly minimum) +- Ensure backup infrastructure is configured and tested + +### Operational Lead Responsibilities + +- Maintain the playbook and keep documentation current +- Coordinate across all multisigs +- Conduct periodic audits of multisig configurations +- Escalate security concerns to the security contact +- Report on operational status + +### Review Schedule + +| Review Type | Frequency | Owner | +|------------|-----------|-------| +| Signer access review | Quarterly | Multisig Admin | +| Classification review | Quarterly or after major changes | Ops Lead | +| Emergency contact verification | Every 6 months | Ops Lead | +| Full policy review | Annually | Ops Lead + Security | + ## Update Template Use this template when making changes to signer composition: @@ -124,6 +191,89 @@ Transaction: [Link to executed transaction] - Update classification if operational patterns change - Maintain current contact information +### Transaction Review Records + +Maintain audit trails for: + +- Transaction reviews and approvals +- Execution and post-execution confirmation +- Verification evidence +- Issues encountered + +**Retention**: 3 years minimum + +Transaction records should capture: + +```text +Transaction: [Brief Description] + +Date: [YYYY-MM-DD] +Multisig: [Name] +Status: Proposed / Signing / Executed / Rejected + +Transaction Details +- Network +- Safe or Squad address +- Nonce +- Transaction type + +What This Transaction Does +- Plain language description of what the transaction accomplishes + +Initiation +- Proposed by +- Proposed date +- Reason or justification +- Runbook followed + +Verification & Signing +- Signer +- Verified +- Signed +- Date +- Notes + +Verification Checklist +- Correct multisig address +- Correct network +- Expected nonce +- Target address verified +- Calldata or amount verified +- Simulation performed +- Hash matched hardware wallet + +Simulation Results +- Tool used +- Result +- Expected behavior confirmed +- Link + +Execution +- Executed by +- Execution date +- Transaction hash +- Block explorer link +- Gas used + +Post-Execution Verification +- Transaction confirmed on-chain +- Expected state change verified +- Registration updated if applicable +- Team notified + +Issues Encountered +- Document any issues, delays, or anomalies + +Attachments +- Screenshot of simulation +- Screenshot of hardware wallet confirmation +- Communication thread link + +Sign-Off +- Proposer +- Final executor +``` + ## Ongoing Management ### Regular reviews @@ -164,6 +314,7 @@ Use the template in [Registration & Documentation → Update Template](/multisig - [Planning & Classification](/multisig-for-protocols/planning-and-classification) - How to classify your multisig - [Joining a Multisig](/multisig-for-protocols/joining-a-multisig) - Signer verification process +- [Operational Runbooks](/multisig-for-protocols/operational-runbooks) - Example procedures for common operations diff --git a/vocs.config.tsx b/vocs.config.tsx index ae34506d..b4add382 100644 --- a/vocs.config.tsx +++ b/vocs.config.tsx @@ -145,6 +145,7 @@ const config = { { text: 'Registration & Documentation', link: '/multisig-for-protocols/registration-and-documentation' }, { text: 'Communication Setup', link: '/multisig-for-protocols/communication-setup' }, { text: 'Use-Case Specific Requirements', link: '/multisig-for-protocols/use-case-specific-requirements' }, + { text: 'Operational Runbooks', link: '/multisig-for-protocols/operational-runbooks', dev: true }, ] }, {