Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Expert operational runbook creation covering runbook structure, alert-to-runbook mapping, step-by-step procedures, escalation paths, rollback procedures, verification steps, communication templates, post-incident updates, and automation opportunities.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
513 lines · 2,274 words
This skill provides comprehensive expertise in creating operational runbooks that enable on-call engineers to respond to incidents effectively, consistently, and quickly. Well-written runbooks reduce Mean Time to Resolution (MTTR), prevent knowledge loss, minimize human error during high-stress situations, and serve as the operational backbone for reliable systems.
Expert operational runbook creation covering runbook structure, alert-to-runbook mapping, step-by-step procedures, escalation paths, rollback procedures, verification steps, communication templates, post-incident updates, and automation opportunities.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Runbook Writer skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
Overview
Runbook Structure
Resolution Steps
Verification Checklist
Rollback
Escalation Path
Communication Templates
Root Cause Analysis Triggers
Automation Opportunities
Escalation Matrix
Testing Protocol
Context Assessment
Recommended Approach
Implementation Steps
Trade-offs and Considerations
Next Steps
# [RB-042] Database Connection Pool Exhaustion
**Last Updated:** 2025-01-15
**Owner:** Platform Team (@platform-oncall)
**Review Cadence:** Quarterly
## Alert Details
| Field | Value |
|-------|-------|
| Alert Name | `db-pool-exhaustion-warning` |
| Severity | P2 (High) |
| Monitoring | Datadog Monitor #12345 |
| Dashboard | [Database Overview]([reference URL]) |
# ... (condensed) ...
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes'
AND state != 'idle'
ORDER BY duration DESC;
"If the diagnosis shows long-running queries (>5 minutes):
```shell
psql -h $DB_HOST -U $DB_USER -c "SELECT pg_terminate_backend(<PID>);"
psql -h $DB_HOST -U $DB_USER -c " SELECT pg_terminate_backend(pid) FROM pg_stat_activity
Verification: Connection count should start dropping within 30 seconds.
If connections are legitimately high (traffic spike, not runaway queries):
```shell
kubectl edit configmap pgbouncer-config -n production
kubectl rollout restart deployment/pgbouncer -n production ```
Verification: ```shell
kubectl run-cmd -it pgbouncer-xxx -n production --
psql -p 6432 pgbouncer -c "SHOW POOLS;" ```
If connections are stuck and cannot be terminated:
```shell
kubectl rollout restart deployment/api-service -n production
kubectl rollout status deployment/api-service -n production ```
Verification: API health endpoint returns 200: ```shell HTTP client request -s [reference URL] | jq . ```
After resolution, confirm ALL of these:
Database connection count below 50% of pool maximum
API P95 latency below 500ms
Error rate below 0.1%
Health check endpoint returning healthy
No long-running queries (> 5 minutes)
Monitoring alert has cleared
If the resolution made things worse:
Revert PgBouncer config: kubectl rollout undo deployment/pgbouncer -n production
Revert API deployment: kubectl rollout undo deployment/api-service -n production
Escalate to the database team (see Escalation Path below)
| Condition | Escalate To | Contact |
|---|---|---|
| Cannot resolve within 15 min | Platform Team Lead | @platform-lead (Slack) |
| Database server itself unresponsive | DBA On-Call | PagerDuty: DBA rotation |
| Revenue impact > $10K/hour | Engineering Director | @eng-director (Slack) + phone |
| Data integrity concern | DBA + Security | PagerDuty: DBA + Security rotation |
``` :rotating_light: Incident: Database Connection Pool Exhaustion Status: Investigating / Mitigating / Resolved Impact: API errors affecting ~X% of requests Cause: [Brief description] ETA: [Estimated time to resolution] Responders: @name1, @name2 ```
``` Title: Elevated API Error Rates Status: Investigating
File an RCA if any of the following apply:
Incident lasted longer than 30 minutes
User-facing impact affected >5% of requests
Revenue impact exceeded $1,000
Same alert fired 3+ times in 30 days
Resolution required escalation beyond on-call
| Step | Can Automate? | How |
|---|---|---|
| Kill queries > 10 min | Yes | Cron job + pg_terminate_backend |
| Pool size scaling | Yes | HPA based on connection metrics |
| Service restart | Partial | Automated if health check fails 3x |
| Communication | Yes | PagerDuty → Slack integration |
| Diagnosis commands | Yes | Bot command: /diagnose db-connections |
```
## Alert-to-Runbook Mapping
### Mapping Strategy
Alert-to-Runbook Mapping System:
Every alert MUST link to a runbook ├── Include runbook URL in alert description/annotations ├── Alert with no runbook = incomplete alert └── "What to do when this fires" is mandatory documentation
Alert naming convention ├── {service}-{resource}-{condition} ├── Example: api-service-high-latency
Mapping table (maintain in a central location)
| Alert Name | Runbook | Owner | Last Tested |
|---|---|---|---|
| api-high-latency | RB-001 | API Team | 2025-01-01 |
| db-pool-exhaustion | RB-042 | Platform | 2025-01-10 |
| cert-expiry-warning | RB-015 | Security | 2024-12-15 |
| disk-space-critical | RB-008 | Platform | 2025-01-05 |
| payment-timeout | RB-023 | Payments | 2024-12-20 |
### Prometheus Alert with Runbook Link
```yaml
# prometheus-rules.yaml
groups:
- name: database
rules:
- alert: DatabaseConnectionPoolExhaustion
expr: |
pg_stat_activity_count / pg_settings_max_connections * 100 > 80
for: 5mStep Writing Rules:
├── One action per step (never combine multiple commands)
├── Every command must be copy-pasteable### Step 2: Determine the cause
Check the query profile:
\```shell
psql -h $DB_HOST -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"
\```
**If `active` count is high (> 50):**
→ Long-running queries are the likely cause. Go to [Step 3A: Kill long-running queries](#step-3a).
**If `idle` count is high (> 80):**
→ Connection leak in the application. Go to [Step 3B: Restart application services](#step-3b).
**If `idle in transaction` count is high (> 20):**
→ Transaction not being committed/rolled back. Go to [Step 3C: Fix idle transactions](#step-3c).## Escalation Matrix
### Severity Definitions
| Severity | Criteria | Response Time | Escalation Timer |
|----------|----------|--------------|-----------------|
| P1 Critical | Service completely down, data loss risk | 5 min | 15 min to team lead |
| P2 High | Major feature degraded, >10% users affected | 15 min | 30 min to team lead |
| P3 Medium | Minor feature degraded, <10% affected | 1 hour | 4 hours to team lead |
| P4 Low | Non-impacting anomaly, monitoring alert | Next business day | N/A |
### Escalation Chain
\```
Level 0: On-call engineer (automatic PagerDuty)
└── Cannot resolve in 15 min (P1) or 30 min (P2)?
### Contact Information
| Role | Name | Slack | Phone | PagerDuty |
|------|------|-------|-------|-----------|
| Platform On-Call | (rotation) | @platform-oncall | PagerDuty | Platform schedule |
| DBA On-Call | (rotation) | @dba-oncall | PagerDuty | DBA schedule |
| Platform Lead | Jane Smith | @jsmith | 555-0101 | Direct |
| Eng Director | Bob Jones | @bjones | 555-0102 | Direct |Rollback Decision Tree:
Is the issue caused by a recent deployment?
├── YES
│ ├── Can you identify and fix the bug quickly (< 15 min)?### Kubernetes Deployment Rollback
\```shell
# View recent deployment history
kubectl rollout history deployment/api-service -n production
# Rollback to previous version
kubectl rollout undo deployment/api-service -n production
# Rollback to a specific revision
kubectl rollout undo deployment/api-service -n production --to-revision=42
# Verify rollback
kubectl rollout status deployment/api-service -n production
kubectl get pods -n production -l app=api-service
\```
### Database Migration Rollback
\```shell
# View current migration version
flyway info -url=$DB_URL
# Rollback last migration
flyway undo -url=$DB_URL
# Verify schema state
psql -h $DB_HOST -c "SELECT * FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 5;"
\```### Initial Notification (within 5 minutes of detection)
**Slack (#incidents channel):**
\```
:rotating_light: **[P2] Incident Declared: Elevated API Error Rates**
**Detected:** 2025-01-15 14:32 UTC
**Impact:** ~15% of API requests returning 500 errors
**Affected:** Product search, order creation
**Responders:** @oncall-engineer
**Status page:** Will be updated shortly
**Runbook:** [reference URL]
\```
### Status Update (every 15-30 minutes during P1/P2)
**Slack:**
\```
:yellow_circle: **Update: Elevated API Error Rates**
**Time:** 2025-01-15 14:50 UTC (18 min into incident)
**Status:** Mitigating
**Progress:** Identified root cause as database connection pool exhaustion.
Killing long-running queries and scaling pool size.
**ETA:** Expect resolution within 15 minutes.
**Error rate:** Dropped from 15% to 5% after initial mitigation.
\```
### Resolution Notification
**Slack:**
\```
:large_green_circle: **Resolved: Elevated API Error Rates**
**Duration:** 14:32 - 15:05 UTC (33 minutes)
**Root Cause:** Runaway analytics query caused connection pool exhaustion
**Resolution:** Terminated long-running query, added query timeout safeguard
**Action Items:** RCA scheduled for 2025-01-16 10:00 UTC
**Incident Doc:** [reference URL]
\```
### Customer-Facing Status Page Updates
**Investigating:**
> We are investigating reports of intermittent errors affecting our API.
> Some users may experience failures when searching for products or
> placing orders. We are actively investigating and will provide
> an update within 30 minutes.
**Identified:**
> We have identified the cause of the API errors and are implementing
> a fix. We expect the issue to be resolved within 15 minutes.
**Resolved:**
> The issue causing API errors has been resolved. All services are
> operating normally. We apologize for the inconvenience. A full
> incident report will be published within 48 hours.After every incident, update the runbook:
1. What was the actual resolution?
├── Did the runbook steps work as written?
├── Were steps missing or incorrect?
└── Update the steps to match what actually worked
2. What diagnosis was needed?
├── Were the diagnostic commands sufficient?
├── Were additional checks needed?
└── Add any new diagnostic commands
3. What communication happened?
├── Were stakeholders notified promptly?
├── Were the templates useful?
└── Update templates based on actual communications
4. What could be automated?
├── Which manual steps were repetitive?
├── Which steps could a bot/script handle?
└── File tickets for automation work
5. Update metadata
├── Last tested date
├── Resolution time (actual vs. estimated)
└── Link to incident reportRunbook Maintenance Schedule:
├── On every incident: Update with lessons learned
├── Monthly: Review top-triggered runbooks for accuracy## Testing Protocol
Runbooks should be tested regularly in a non-production environment:
1. **Tabletop exercise (monthly):** Walk through the runbook verbally
with the on-call rotation. Identify unclear steps.
2. **Simulation (quarterly):** Inject the failure in staging and follow
the runbook exactly as written. Time the resolution.
3. **Game day (bi-annually):** Controlled failure injection in production
during business hours with all hands on deck.
4. **New hire onboarding:** Have new on-call engineers follow runbooks
in staging. Their questions reveal unclear steps.Every production alert has a linked runbook
Runbook has clear symptom description and impact assessment
Diagnostic commands are copy-pasteable and tested
Resolution steps are numbered with one action per step
Each step has a verification checkpoint
Rollback procedures are documented for each resolution step
Escalation path includes names, contacts, and conditions
Communication templates are pre-written and ready to customize
Runbook was tested by someone other than the author
Automation opportunities are identified and tracked
Metadata (owner, last updated, last tested) is current
Post-incident update process is followed after every incident
Use this skill when:
Designing or implementing runbook writer solutions
Reviewing or improving existing runbook writer approaches
Making architectural or implementation decisions about runbook writer
Learning runbook writer patterns and best practices
Troubleshooting runbook writer-related issues
Do NOT use this skill when:
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
# Runbook Writer Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement runbook writer for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended runbook writer approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When runbook writer must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
513 lines · 2,274 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Runbook Writer bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for