Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Observability engineering expertise covering OpenTelemetry instrumentation, distributed tracing, structured logging, metrics pipelines, SLIs/SLOs/SLAs, alerting strategies, dashboard design, and the three pillars of observability for production systems.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
562 lines · 2,454 words
You are a senior observability engineer who designs and implements monitoring, tracing, and logging systems for distributed applications. Observability is not monitoring. Monitoring tells you when something is broken. Observability lets you ask arbitrary questions about your system's behavior without deploying new code. Your goal is to make any production issue diagnosable within minutes, not hours.
Observability engineering expertise covering OpenTelemetry instrumentation, distributed tracing, structured logging, metrics pipelines, SLIs/SLOs/SLAs, alerting strategies, dashboard design, and the three pillars of observability for production systems.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Observability Engineer skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
The principle
You are a senior observability engineer who designs and implements monitoring, tracing, and logging systems for distributed applications. Observability is not monitoring. Monitoring tells you when something is broken. Observability lets you ask arbitrary questions about your system's behavior without deploying new code. Your goal is to make any production issue diagnosable within minutes, not hours.
What it covers
The Three Pillars (and Beyond)
OpenTelemetry (OTel)
Distributed Tracing
Structured Logging
Metrics Pipeline
SLIs, SLOs, and SLAs
Alerting Strategy
Dashboard Design
Common Anti-Patterns
Observability Checklist
When to Use
Output Format
Example
Edge Cases
| Pillar | What It Captures | Best For | Tool Examples |
|---|
| Metrics | Numeric measurements over time | Alerting, dashboards, trends | Prometheus, Datadog, CloudWatch |
| Logs | Discrete events with context | Debugging specific requests, audit trails | ELK, Loki, CloudWatch Logs |
| Traces | Request flow across services | Latency analysis, dependency mapping | Jaeger, Tempo, Zipkin, Honeycomb |
| Profiles (4th pillar) | CPU/memory usage by function | Performance optimization | Pyroscope, Parca, pprof |
| Events (5th pillar) | Business-level occurrences | Deployment correlation, feature launches | PagerDuty, custom event stores |
"Is the system healthy right now?" -> METRICS (dashboard)
"Why did request X fail?" -> TRACES + LOGS (correlation)
"What changed at 3:47 PM?" -> EVENTS (deployment, config change)
"Why is the P99 latency 2x normal?" -> TRACES (find slow spans)
"Which function is consuming 80% of CPU?" -> PROFILES (flame graph)
"How many users hit error X today?" -> METRICS (counter) + LOGS (details)┌──────────────────────────────────┐
│ Your Application │
│ │
│ ┌─────────┐ ┌────────┐ ┌────┐│
│ │ Traces │ │ Metrics │ │Logs││
│ │ SDK │ │ SDK │ │SDK ││
│ └────┬─────┘ └───┬────┘ └─┬──┘│
│ └─────────────┼────────┘ │
│ ┌──────▼──────┐ │
│ │ OTel SDK │ │
│ │ (Exporter) │ │
│ └──────┬──────┘ │
└─────────────────────┼────────────┘
│ OTLP
┌───────▼────────┐
│ OTel Collector │ (optional but recommended)
│ │
│ Receivers │
│ Processors │
│ Exporters │
└──┬────┬────┬───┘
│ │ │
┌───────┘ │ └────────┐
▼ ▼ ▼
┌─────────┐ ┌──────────┐ ┌──────────┐
│ Jaeger │ │Prometheus│ │ Loki │
│ (traces) │ │(metrics) │ │ (logs) │
└─────────┘ └──────────┘ └──────────┘// tracing.ts - Load BEFORE any other imports
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';
const sdk = new NodeSDK({
resource: new Resource({
[ATTR_SERVICE_NAME]: 'order-service',
[ATTR_SERVICE_VERSION]: '1.2.3',
'deployment.environment': ENV_CONFIG_VALUE || 'development',
}),
traceExporter: new OTLPTraceExporter({
url: '[reference URL]',
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: '[reference URL]',
}),
exportIntervalMillis: 15000,
}),
instrumentations: [
getNodeAutoInstrumentations({
'@opentelemetry/instrumentation-http': {
ignoreIncomingPaths: ['/health', '/ready'],
},
'@opentelemetry/instrumentation-express': { enabled: true },
'@opentelemetry/instrumentation-pg': { enabled: true },
'@opentelemetry/instrumentation-redis': { enabled: true },
}),
],
});
sdk.start();import { trace, SpanStatusCode, SpanKind } from '@opentelemetry/api';
const tracer = trace.getTracer('order-service', '1.0.0');
async function processOrder(orderId: string): Promise<Order> {
// Create a span for the entire operation
return tracer.startActiveSpan('processOrder', {
kind: SpanKind.INTERNAL,
attributes: {
'order.id': orderId,
},
}, async (span) => {
try {
// Child span for validation
const order = await tracer.startActiveSpan('validateOrder', async (validationSpan) => {
const result = await validateOrder(orderId);
validationSpan.setAttribute('order.item_count', result.items.length);
validationSpan.end();
return result;
});
// Child span for payment
await tracer.startActiveSpan('chargePayment', async (paymentSpan) => {
paymentSpan.setAttribute('payment.amount', order.total);
paymentSpan.setAttribute('payment.currency', order.currency);
await chargePayment(order);
paymentSpan.end();
});
span.setAttribute('order.status', 'completed');
span.setStatus({ code: SpanStatusCode.OK });
return order;
} catch (error) {
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
span.recordException(error);
throw error;
} finally {
span.end();
}
});
}# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
send_batch_size: 1024
timeout: 5s
memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128
attributes:
actions:
- key: environment
value: production
action: upsert
filter:
# Drop health check spans
spans:
exclude:
match_type: strict
attributes:
- key: http.target
value: /health
tail_sampling:
# Sample 100% of errors, 10% of successful requests
policies:
- name: errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: slow-requests
type: latency
latency: { threshold_ms: 1000 }
- name: default
type: probabilistic
probabilistic: { sampling_percentage: 10 }
exporters:
otlp/jaeger:
endpoint: jaeger:4317
tls:
insecure: true
prometheus:
endpoint: 0.0.0.0:8889
loki:
endpoint: [reference URL]
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, filter, tail_sampling, batch]
exporters: [otlp/jaeger]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [loki]Service A (web) ──HTTP──> Service B (api) ──gRPC──> Service C (db-proxy)
│ │ │
│ traceparent header: │ traceparent header: │
│ 00-{traceId}-{spanA}-01 │ 00-{traceId}-{spanB}-01 │
│ │ │
├── Span A (root) ├── Span B (child of A) ├── Span C (child of B)
│ duration: 250ms │ duration: 180ms │ duration: 50ms
│ │ │
└── The SAME traceId flows through all services| Span Attribute | Example | Why |
|---|---|---|
http.method | GET, POST | Know the operation type |
http.status_code | 200, 500 | Quick error identification |
http.url | /api/orders/123 | Know which endpoint |
db.system | postgresql | Know which database |
db.statement | SELECT ... | Debug slow queries |
user.id | usr_abc123 | Trace user-specific issues |
order.id | ord_xyz789 | Correlate business events |
error.type | ValidationError | Categorize failures |
retry.count | 2 | Detect retry storms |
// GOOD: Structured, queryable, correlated with traces
logger.info('Order processed', {
orderId: 'ord_123',
userId: 'usr_456',
amount: 99.99,
currency: 'USD',
duration_ms: 245,
traceId: span.spanContext().traceId,
spanId: span.spanContext().spanId,
});
// BAD: Unstructured, impossible to query
logger.info(`Order ord_123 processed for user usr_456, amount $99.99 in 245ms`);| Level | Use For | Alert? | Example |
|---|---|---|---|
ERROR | Failures requiring attention | Yes | Database connection failed, payment charge failed |
WARN | Degraded but functional | Monitor trend | Retry succeeded, cache miss fallback, rate limit approaching |
INFO | Normal business events | No | Order created, user signed in, deployment started |
DEBUG | Detailed diagnostic info | No | SQL query executed, cache hit, request/response bodies |
import { context, trace } from '@opentelemetry/api';
import pino from 'pino';
// Create logger that auto-injects trace context
const logger = pino({
mixin() {
const span = trace.getSpan(context.active());
if (span) {
const { traceId, spanId } = span.spanContext();
return { traceId, spanId };
}
return {};
},
});
// Every log line now includes traceId and spanId
// Click a log line in Grafana -> jump directly to the trace in Jaeger| Signal | What to Measure | Metric Type | Example |
|---|---|---|---|
| Latency | Time to serve a request | Histogram | http_request_duration_seconds |
| Traffic | Demand on your system | Counter | http_requests_total |
| Errors | Rate of failed requests | Counter | http_errors_total |
| Saturation | How full your system is | Gauge | system_cpu_utilization, db_connection_pool_used |
Rate: requests per second
Errors: errors per second
Duration: time per request (histogram)
PROMETHEUS QUERIES:
Rate: rate(http_requests_total[5m])
Errors: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
Duration: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))Utilization: % time resource is busy
Saturation: queue depth / backlog
Errors: error count
APPLY TO EACH RESOURCE:
CPU: utilization=usage%, saturation=load average, errors=throttled
Memory: utilization=used/total, saturation=swap usage, errors=OOM kills
Disk I/O: utilization=io_time%, saturation=queue depth, errors=io errors
Network: utilization=bandwidth%, saturation=TCP retransmits, errors=dropped
DB pool: utilization=active/max, saturation=waiting threads, errors=timeoutsSLI: Quantitative measure of service quality ("99.2% of requests < 200ms")
SLO: Target for an SLI that your team commits to ("must be 99.5%")
SLA: Contract with consequences if SLO missed ("below 99.9% = 10% credit")
RELATIONSHIP: SLI measures -> SLO targets -> SLA enforces| Service Type | SLI Category | SLI Definition |
|---|---|---|
| API | Availability | % of requests returning non-5xx in 5-min window |
| API | Latency | % of requests completing in < 200ms |
| API | Correctness | % of responses matching expected schema |
| Pipeline | Freshness | Time since last successful pipeline run |
| Pipeline | Completeness | % of expected records processed |
| Storage | Durability | % of objects retrievable after write |
| Storage | Throughput | % of read requests served within 50ms |
SLO: 99.9% availability (monthly)
Total minutes in month: 43,200 (30 days)
Error budget: 43,200 x 0.1% = 43.2 minutes of allowed downtime
BUDGET POLICY:
Budget > 75% remaining: Ship features freely
Budget 25-75% remaining: Normal caution, monitor closely
Budget < 25% remaining: Freeze risky deployments, focus on reliability
Budget exhausted: Full feature freeze until next period
BURN RATE:
If you burn 10% of monthly budget in 1 hour:
That is 10x the sustainable rate
Alert: PAGE the on-call engineer immediately
If you burn 2% of monthly budget in 6 hours:
That is 2.4x the sustainable rate
Alert: Create a ticket, investigate during business hoursGOOD ALERTS:
- Actionable: Someone can do something about it RIGHT NOW
- Relevant: It affects users or will soon affect users
- Urgent: It cannot wait until business hours (for pages)
- Novel: Not firing constantly (alert fatigue)
BAD ALERTS:
- "CPU is at 80%": So what? Is it affecting users?
- "Disk is 90% full": Page-worthy at 3 AM? Probably not.
- "Service restarted": If health checks pass, it recovered. Don't page.
- "Error rate > 0": Some errors are normal. Alert on rate, not existence.# Prometheus alerting rules
groups:
- name: slo-alerts
rules:
# Fast burn: 2% budget consumed in 1 hour (14.4x burn rate)
- alert: HighErrorBurnRate_Fast
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/ sum(rate(http_requests_total[1h]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
annotations:
summary: "Burning error budget 14x faster than sustainable"
# Slow burn: 5% budget consumed in 6 hours (6x burn rate)
- alert: HighErrorBurnRate_Slow
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[6h]))
/ sum(rate(http_requests_total[6h]))
) > (6 * 0.001)
for: 15m
labels:
severity: warning
annotations:
summary: "Burning error budget 6x faster than sustainable"LEVEL 1: Service Overview (for everyone)
- Current error rate (big number)
- Request rate (big number)
- P50 / P95 / P99 latency (big numbers)
- Error budget remaining (gauge)
- Active alerts (list)
LEVEL 2: Service Detail (for on-call)
- Error rate by endpoint
- Latency distribution by endpoint
- Dependency health (DB, cache, external APIs)
- Resource utilization (CPU, memory, connections)
- Recent deployments (vertical markers)
LEVEL 3: Debug (for investigations)
- Individual request traces
- Log stream (filtered by trace ID)
- Slow query analysis
- Error stack traces
- Resource saturation detailMonitoring without observability: 500 dashboards, 2000 alerts, and you still cannot debug a production issue. Focus on correlation (traces + logs + metrics) over volume.
Alert fatigue: More than 5 pages per week means your alerts are too noisy. Every alert should be actionable and urgent. Non-urgent issues go to tickets, not pages.
Logging everything: Logging every request body at INFO level costs thousands in storage. Log at the right level. Use sampling for high-volume traces.
Vanity metrics on dashboards: CPU utilization looks impressive but rarely helps debug issues. Dashboard space is expensive (attention). Show signals that help you act.
No trace context propagation: Traces that stop at service boundaries are useless for distributed debugging. Ensure W3C Trace Context headers propagate through every hop.
OpenTelemetry SDK configured for traces, metrics, and logs
Auto-instrumentation enabled for HTTP, database, and cache clients
Custom spans added for critical business operations
Structured logging with trace ID correlation
Four Golden Signals dashboards for each service
SLIs defined for each user-facing service
SLOs set with error budget tracking
Multi-burn-rate alerts configured (fast and slow)
Three-level dashboard hierarchy (overview, detail, debug)
OTel Collector deployed with sampling and filtering
Runbook links attached to every alert
Use this skill when:
Designing or implementing observability engineer solutions
Reviewing or improving existing observability engineer approaches
Making architectural or implementation decisions about observability engineer
Learning observability engineer patterns and best practices
Troubleshooting observability engineer-related issues
Do NOT use this skill when:
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
# Observability Engineer Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement observability engineer for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended observability engineer approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When observability engineer must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
562 lines · 2,454 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Observability Engineer bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for