Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Expert guide for designing reliable background job systems including worker architectures, queue selection, retry strategies, dead letter queues, idempotency patterns, job scheduling, and monitoring for production workloads.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
370 lines · 1,468 words
You are an expert in designing and operating reliable background job processing systems. You guide teams through queue architecture, worker design, retry strategies, idempotency, failure handling, and operational best practices for asynchronous workloads at scale.
Every job will fail eventually - Design for retries, idempotency, and graceful degradation from the start.
Expert guide for designing reliable background job systems including worker architectures, queue selection, retry strategies, dead letter queues, idempotency patterns, job scheduling, and monitoring for production workloads.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Background Job Designer skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
The principle
You are an expert in designing and operating reliable background job processing systems. You guide teams through queue architecture, worker design, retry strategies, idempotency, failure handling, and operational best practices for asynchronous workloads at scale.
What it covers
Core Principles
Queue Technology Selection
Job Design
Retry Strategies
Dead Letter Queues
Idempotency
Job Scheduling
Worker Lifecycle
Monitoring
Production Checklist
When to Use
Output Format
Example
Edge Cases
Separate concerns by queue - Different job types have different latency, throughput, and retry requirements.
TECHNOLOGY BEST FOR ORDERING LATENCY
---------------------------------------------------------------------------
Redis (BullMQ) Web app jobs, ~10K/s Per-queue <10ms
RabbitMQ Complex routing, fanout Per-queue <5ms
Amazon SQS Cloud-native, very high throughput Standard: ~20ms
none
PostgreSQL Simple apps, transactional enqueue Per-query ~50ms
(SKIP LOCKED)
Kafka Event streaming, replay, 100K+/s Per-part. <10msPOSTGRES: Already use PG, <1K jobs/min, need transactional enqueue
REDIS: Sub-second latency, 1K-50K/min, Node.js stack
SQS: Managed infra, >50K/min, multi-language, fanout (SNS+SQS)
KAFKA: Event replay, multiple consumers, >100K/mininterface Job<T = unknown> {
id: string;
type: string;
payload: T;
metadata: {
createdAt: string;
priority: number;
attempt: number;
maxAttempts: number;
idempotencyKey?: string;
correlationId: string;
};
}import { Queue, Worker } from 'bullmq';
const emailQueue = new Queue('email', { connection });
// Enqueue
await emailQueue.add('send-welcome', {
to: 'user@example.com', templateId: 'welcome-v2',
}, {
attempts: 5,
backoff: { type: 'exponential', delay: 1000 },
removeOnComplete: { age: 86400, count: 1000 },
removeOnFail: { age: 604800 },
jobId: `welcome-${userId}`, // Idempotency
});
// Worker
const worker = new Worker('email', async (job) => {
switch (job.name) {
case 'send-welcome': return sendEmail(job.data);
case 'send-receipt': return sendReceipt(job.data);
default: throw new Error(`Unknown job: ${job.name}`);
}
}, {
connection,
concurrency: 10,
limiter: { max: 100, duration: 60_000 }, // Rate limit
});
worker.on('failed', (job, err) => {
logger.error('Job failed', { jobId: job?.id, attempt: job?.attemptsMade, error: err.message });
});CREATE TABLE jobs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
queue VARCHAR(100) NOT NULL DEFAULT 'default',
type VARCHAR(200) NOT NULL,
payload JSONB NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'pending',
priority INTEGER NOT NULL DEFAULT 100,
attempt INTEGER NOT NULL DEFAULT 0,
max_attempts INTEGER NOT NULL DEFAULT 5,
scheduled_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
error_message TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_jobs_fetch ON jobs (queue, priority, scheduled_at)
WHERE status = 'pending';
-- Atomic job get (concurrent-safe)
WITH next_job AS (
SELECT id FROM jobs
WHERE queue = $1 AND status = 'pending' AND scheduled_at <= NOW()
ORDER BY priority ASC, scheduled_at ASC
LIMIT 1 FOR UPDATE SKIP LOCKED
)
UPDATE jobs SET status = 'running', attempt = attempt + 1
FROM next_job WHERE jobs.id = next_job.id
RETURNING jobs.*;STRATEGY FORMULA USE CASE
--------------------------------------------------------------
Fixed delay = 5s Simple, predictable
Exponential delay = 2^attempt * 1s Standard for most jobs
Exp + Jitter delay = random(0, 2^a * 1s) Prevent thundering herd
Custom schedule [1s, 5s, 30s, 300s, 3600s] Specific timing needsfunction classifyError(error: Error): 'retry' | 'fail' | 'discard' {
if (/ECONNRESET|ETIMEDOUT|429|503/.test(error.message)) return 'retry';
if (/400|404|Invalid/.test(error.message)) return 'fail';
if (/duplicate/.test(error.message)) return 'discard';
return 'retry'; // Unknown errors: retry conservatively
}Main Queue -> Worker -> [Success] -> Done
|
+-> [Max retries exceeded] -> Dead Letter Queue
|
+--------------+-----------+
| | |
Alert team Dashboard Store for
| manual review
Manual review -> Replay to main queueasync function handleDeadLetter(job: Job): Promise<void> {
await db.deadLetters.insert({
originalJobId: job.id,
queue: job.queueName,
type: job.name,
payload: job.data,
failureReason: job.failedReason,
attempts: job.attemptsMade,
});
await alerting.send({
channel: '#job-failures',
message: `Job ${job.name} (${job.id}) moved to DLQ after ${job.attemptsMade} attempts`,
});
}
// Replay mechanism
async function replayDeadLetters(filter: { queue?: string; type?: string }): Promise<number> {
const records = await db.deadLetters.find({ ...filter, resolution: null });
let replayed = 0;
for (const record of records) {
const queue = getQueue(record.queue);
await queue.add(record.type, record.payload);
await db.deadLetters.update(record.id, { resolution: 'replayed' });
replayed++;
}
return replayed;
}async function processIdempotently<T>(
key: string, ttl: number, processor: () => Promise<T>,
): Promise<T | null> {
const acquired = await redis.set(`idem:${key}`, 'processing', 'NX', 'EX', ttl);
if (!acquired) {
const result = await redis.get(`idem:${key}:result`);
return result ? JSON.parse(result) : null;
}
try {
const result = await processor();
await redis.set(`idem:${key}:result`, JSON.stringify(result), 'EX', ttl);
return result;
} catch (err) {
await redis.del(`idem:${key}`);
throw err;
}
}// Cron-based repeatable jobs
await reportQueue.add('daily-summary', { reportType: 'daily' }, {
repeat: { pattern: '0 6 * * *', tz: 'America/New_York' },
jobId: 'daily-summary',
});
// Delayed jobs (e.g., auto-cancel unpaid order after 30 min)
await orderQueue.add('auto-cancel', { orderId: order.id }, {
delay: 30 * 60 * 1000,
jobId: `cancel-${order.id}`,
});
// Cancel delayed job if payment arrives
const job = await orderQueue.getJob(`cancel-${order.id}`);
if (job) await job.remove();async function shutdown(signal: string) {
logger.info(`Received ${signal}, shutting down...`);
await worker.close(); // Stop accepting new jobs
await connection.quit(); // Close Redis
process.exit(0);
}
process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));-- PostgreSQL: reap stalled jobs (workers that died mid-processing)
UPDATE jobs
SET status = CASE WHEN attempt >= max_attempts THEN 'dead' ELSE 'pending' END,
error_message = 'Job stalled (worker timeout)',
scheduled_at = NOW() + INTERVAL '30 seconds'
WHERE status = 'running'
AND started_at < NOW() - INTERVAL '600 seconds';METRIC ALERT CONDITION
-------------------------------------------------
Queue depth > 10,000 sustained 5min
Processing time (p99) > 2x baseline
Error rate > 5% of processed
DLQ size > 0 new entries
Stalled jobs > 0
Worker utilization > 90% sustainedDESIGN:
[ ] Jobs are idempotent
[ ] Jobs carry all data needed (no stale state fetching)
[ ] Separate queues for different priorities
[ ] Payload serializable and under size limits
RELIABILITY:
[ ] Exponential backoff with jitter
[ ] Error classification (transient vs permanent)
[ ] Dead letter queue with alerting and replay
[ ] Stalled job detection and recovery
[ ] Graceful shutdown with in-flight completion
OPERATIONS:
[ ] Structured logging with job ID and correlation ID
[ ] Queue depth monitoring and alerting
[ ] Processing time and error rate dashboards
[ ] DLQ review process defined
[ ] Job retention and cleanup policyUse this skill when:
Designing or implementing background job designer solutions
Reviewing or improving existing background job designer approaches
Making architectural or implementation decisions about background job designer
Learning background job designer patterns and best practices
Troubleshooting background job designer-related issues
Do NOT use this skill when:
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
# Background Job Designer Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement background job designer for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended background job designer approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When background job designer must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
370 lines · 1,468 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Background Job Designer bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for