Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
File processing pipeline expertise covering upload handling (multipart, streaming, resumable), virus scanning, image processing, PDF generation, storage backends, presigned URLs, and file type validation.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
415 lines · 1,622 words
Design and implement robust file processing pipelines that handle upload, validation, processing, storage, and delivery. This skill covers upload strategies, security considerations, media processing, cloud storage patterns, and production operational concerns.
File processing pipeline expertise covering upload handling (multipart, streaming, resumable), virus scanning, image processing, PDF generation, storage backends, presigned URLs, and file type validation.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the File Processor skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
Purpose
Upload Strategy Selection
Multipart Upload Handling
Presigned URL Upload
Resumable Upload (tus Protocol)
File Type Validation
Virus Scanning
Image Processing (Sharp)
PDF Generation
Storage Backends
CDN Integration
File Processing Pipeline
Cleanup and Lifecycle
File Processing Checklist
When to Use
Output Format
Example
Edge Cases
SCENARIO STRATEGY
------------------------------------------------------
Small files (<10MB) Multipart form upload
Large files (>10MB) Presigned URL (direct-to-storage)
Very large files (>1GB) Resumable upload (tus protocol)
Multiple small files Batch multipart
Sensitive files Server-side upload (no direct)
User avatars/profile images Multipart + server processing
Video uploads Resumable + async processing
API file attachments Base64 in JSON (small) or multipart
PRESIGNED URL FLOW (recommended for large files):
1. Client requests upload URL from API
2. API generates presigned URL (S3, GCS, Azure)
3. Client uploads directly to storage
4. Storage notifies API (webhook or client confirms)
5. API processes file (resize, scan, etc.)import multer from 'multer';
import path from 'path';
// Configuration
const upload = multer({
storage: multer.memoryStorage(),
limits: {
fileSize: 10 * 1024 * 1024, // 10MB max
files: 5, // Max 5 files per request
fields: 10, // Max 10 non-file fields
},
fileFilter: (req, file, cb) => {
const allowedTypes = [
'image/jpeg', 'image/png', 'image/webp', 'image/gif',
# ... (condensed) ...
} catch (error) {
res.status(422).json({ error: error.message });
}
}
);import { S3Client, PutObjectCommand, HeadObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
const s3 = new S3Client({ region: ENV_CONFIG_VALUE });
// Generate presigned upload URL
app.post('/api/upload/presign', authenticate, async (req, res) => {
const { filename, contentType, size } = req.body;
// Validate before generating URL
if (size > 100 * 1024 * 1024) {
return res.status(400).json({ error: 'File too large (max 100MB)' });
}
# ... (condensed) ...
await db.uploads.update({ where: { id: upload.id }, data: { status: 'processing' } });
res.json({ status: 'processing' });
});import { Server as TusServer } from '@tus/server';
import { S3Store } from '@tus/s3-store';
const tusServer = new TusServer({
path: '/api/upload/resumable',
datastore: new S3Store({
s3ClientConfig: {
region: ENV_CONFIG_VALUE,
},
bucket: ENV_CONFIG_VALUE!,
partSize: 8 * 1024 * 1024,
}),
maxSize: 5 * 1024 * 1024 * 1024, // 5GB max
onUploadCreate: async (req, res, upload) => {
# ... (condensed) ...
return res;
},
});
app.all('/api/upload/resumable/*', (req, res) => tusServer.handle(req, res));import { fileTypeFromBuffer } from 'file-type';
async function validateFileContent(file: { buffer: Buffer; mimetype: string }): Promise<void> {
// Check magic bytes (actual file content)
const detected = await fileTypeFromBuffer(file.buffer);
if (!detected) {
throw new Error('Unable to determine file type');
}
const allowedMimeTypes: Record<string, string[]> = {
'image/jpeg': ['jpg', 'jpeg'],
'image/png': ['png'],
'image/webp': ['webp'],
# ... (condensed) ...
const pixels = (metadata.width ?? 0) * (metadata.height ?? 0);
if (pixels > 100_000_000) {
throw new Error('Image too large (decompression bomb protection)');
}
}import NodeClam from 'clamscan';
const clam = await new NodeClam().init({
clamdscan: {
socket: '/var/run/clamav/clamd.ctl',
timeout: 60000,
multiscan: true,
},
preference: 'clamdscan',
});
async function scanBuffer(buffer: Buffer): Promise<{ clean: boolean; virus?: string }> {
try {
const { Readable } = await import('stream');
# ... (condensed) ...
}
// 3. Process and store
return await storeFile(file);
}import sharp from 'sharp';
// Generate multiple image sizes
async function processImage(
input: Buffer,
options: { maxWidth?: number; maxHeight?: number; quality?: number; format?: 'webp' | 'avif' }
): Promise<ProcessedImage> {
const { quality = 80, format = 'webp' } = options;
const sizes = [
{ name: 'thumbnail', width: 200, height: 200, fit: 'cover' as const },
{ name: 'small', width: 400, height: 400, fit: 'inside' as const },
{ name: 'medium', width: 800, height: 800, fit: 'inside' as const },
{ name: 'large', width: 1600, height: 1600, fit: 'inside' as const },
# ... (condensed) ...
return sharp(input)
.rotate() // Auto-rotate based on EXIF before stripping
.withMetadata({ orientation: undefined })
.toBuffer();
}import PDFDocument from 'pdfkit';
async function generateInvoicePDF(invoice: Invoice): Promise<Buffer> {
return new Promise((resolve, reject) => {
const doc = new PDFDocument({ margin: 50 });
const chunks: Buffer[] = [];
doc.on('data', (chunk: Buffer) => chunks.push(chunk));
doc.on('end', () => resolve(Buffer.concat(chunks)));
doc.on('error', reject);
// Header
doc.fontSize(24).text('INVOICE', { align: 'right' });
doc.fontSize(10).text(`Invoice #${invoice.number}`, { align: 'right' });
# ... (condensed) ...
doc.text(formatCurrency(invoice.total), 460, y + 15);
doc.end();
});
}import { S3Client, PutObjectCommand, GetObjectCommand, DeleteObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
class FileStorage {
private s3: S3Client;
private bucket: string;
constructor() {
this.s3 = new S3Client({ region: ENV_CONFIG_VALUE });
this.bucket = ENV_CONFIG_VALUE!;
}
async upload(key: string, buffer: Buffer, contentType: string): Promise<string> {
await this.s3.send(new PutObjectCommand({
# ... (condensed) ...
Bucket: this.bucket,
Key: key,
}));
}
}KEY STRUCTURE:
{type}/{user_id}/{year}/{month}/{file_id}{extension}
Example: avatars/usr_abc123/2025/01/f7a9b3c4.webp
uploads/{user_id}/{file_id}/{variant}.{format}
Example: uploads/usr_abc123/f7a9b3c4/thumbnail.webp
uploads/usr_abc123/f7a9b3c4/medium.webp
uploads/usr_abc123/f7a9b3c4/original.webp
RULES:
- NEVER use original filename in storage key (security)
- Generate unique IDs (UUID, ULID)
- Include user ID for access control and cleanup
- Date-based prefix helps with S3 partitioning
- Keep extensions for CDN content-type detectionclass CDNService {
private readonly cdnDomain = ENV_CONFIG_VALUE;
getPublicUrl(key: string): string {
return `[reference URL]
}
// Image transformation via CDN (Cloudflare, Imgix, etc.)
getImageUrl(key: string, transforms: ImageTransform): string {
const params = new URLSearchParams();
if (transforms.width) params.set('w', String(transforms.width));
if (transforms.height) params.set('h', String(transforms.height));
if (transforms.quality) params.set('q', String(transforms.quality));
if (transforms.format) params.set('f', transforms.format);
params.set('fit', transforms.fit ?? 'cover');
return `[reference URL])}/${key}`;
}
}import { Queue, Worker } from 'bullmq';
const processingQueue = new Queue('file-processing');
const worker = new Worker('file-processing', async (job) => {
const { uploadId, key, contentType } = job.data;
// 1. Download from storage
const fileBuffer = await storage.download(key);
// 2. Virus scan
await job.updateProgress(10);
const scanResult = await scanBuffer(fileBuffer);
if (!scanResult.clean) {
# ... (condensed) ...
await alerting.notify(`File processing permanently failed: ${job.data.uploadId}`, {
error: error.message,
});
}
});// Clean up orphaned uploads (pending > 24 hours)
async function cleanupOrphanedUploads() {
const cutoff = new Date(Date.now() - 24 * 60 * 60 * 1000);
const orphans = await db.uploads.findMany({
where: { status: 'pending', createdAt: { lt: cutoff } },
});
for (const upload of orphans) {
try {
await storage.remove(upload.key);
} catch {
// File may not exist, that is fine
}
# ... (condensed) ...
// S3 lifecycle policy (set via AWS console or Terraform)
// - Move to Glacier after 90 days
// - Delete incomplete multipart uploads after 7 days
// - Delete expired presigned upload objects after 1 dayUpload strategy selected (multipart, presigned URL, resumable)
File size limits enforced (client-side and server-side)
File type validated by content (magic bytes), not just extension
Virus scanning integrated into processing pipeline
Image decompression bomb protection implemented
Images processed into multiple sizes/formats (WebP/AVIF)
EXIF metadata stripped for privacy
Storage keys use random IDs (never user-provided filenames)
Presigned URLs have appropriate expiration times
CDN configured for file delivery with cache headers
Processing pipeline runs asynchronously (queue-based)
Failed processing triggers alerts and retry
File cleanup job handles orphaned uploads
Storage lifecycle policies configured for cost optimization
Access control verified before file delivery
Use this skill when:
Designing or implementing file processor solutions
Reviewing or improving existing file processor approaches
Making architectural or implementation decisions about file processor
Learning file processor patterns and best practices
Troubleshooting file processor-related issues
Do NOT use this skill when:
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
# File Processor Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement file processor for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended file processor approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When file processor must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
415 lines · 1,622 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
File Processor bot
Waiting in your sidebar once you say go.