Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Computer vision implementation covering image classification, object detection (YOLO), image segmentation, OCR (Tesseract), face detection, image preprocessing, data augmentation, transfer learning, model deployment, and edge inference.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
538 lines · 1,769 words
Computer vision enables machines to interpret and act on visual data. This skill covers practical implementation of core CV tasks -- classification, detection, segmentation, and OCR -- using modern frameworks and pre-trained models, with guidance on data augmentation, transfer learning, and deployment.
Computer vision implementation covering image classification, object detection (YOLO), image segmentation, OCR (Tesseract), face detection, image preprocessing, data augmentation, transfer learning, model deployment, and edge inference.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Computer Vision skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
Overview
Image Preprocessing
Data Augmentation
Image Classification
Object Detection
Image Segmentation
OCR (Optical Character Recognition)
Face Detection
Model Deployment
Checklist
When to Use
Output Format
Example
Edge Cases
import cv2
import numpy as np
from PIL import Image
import torchvision.transforms as T
class ImagePreprocessor:
"""Standard image preprocessing for CV models."""
def __init__(self, target_size: tuple = (224, 224), normalize: bool = True):
self.target_size = target_size
self.normalize = normalize
def preprocess_opencv(self, image_path: str) -> np.ndarray:
"""Preprocess with OpenCV."""
img = cv2.imread(image_path)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
# Resize
# ... (condensed) ...
transforms.extend([
T.ToTensor(),
T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
return T.Compose(transforms)| Operation | When to Use | Library |
|---|---|---|
| Resize | Always (model input size) | cv2, PIL, torchvision |
| Normalize | Always (ImageNet stats for pretrained) | torchvision |
| Grayscale | OCR, edge detection | cv2 |
| Histogram equalization | Low contrast images | cv2 (CLAHE) |
| Denoising | Noisy images | cv2.fastNlMeansDenoising |
| Crop | Focus on region of interest | cv2, PIL |
import albumentations as A
from albumentations.pytorch import ToTensorV2
def get_training_augmentations(image_size: int = 224) -> A.Compose:
"""Production training augmentation pipeline."""
return A.Compose([
A.RandomResizedCrop(height=image_size, width=image_size, scale=(0.8, 1.0)),
A.HorizontalFlip(p=0.5),
A.VerticalFlip(p=0.1),
A.ShiftScaleRotate(
shift_limit=0.1, scale_limit=0.15, rotate_limit=15, p=0.5
),
A.OneOf([
A.GaussNoise(var_limit=(10.0, 50.0)),
A.GaussianBlur(blur_limit=(3, 7)),
A.MotionBlur(blur_limit=5),
], p=0.3),
A.OneOf([
# ... (condensed) ...
A.RandomResizedCrop(height=image_size, width=image_size, scale=(0.5, 1.0)),
A.HorizontalFlip(p=0.5),
A.RandomBrightnessContrast(p=0.3),
A.HueSaturationValue(p=0.3),
A.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
ToTensorV2(),
], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['labels']))Dataset size < 500 images:
-> Heavy augmentation + transfer learning essential
-> Use MixUp, CutMix, Mosaic
-> Consider synthetic data generation
Dataset size 500-5000:
-> Moderate augmentation
-> Standard flips, crops, color jitter
-> Transfer learning highly recommended
Dataset size > 5000:
-> Light augmentation
-> Basic flips and normalization
-> Can train from scratch for simple tasksimport torch
import torch.nn as nn
from torchvision import models
def create_classifier(
num_classes: int,
backbone: str = "resnet50",
pretrained: bool = True,
freeze_backbone: bool = True,
) -> nn.Module:
"""Create a classifier with transfer learning."""
if backbone == "resnet50":
model = models.resnet50(weights="IMAGENET1K_V2" if pretrained else None)
num_features = model.fc.in_features
model.fc = nn.Sequential(
nn.Dropout(0.3),
nn.Linear(num_features, num_classes),
# ... (condensed) ...
# Freeze backbone layers
if freeze_backbone:
for param in list(model.parameters())[:-2]:
param.requires_grad = False
return modelfrom transformers import pipeline, AutoModelForImageClassification, AutoImageProcessor
# Quick inference with pipeline
classifier = pipeline("image-classification", model="google/vit-base-patch16-224")
result = classifier("photo.jpg")
# [{"label": "golden retriever", "score": 0.95}, ...]
# Fine-tuning
from transformers import TrainingArguments, Trainer
from datasets import load_dataset
dataset = load_dataset("imagefolder", data_dir="./data")
model = AutoModelForImageClassification.from_pretrained(
"google/vit-base-patch16-224",
num_labels=len(dataset["train"].features["label"].names),
ignore_mismatched_sizes=True,
)
# ... (condensed) ...
load_best_model_at_end=True,
),
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
)
trainer.train()| Model | Params | Accuracy (ImageNet) | Speed | Best For |
|---|---|---|---|---|
| MobileNetV3-Small | 2.5M | 67.4% | Very fast | Mobile/edge |
| EfficientNet-B0 | 5.3M | 77.1% | Fast | Balanced |
| ResNet-50 | 25M | 80.4% | Medium | General purpose |
| EfficientNet-V2-S | 21M | 84.2% | Medium | High accuracy |
| ViT-B/16 | 86M | 84.5% | Slower | Max accuracy |
| ConvNeXt-Base | 88M | 85.8% | Slower | SOTA accuracy |
from ultralytics import YOLO
# Load pretrained model
model = YOLO("yolov8n.pt") # nano (fastest)
# model = YOLO("yolov8s.pt") # small
# model = YOLO("yolov8m.pt") # medium
# model = YOLO("yolov8l.pt") # large
# Inference
results = model("image.jpg")
for result in results:
boxes = result.boxes
for box in boxes:
cls = int(box.cls[0])
conf = float(box.conf[0])
xyxy = box.xyxy[0].tolist() # [x1, y1, x2, y2]
label = model.names[cls]
# ... (condensed) ...
patience=20,
device="0", # GPU
)
# Export for deployment
model.export(format="onnx")
model.export(format="tflite")# dataset.yaml
path: /path/to/dataset
train: images/train
val: images/val
test: images/test
names:
0: person
1: car
2: bicycle# Label format (one .txt per image)
# class_id center_x center_y width height (all normalized 0-1)
0 0.5 0.6 0.3 0.4
1 0.2 0.3 0.15 0.2| Model | mAP@50 (COCO) | Speed (ms) | Params | Best For |
|---|---|---|---|---|
| YOLOv8n | 37.3 | 1.2 | 3.2M | Real-time, edge |
| YOLOv8s | 44.9 | 2.0 | 11.2M | Balanced |
| YOLOv8m | 50.2 | 4.2 | 25.9M | High accuracy |
| YOLOv8x | 53.9 | 8.7 | 68.2M | Max accuracy |
| RT-DETR-L | 53.0 | 9.3 | 32M | Transformer-based |
from transformers import pipeline
segmenter = pipeline("image-segmentation", model="nvidia/segformer-b0-finetuned-ade-512-512")
results = segmenter("street_scene.jpg")
for segment in results:
print(f"{segment['label']}: score={segment['score']:.2f}")from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator
# Load SAM model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth")
sam.to("cuda")
# Automatic mask generation
mask_generator = SamAutomaticMaskGenerator(sam)
masks = mask_generator.generate(image)
# Point-prompted segmentation
predictor = SamPredictor(sam)
predictor.set_image(image)
# Click a point to segment the object there
masks, scores, logits = predictor.predict(
point_coords=np.array([[500, 375]]),
point_labels=np.array([1]), # 1 = foreground
multimask_output=True,
)
# Use the highest-scoring mask
best_mask = masks[np.argmax(scores)]from ultralytics import YOLO
model = YOLO("yolov8n-seg.pt") # Instance segmentation
results = model("image.jpg")
for result in results:
if result.masks:
for mask, box in zip(result.masks.data, result.boxes):
cls = int(box.cls[0])
label = model.names[cls]
binary_mask = mask.cpu().numpy() # H x W binary maskimport pytesseract
from PIL import Image
import cv2
def extract_text(image_path: str, preprocess: bool = True) -> str:
"""Extract text from image using Tesseract."""
img = cv2.imread(image_path)
if preprocess:
# Convert to grayscale
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# Adaptive thresholding
binary = cv2.adaptiveThreshold(
gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY, 11, 2
)
# ... (condensed) ...
'x': data['left'][i],
'y': data['top'][i],
'width': data['width'][i],
'height': data['height'][i],
},
})
return resultsimport easyocr
reader = easyocr.Reader(['en', 'fr', 'de'])
results = reader.readtext('document.jpg')
for (bbox, text, confidence) in results:
print(f"'{text}' (confidence: {confidence:.2f})")import mediapipe as mp
mp_face = mp.solutions.face_detection
mp_drawing = mp.solutions.drawing_utils
def detect_faces(image_path: str) -> list[dict]:
"""Detect faces using MediaPipe."""
with mp_face.FaceDetection(model_selection=1, min_detection_confidence=0.5) as detector:
image = cv2.imread(image_path)
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
results = detector.process(rgb)
faces = []
if results.detections:
h, w = image.shape[:2]
for detection in results.detections:
bbox = detection.location_data.relative_bounding_box
faces.append({
"confidence": detection.score[0],
"bbox": {
"x": int(bbox.xmin * w),
"y": int(bbox.ymin * h),
"width": int(bbox.width * w),
"height": int(bbox.height * h),
},
})
return facesimport torch
def export_to_onnx(model, sample_input, output_path: str = "model.onnx"):
"""Export PyTorch model to ONNX format."""
model.cpu()
torch.onnx.export(
model,
sample_input,
output_path,
export_params=True,
opset_version=17,
input_names=['input'],
output_names=['output'],
dynamic_axes={
'input': {0: 'batch_size'},
'output': {0: 'batch_size'},
},
)
# ... (condensed) ...
session = ort.InferenceSession(
model_path,
providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)
input_name = session.get_inputs()[0].name
output = session.run(None, {input_name: input_data})
return output[0]import tensorflow as tf
def convert_to_tflite(saved_model_path: str, output_path: str, quantize: bool = True):
"""Convert model to TFLite for edge deployment."""
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
if quantize:
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]
tflite_model = converter.convert()
with open(output_path, 'wb') as f:
f.write(tflite_model)Target platform?
CLOUD (GPU available):
-> PyTorch/TensorRT (best performance)
-> ONNX Runtime (portable)
EDGE (mobile, IoT):
-> TFLite (Android, Raspberry Pi)
-> CoreML (iOS, macOS)
-> ONNX Runtime Mobile
BROWSER:
-> TensorFlow.js
-> ONNX Runtime Web
Latency requirement?
< 10ms: TensorRT or hardware-specific optimization
10-100ms: ONNX Runtime with GPU
> 100ms: Any framework worksSet up image preprocessing pipeline with proper normalization
Implement data augmentation appropriate to dataset size
Choose model architecture based on accuracy vs speed tradeoffs
Use transfer learning for datasets under 10K images
Set up proper train/val/test splits
Track training metrics with experiment tracker
Export model to deployment format (ONNX, TFLite, TensorRT)
Benchmark inference latency on target hardware
Implement preprocessing in the deployment pipeline (not just training)
Test with edge cases (poor lighting, occlusion, unusual angles)
Monitor model performance in production
Use this skill when:
Designing or implementing computer vision solutions
Reviewing or improving existing computer vision approaches
Making architectural or implementation decisions about computer vision
Learning computer vision patterns and best practices
Troubleshooting computer vision-related issues
Do NOT use this skill when:
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
# Computer Vision Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement computer vision for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended computer vision approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When computer vision must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
538 lines · 1,769 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Computer Vision bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for