Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Synthetic data generation covering LLM-based data creation, statistical data synthesis, privacy-preserving techniques (differential privacy, k-anonymity), tabular data generation, text augmentation, image augmentation, validation frameworks, and quality assurance for generated datasets.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
381 lines · 1,614 words
Synthetic data generation creates artificial datasets that preserve the statistical properties of real data without exposing sensitive information. This skill covers LLM-based text generation, statistical tabular synthesis, privacy-preserving methods, augmentation strategies, and validation to ensure synthetic data is useful and safe.
Synthetic data generation covering LLM-based data creation, statistical data synthesis, privacy-preserving techniques (differential privacy, k-anonymity), tabular data generation, text augmentation, image augmentation, validation frameworks, and quality assurance for generated datasets.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Synthetic Data Generator skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
Overview
When to Use Synthetic Data
Tabular Data Generation
LLM-Based Text Generation
Text Augmentation
Image Augmentation
Privacy-Preserving Generation
Validation Framework
Cost and Performance
Checklist
Output Format
Example
Edge Cases
| Use Case | Why Synthetic | Risk Level |
|---|
| ML training augmentation | Insufficient labeled data | Low |
| Software testing | Need realistic test fixtures | Low |
| Data sharing across teams | Privacy regulations (GDPR, HIPAA) | Medium |
| Bias correction | Underrepresented groups | Medium |
| Rare event simulation | Insufficient edge case examples | Low |
| Public datasets / demos | Cannot expose real customer data | High |
Privacy / compliance:
Sharing externally? -> Full synthetic with differential privacy
Internal only? -> Pseudonymization may suffice
Training data:
Enough real data? -> Augmentation for diversity / class balance
Insufficient? -> Full synthetic generation to bootstrap
Testing / development:
Need realistic distributions? -> Statistical synthesis from real profile
Just need structure? -> Rule-based Faker data
Research / demos:
Must look realistic? -> LLM-generated + statistical synthesis
Just need volume? -> Simple random generationfrom sdv.metadata import Metadata
from sdv.single_table import GaussianCopulaSynthesizer, CTGANSynthesizer
import pandas as pd
class TabularSynthesizer:
"""Generate synthetic tabular data preserving statistical properties."""
def __init__(self, method: str = "gaussian_copula"):
self.method = method
self.synthesizer = None
def fit(self, real_data: pd.DataFrame) -> None:
metadata = Metadata.detect_from_dataframe(real_data)
if self.method == "gaussian_copula":
self.synthesizer = GaussianCopulaSynthesizer(metadata)
elif self.method == "ctgan":
self.synthesizer = CTGANSynthesizer(
metadata, epochs=300, batch_size=500,
generator_dim=(256, 256), discriminator_dim=(256, 256),
)
self.synthesizer.fit(real_data)
def generate(self, n_rows: int) -> pd.DataFrame:
return self.synthesizer.sample(num_rows=n_rows)| Method | Speed | Quality | Privacy | Best For |
|---|---|---|---|---|
| Gaussian Copula | Fast | Good | Medium | Continuous data, prototyping |
| CTGAN | Slow | Very Good | Medium | Mixed types, complex distributions |
| TVAE | Medium | Good | Medium | Continuous-heavy datasets |
| Faker (rule-based) | Very Fast | Low-Medium | High | Testing, structure only |
from faker import Faker
import random
import pandas as pd
fake = Faker()
Faker.seed(42)
def generate_customer_data(n: int = 1000) -> pd.DataFrame:
records = []
for _ in range(n):
records.append({
"customer_id": fake.uuid4(),
"name": fake.name(),
"email": fake.email(),
"city": fake.city(),
"signup_date": fake.date_between(start_date="-3y", end_date="today"),
"plan": random.choices(
["free", "basic", "pro", "enterprise"], weights=[0.5, 0.3, 0.15, 0.05],
)[0],
"monthly_spend": round(random.lognormvariate(3.5, 1.2), 2),
"is_active": random.random() > 0.15,
})
return pd.DataFrame(records)from openai import OpenAI
import json
class LLMDataGenerator:
"""Generate synthetic text data using LLMs."""
def __init__(self, model: str = "gpt-4o-mini"):
self.client = OpenAI()
self.model = model
def generate_batch(
self, schema: dict, n_samples: int,
context: str = "", examples: list[dict] = None, batch_size: int = 10,
) -> list[dict]:
all_samples = []
for i in range(0, n_samples, batch_size):
current_batch = min(batch_size, n_samples - i)
prompt = (
f"Generate exactly {current_batch} synthetic data samples.\n\n"
f"Schema:\n```json\n{json.dumps(schema, indent=2)}\n```\n\n"
)
if context:
prompt += f"Context: {context}\n\n"
if examples:
prompt += f"Examples:\n{json.dumps(examples[:3], indent=2)}\n\n"
prompt += "Return as JSON: {\"samples\": [...]}\n"
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": "Generate realistic, diverse data matching the schema."},
{"role": "user", "content": prompt},
],
response_format={"type": "json_object"},
temperature=1.0,
)
batch = json.loads(response.choices[0].message.content)
all_samples.extend(batch.get("samples", []))
return all_samples[:n_samples]| Technique | Quality | Speed | Best For |
|---|---|---|---|
| Synonym replacement | Medium | Fast | Simple vocabulary variation |
| Random deletion | Low-Medium | Fast | Robustness training |
| Back-translation | High | Slow | Natural paraphrasing |
| LLM paraphrase | Very High | Slow | High-quality augmentation |
| Entity swapping | Medium | Fast | Named entity variation |
class TextAugmenter:
def synonym_replace(self, text: str, n: int = 2) -> str:
"""Replace n random words with synonyms (uses WordNet)."""
words = text.split()
import random
indices = random.sample(range(len(words)), min(n, len(words)))
for idx in indices:
syn = self._get_synonym(words[idx])
if syn:
words[idx] = syn
return " ".join(words)
def random_deletion(self, text: str, p: float = 0.1) -> str:
"""Randomly delete words with probability p."""
import random
words = text.split()
remaining = [w for w in words if random.random() > p]
return " ".join(remaining) if remaining else words[0]
def llm_paraphrase(self, text: str, style: str = "neutral") -> str:
"""Use an LLM to paraphrase while preserving meaning."""
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": f"Paraphrase in {style} style:\n{text}"}],
temperature=0.8,
)
return response.choices[0].message.content.strip()import albumentations as A
from albumentations.pytorch import ToTensorV2
def get_training_augmentation(task: str = "classification") -> A.Compose:
if task == "classification":
return A.Compose([
A.RandomResizedCrop(224, 224, scale=(0.8, 1.0)),
A.HorizontalFlip(p=0.5),
A.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1, p=0.8),
A.GaussianBlur(blur_limit=(3, 7), p=0.3),
A.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
ToTensorV2(),
])
elif task == "detection":
return A.Compose([
A.RandomResizedCrop(640, 640, scale=(0.5, 1.0)),
A.HorizontalFlip(p=0.5),
A.RandomBrightnessContrast(p=0.5),
A.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
ToTensorV2(),
], bbox_params=A.BboxParams(format="pascal_voc", min_visibility=0.3, label_fields=["labels"]))import numpy as np
class DPMechanism:
@staticmethod
def laplace_mechanism(true_value: float, sensitivity: float, epsilon: float) -> float:
"""Add Laplace noise for epsilon-differential privacy."""
return true_value + np.random.laplace(0, sensitivity / epsilon)
@staticmethod
def dp_histogram(data: np.ndarray, bins: int, epsilon: float) -> tuple:
"""Create differentially private histogram."""
counts, edges = np.histogram(data, bins=bins)
noisy = np.array([max(0, c + np.random.laplace(0, 2 / epsilon)) for c in counts])
return noisy, edgesEpsilon guidelines:
< 1.0 -> Strong privacy (significant noise)
1-5 -> Moderate privacy (typical)
5-10 -> Weak privacy (minimal noise)
> 10 -> Negligible privacyimport pandas as pd
def check_k_anonymity(data: pd.DataFrame, quasi_ids: list[str], k: int = 5) -> dict:
group_sizes = data.groupby(quasi_ids).size()
min_group = group_sizes.min()
violating = (group_sizes < k).sum()
return {
"satisfies": min_group >= k,
"k_actual": int(min_group),
"violating_groups": int(violating),
}from scipy import stats
import pandas as pd
import numpy as np
class SyntheticDataValidator:
def __init__(self, real: pd.DataFrame, synthetic: pd.DataFrame):
self.real = real
self.synthetic = synthetic
def validate_distributions(self) -> dict:
results = {}
for col in self.real.columns:
if col not in self.synthetic.columns:
continue
if self.real[col].dtype in ["float64", "int64"]:
stat, p = stats.ks_2samp(self.real[col].dropna(), self.synthetic[col].dropna())
results[col] = {"test": "ks", "stat": round(stat, 4), "p": round(p, 4), "ok": p > 0.05}
return results
def validate_correlations(self) -> dict:
cols = [c for c in self.real.select_dtypes(include=[np.number]).columns if c in self.synthetic.columns]
real_corr = self.real[cols].corr()
synth_corr = self.synthetic[cols].corr()
diff = np.linalg.norm(real_corr.values - synth_corr.values, "fro")
norm_diff = diff / (np.linalg.norm(real_corr.values, "fro") * 2)
return {"diff": round(diff, 4), "normalized": round(norm_diff, 4),
"quality": "good" if norm_diff < 0.1 else "moderate" if norm_diff < 0.25 else "poor"}
def validate_utility(self, target: str) -> dict:
"""Train-on-synthetic, test-on-real (TSTR) evaluation."""
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score
feats = [c for c in self.real.select_dtypes(include=[np.number]).columns if c != target]
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(self.synthetic[feats].fillna(0), self.synthetic[target])
pred = model.predict(self.real[feats].fillna(0))
tstr = f1_score(self.real[target], pred, average="macro")
return {"tstr_score": round(tstr, 4)}| Method | Speed | Setup Cost | Quality | Privacy |
|---|---|---|---|---|
| Faker | 100K+ rows/sec | Low | Low | High |
| Gaussian Copula | 10K rows/sec | Low | Good | Medium |
| CTGAN | 100 rows/sec | Medium (GPU) | Very Good | Medium |
| LLM generation | 10-50 rows/min | Low (API cost) | Excellent | High |
Define purpose: training augmentation, privacy, testing, or sharing
Profile the real dataset (distributions, correlations, constraints)
Select generation method matching quality and compute budget
Implement pipeline with reproducible seeds
Validate statistical fidelity (distribution and correlation tests)
Run utility tests (train-on-synthetic, test-on-real)
Verify privacy guarantees (k-anonymity, differential privacy)
Check for data leakage (no real records copied verbatim)
Document methodology, parameters, and validation results
Establish monitoring if synthetic data is regenerated periodically
# Synthetic Data Generator Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement synthetic data generator for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended synthetic data generator approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When synthetic data generator must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
381 lines · 1,614 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Synthetic Data Generator bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for