Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Model fine-tuning covering dataset preparation, LoRA and QLoRA, instruction tuning, RLHF and DPO, benchmarking, overfitting prevention, compute requirements, Hugging Face Trainer, and the fine-tuning vs prompt engineering decision.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
527 lines · 1,969 words
Fine-tuning adapts a pre-trained model to a specific task or domain by training on curated data. This skill covers when to fine-tune vs use prompt engineering, dataset preparation, parameter-efficient methods (LoRA/QLoRA), alignment techniques (RLHF/DPO), and practical training with Hugging Face tooling.
Model fine-tuning covering dataset preparation, LoRA and QLoRA, instruction tuning, RLHF and DPO, benchmarking, overfitting prevention, compute requirements, Hugging Face Trainer, and the fine-tuning vs prompt engineering decision.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Fine-Tuning Guide skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
Overview
Fine-Tuning vs Prompt Engineering Decision
Dataset Preparation
LoRA (Low-Rank Adaptation)
QLoRA (Quantized LoRA)
Training with Hugging Face
RLHF and DPO
Overfitting Prevention
Benchmarking Fine-Tuned Models
OpenAI Fine-Tuning API
Checklist
When to Use
Output Format
Example
Edge Cases
Can prompt engineering achieve your quality target?
YES -> Use prompt engineering (cheaper, faster iteration)
NO -> Continue
Do you have < 100 high-quality examples?
YES -> Use few-shot prompting instead
NO -> Continue
Is your task highly specialized or domain-specific?
YES -> Fine-tuning likely needed
NO -> Try prompt engineering harder first
Do you need consistent output format/style?
YES -> Fine-tuning excels at this
# ... (condensed) ...
NO -> Prompt engineering may suffice
Budget for training and hosting?
LOW -> Prompt engineering
HIGH -> Fine-tuning is viable| Factor | Prompt Engineering | Fine-Tuning |
|---|---|---|
| Setup time | Minutes | Hours to days |
| Data needed | 0-20 examples | 100-100K examples |
| Cost | Per-API-call | Training + hosting |
| Iteration speed | Instant | Hours per experiment |
| Task specificity | Moderate | High |
| Output consistency | Moderate | High |
| Knowledge injection | Limited by context | Learned into weights |
| Maintenance | Easy | Requires retraining |
[
{
"instruction": "Summarize the following medical report in plain language for a patient.",
"input": "MRI findings indicate a 2.3cm hyperintense lesion in the right frontal lobe...",
"output": "Your brain scan showed a small spot (about 1 inch) in the right front part of your brain..."
},
{
"instruction": "Extract all medication names and dosages from this clinical note.",
"input": "Patient prescribed metformin 500mg BID and lisinopril 10mg daily...",
"output": "{\"medications\": [{\"name\": \"metformin\", \"dose\": \"500mg\", \"frequency\": \"twice daily\"}, {\"name\": \"lisinopril\", \"dose\": \"10mg\", \"frequency\": \"once daily\"}]}"
}
]{
"messages": [
{"role": "system", "content": "You are a medical coding assistant."},
{"role": "user", "content": "Assign ICD-10 codes to: Patient presents with acute bronchitis and type 2 diabetes."},
{"role": "assistant", "content": "1. J20.9 - Acute bronchitis, unspecified\n2. E11.9 - Type 2 diabetes mellitus without complications"}
]
}import json
from dataclasses import dataclass
@dataclass
class QualityMetrics:
total: int
passed: int
failed: int
failure_reasons: dict
def validate_dataset(data: list[dict]) -> QualityMetrics:
"""Validate fine-tuning dataset quality."""
failures = {}
passed = 0
# ... (condensed) ...
total=len(data),
passed=passed,
failed=len(data) - passed,
failure_reasons=failures,
)def augment_instruction_data(
samples: list[dict],
client,
augment_factor: int = 3,
) -> list[dict]:
"""Generate additional training examples from existing ones."""
augmented = list(samples)
for sample in samples:
for _ in range(augment_factor):
# Paraphrase the instruction
new_instruction = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{
# ... (condensed) ...
"input": sample["input"],
"output": sample["output"],
})
return augmentedLoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into each transformer layer. Instead of updating W (d x d), it trains A (d x r) and B (r x d) where r << d.
Original: h = W * x
LoRA: h = W * x + (B * A) * x
Where W is frozen, A and B are trainable
r (rank) is typically 8-64 (vs d = 4096+)from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer
def setup_lora_model(
base_model: str = "meta-llama/Llama-3.1-8B-Instruct",
lora_r: int = 16,
lora_alpha: int = 32,
lora_dropout: float = 0.05,
target_modules: list[str] = None,
):
"""Configure a model with LoRA adapters."""
model = AutoModelForCausalLM.from_pretrained(
base_model,
# ... (condensed) ...
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Trainable: {trainable:,} / {total:,} ({trainable/total:.2%})")
return model| Parameter | Range | Effect |
|---|---|---|
| r (rank) | 4-128 | Higher = more capacity, more parameters |
| lora_alpha | r to 2*r | Scaling factor; alpha/r is the actual scale |
| dropout | 0.0-0.1 | Regularization; 0.05 is typical |
| target_modules | varies | Which layers to adapt (all linear layers recommended) |
Rules of thumb:
Start with r=16, alpha=32
For simple tasks (classification): r=8 may suffice
For complex tasks (code, creative): r=32-64
Always target all linear layers (not just attention)
QLoRA combines 4-bit quantization of the base model with LoRA, enabling fine-tuning of large models on consumer GPUs.
from transformers import BitsAndBytesConfig
import torch
def setup_qlora_model(
base_model: str = "meta-llama/Llama-3.1-8B-Instruct",
lora_r: int = 16,
):
"""Setup QLoRA: 4-bit base model + LoRA adapters."""
# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat4
bnb_4bit_compute_dtype=torch.bfloat16,
# ... (condensed) ...
task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)
return model| Model Size | Full Fine-Tune | LoRA (fp16) | QLoRA (4-bit) |
|---|---|---|---|
| 7B | ~120 GB | ~20 GB | ~6 GB |
| 13B | ~240 GB | ~40 GB | ~12 GB |
| 70B | ~1.2 TB | ~160 GB | ~40 GB |
from trl import SFTTrainer, SFTConfig
from transformers import AutoTokenizer
from datasets import load_dataset
def train_sft(
model,
tokenizer,
train_dataset,
output_dir: str = "./fine-tuned-model",
num_epochs: int = 3,
batch_size: int = 4,
learning_rate: float = 2e-4,
):
"""Supervised fine-tuning with SFTTrainer."""
# ... (condensed) ...
)
trainer.train()
trainer.save_model(output_dir)
return trainerfrom transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
def main():
# Load base model and tokenizer
model_name = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
# ... (condensed) ...
trainer.train()
trainer.save_model("./final-model")
if __name__ == "__main__":
main()DPO is simpler than full RLHF -- it directly optimizes from preference pairs without a separate reward model.
from trl import DPOTrainer, DPOConfig
def train_dpo(
model,
tokenizer,
preference_dataset,
output_dir: str = "./dpo-model",
):
"""Train with DPO from preference pairs."""
# Dataset format: each row has "chosen" and "rejected" responses
# {
# "prompt": "Explain photosynthesis",
# "chosen": "Photosynthesis is the process by which plants...",
# ... (condensed) ...
args=training_args,
)
trainer.train()
return trainerdef create_preference_pairs(
prompts: list[str],
model,
judge_client,
) -> list[dict]:
"""Generate preference pairs using an LLM judge."""
pairs = []
for prompt in prompts:
# Generate two responses with different temperatures
response_a = generate(model, prompt, temperature=0.3)
response_b = generate(model, prompt, temperature=0.9)
# Judge which is better
# ... (condensed) ...
pairs.append({"prompt": prompt, "chosen": response_a, "rejected": response_b})
else:
pairs.append({"prompt": prompt, "chosen": response_b, "rejected": response_a})
return pairsEarly stopping: Monitor validation loss, stop when it increases
from transformers import EarlyStoppingCallback
trainer = SFTTrainer(
...,
callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
)Data diversity: Ensure training data covers the full distribution
Regularization: LoRA dropout, weight decay
Validation split: Always hold out 10-20% for validation
dataset = dataset.train_test_split(test_size=0.1, seed=42)
trainer = SFTTrainer(
...,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
args=SFTConfig(
...,
eval_strategy="steps",
eval_steps=50,
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
),
)| Signal | Indicator | Action |
|---|---|---|
| Train loss << val loss | Gap increasing | Reduce epochs, increase data |
| Val loss increasing | After initial decrease | Early stopping |
| Repetitive outputs | Model parrots training data | More diverse data |
| Poor generalization | Works only on training-like inputs | Augment data distribution |
def benchmark_model(model, tokenizer, test_set: list[dict]) -> dict:
"""Run benchmarks against held-out test set."""
results = {
"exact_match": 0,
"format_compliance": 0,
"total": len(test_set),
}
for sample in test_set:
prompt = tokenizer.apply_chat_template(
sample["messages"][:-1], # All except assistant reply
tokenize=False,
add_generation_prompt=True,
)
# ... (condensed) ...
results["exact_match_rate"] = results["exact_match"] / results["total"]
results["format_compliance_rate"] = results["format_compliance"] / results["total"]
return resultsdef compare_models(
base_model,
fine_tuned_model,
tokenizer,
test_prompts: list[str],
judge_client,
) -> dict:
"""Compare base and fine-tuned model outputs using LLM judge."""
wins = {"base": 0, "fine_tuned": 0, "tie": 0}
for prompt in test_prompts:
base_output = generate(base_model, tokenizer, prompt)
ft_output = generate(fine_tuned_model, tokenizer, prompt)
# ... (condensed) ...
wins["fine_tuned"] += 1
else:
wins["tie"] += 1
return winsfrom openai import OpenAI
client = OpenAI()
# Upload training file
file = client.files.create(
file=open("training_data.jsonl", "rb"),
purpose="fine-tune",
)
# Create fine-tuning job
job = client.fine_tuning.jobs.create(
training_file=file.id,
model="gpt-4o-mini-2024-07-18",
# ... (condensed) ...
# Use fine-tuned model
response = client.chat.completions.create(
model=status.fine_tuned_model,
messages=[{"role": "user", "content": "Your prompt here"}],
)Determine if fine-tuning is necessary vs prompt engineering
Prepare and validate dataset (minimum 100 high-quality examples)
Choose base model appropriate for task and compute budget
Select LoRA vs QLoRA based on available GPU memory
Configure LoRA hyperparameters (r, alpha, target modules)
Set up experiment tracking (W&B or MLflow)
Implement validation split and early stopping
Train with gradient checkpointing for memory efficiency
Benchmark fine-tuned model against base model
Check for overfitting (train vs validation loss gap)
Consider DPO if you have preference data
Save and version the LoRA adapter weights
Use this skill when:
Designing or implementing fine tuning guide solutions
Reviewing or improving existing fine tuning guide approaches
Making architectural or implementation decisions about fine tuning guide
Learning fine tuning guide patterns and best practices
Troubleshooting fine tuning guide-related issues
Do NOT use this skill when:
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
# Fine Tuning Guide Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement fine tuning guide for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended fine tuning guide approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When fine tuning guide must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
527 lines · 1,969 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Fine-Tuning Guide bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for