Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
ML model assessment covering classification metrics (precision, recall, F1, AUC-ROC), regression metrics (MAE, RMSE, R2), confusion matrix analysis, cross-validation strategies, bias detection, fairness metrics, and A/B testing for models.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
523 lines · 1,897 words
Rigorous model assessment is essential to deploying trustworthy ML systems. This skill covers comprehensive metrics for classification and regression, strategies for cross-validation, statistical testing, bias and fairness auditing, and A/B testing frameworks for comparing models in production.
ML model assessment covering classification metrics (precision, recall, F1, AUC-ROC), regression metrics (MAE, RMSE, R2), confusion matrix analysis, cross-validation strategies, bias detection, fairness metrics, and A/B testing for models.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Model Evaluator skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
Overview
Classification Metrics
Confusion Matrix Analysis
Regression Metrics
Cross-Validation Strategies
Statistical Model Comparison
Bias and Fairness Assessment
A/B Testing for Models
Comprehensive Scoring Report
Checklist
When to Use
Output Format
Example
Edge Cases
from sklearn.metrics import (
accuracy_score, precision_score, recall_score, f1_score,
roc_auc_score, average_precision_score, classification_report,
confusion_matrix,
)
import numpy as np
def classification_report_full(y_true, y_pred, y_prob=None) -> dict:
"""Comprehensive classification metrics."""
metrics = {
"accuracy": accuracy_score(y_true, y_pred),
"precision_macro": precision_score(y_true, y_pred, average="macro"),
"recall_macro": recall_score(y_true, y_pred, average="macro"),
"f1_macro": f1_score(y_true, y_pred, average="macro"),
# ... (condensed) ...
metrics["auc_roc_ovr"] = roc_auc_score(
y_true, y_prob, multi_class="ovr", average="macro"
)
return metrics| Metric | When to Use | Sensitive To |
|---|---|---|
| Accuracy | Balanced classes only | Class imbalance |
| Precision | Cost of false positives is high | Threshold selection |
| Recall | Cost of false negatives is high | Threshold selection |
| F1 Score | Balance precision and recall | Threshold selection |
| AUC-ROC | Overall ranking ability | Not threshold-dependent |
| Average Precision | Imbalanced classes, ranking | Class distribution |
| Cohen's Kappa | Agreement beyond chance | None |
Is your dataset balanced (classes within 2x of each other)?
YES -> Accuracy is meaningful, but also report F1
NO -> DO NOT rely on accuracy. Use these instead:
- F1 (balanced view)
- Average Precision (best for heavy imbalance)
- AUC-ROC (threshold-independent ranking)
What is more costly?
False positives (spam filter, fraud alert):
-> Optimize for PRECISION
False negatives (cancer screening, security):
-> Optimize for RECALL
Both equally bad:
-> Optimize for F1 scoreimport matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
def plot_confusion_matrix(
y_true, y_pred,
class_names: list[str] = None,
normalize: str = None,
figsize: tuple = (8, 6),
) -> plt.Figure:
"""Plot confusion matrix with detailed annotations."""
cm = confusion_matrix(y_true, y_pred, normalize=normalize)
fig, ax = plt.subplots(figsize=figsize)
# ... (condensed) ...
})
confused_with.sort(key=lambda x: x["count"], reverse=True)
analysis[cls]["most_confused_with"] = confused_with[:3]
return analysisfrom sklearn.metrics import roc_curve, precision_recall_curve, auc
def plot_roc_pr_curves(y_true, y_prob, figsize=(14, 5)):
"""Plot ROC and Precision-Recall curves side by side."""
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=figsize)
# ROC Curve
fpr, tpr, roc_thresholds = roc_curve(y_true, y_prob)
roc_auc = auc(fpr, tpr)
ax1.plot(fpr, tpr, label=f"AUC = {roc_auc:.3f}")
ax1.plot([0, 1], [0, 1], "k--", alpha=0.3)
ax1.set_xlabel("False Positive Rate")
ax1.set_ylabel("True Positive Rate")
ax1.set_title("ROC Curve")
# ... (condensed) ...
ax2.set_title("Precision-Recall Curve")
ax2.legend()
plt.tight_layout()
return figfrom sklearn.metrics import balanced_accuracy_score
def find_optimal_threshold(
y_true, y_prob,
metric: str = "f1",
) -> tuple[float, float]:
"""Find the threshold that maximizes a given metric."""
thresholds = np.arange(0.1, 0.95, 0.01)
best_threshold = 0.5
best_score = 0
for threshold in thresholds:
y_pred = (y_prob >= threshold).astype(int)
# ... (condensed) ...
if score > best_score:
best_score = score
best_threshold = threshold
return best_threshold, best_scorefrom sklearn.metrics import (
mean_absolute_error, mean_squared_error, r2_score,
mean_absolute_percentage_error, median_absolute_error,
)
def regression_report(y_true, y_pred) -> dict:
"""Comprehensive regression metrics."""
return {
"mae": mean_absolute_error(y_true, y_pred),
"rmse": np.sqrt(mean_squared_error(y_true, y_pred)),
"mse": mean_squared_error(y_true, y_pred),
"r2": r2_score(y_true, y_pred),
"mape": mean_absolute_percentage_error(y_true, y_pred),
"median_ae": median_absolute_error(y_true, y_pred),
"max_error": float(np.max(np.abs(y_true - y_pred))),
}| Metric | Range | Interpretation |
|---|---|---|
| MAE | [0, inf) | Average absolute error in original units |
| RMSE | [0, inf) | Penalizes large errors more than MAE |
| R2 | (-inf, 1] | 1 = perfect; 0 = predicts mean; <0 = worse than mean |
| MAPE | [0, inf) | Percentage error (avoid when y has zeros) |
| Median AE | [0, inf) | Robust to outliers |
from scipy import stats as sp_stats
def plot_residual_analysis(y_true, y_pred, figsize=(14, 10)):
"""Comprehensive residual analysis plots."""
residuals = y_true - y_pred
fig, axes = plt.subplots(2, 2, figsize=figsize)
# Predicted vs Actual
axes[0, 0].scatter(y_pred, y_true, alpha=0.5, s=10)
min_val, max_val = min(y_true.min(), y_pred.min()), max(y_true.max(), y_pred.max())
axes[0, 0].plot([min_val, max_val], [min_val, max_val], "r--")
axes[0, 0].set_xlabel("Predicted")
axes[0, 0].set_ylabel("Actual")
# ... (condensed) ...
sp_stats.probplot(residuals, dist="norm", plot=axes[1, 1])
axes[1, 1].set_title("QQ Plot")
plt.tight_layout()
return figfrom sklearn.model_selection import (
KFold, StratifiedKFold, TimeSeriesSplit,
GroupKFold, RepeatedStratifiedKFold,
cross_val_score,
)
def get_cv_strategy(
task_type: str,
data_type: str = "standard",
n_splits: int = 5,
groups=None,
):
"""Select appropriate cross-validation strategy."""
# ... (condensed) ...
if task_type == "classification":
return StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=42)
return KFold(n_splits=n_splits, shuffle=True, random_state=42)def cv_with_confidence(model, X, y, cv=5, scoring="f1") -> dict:
"""Cross-validation with confidence interval."""
scores = cross_val_score(model, X, y, cv=cv, scoring=scoring)
mean = scores.mean()
std = scores.std()
n = len(scores)
se = std / np.sqrt(n)
# 95% confidence interval
ci_low = mean - 1.96 * se
ci_high = mean + 1.96 * se
return {
"mean": mean,
"std": std,
"scores": scores.tolist(),
"ci_95": (ci_low, ci_high),
"n_folds": n,
}from scipy import stats
def compare_models_statistical(
model_a_scores: list[float],
model_b_scores: list[float],
alpha: float = 0.05,
) -> dict:
"""Statistical comparison of two models using paired t-test."""
t_stat, p_value = stats.ttest_rel(model_a_scores, model_b_scores)
mean_diff = np.mean(model_a_scores) - np.mean(model_b_scores)
return {
"model_a_mean": np.mean(model_a_scores),
"model_b_mean": np.mean(model_b_scores),
"mean_difference": mean_diff,
"t_statistic": t_stat,
"p_value": p_value,
"significant": p_value < alpha,
"better_model": "A" if mean_diff > 0 else "B",
}def mcnemar_test(y_true, y_pred_a, y_pred_b) -> dict:
"""McNemar's test: are two classifiers significantly different?"""
from statsmodels.stats.contingency_tables import mcnemar as mcnemar_fn
correct_a = (y_pred_a == y_true)
correct_b = (y_pred_b == y_true)
n01 = ((~correct_a) & correct_b).sum() # A wrong, B right
n10 = (correct_a & (~correct_b)).sum() # A right, B wrong
table = [[0, n01], [n10, 0]]
result = mcnemar_fn(table, exact=True)
return {
"a_right_b_wrong": int(n10),
"a_wrong_b_right": int(n01),
"p_value": result.pvalue,
"significant": result.pvalue < 0.05,
}def compute_fairness_metrics(
y_true: np.ndarray,
y_pred: np.ndarray,
sensitive_attr: np.ndarray,
privileged_value=1,
unprivileged_value=0,
) -> dict:
"""Compute fairness metrics across a sensitive attribute."""
priv_mask = sensitive_attr == privileged_value
unpriv_mask = sensitive_attr == unprivileged_value
# Demographic parity
rate_priv = y_pred[priv_mask].mean()
# ... (condensed) ...
"tpr_privileged": round(tpr_priv, 4),
"tpr_unprivileged": round(tpr_unpriv, 4),
"fpr_privileged": round(fpr_priv, 4),
"fpr_unprivileged": round(fpr_unpriv, 4),
}| Metric | Definition | Fair When |
|---|---|---|
| Demographic Parity | Selection rate ratio across groups | Ratio between 0.8-1.25 |
| Equal Opportunity | True positive rate ratio | Ratio between 0.8-1.25 |
| Equalized Odds | TPR and FPR equal across groups | Both ratios near 1.0 |
| Predictive Parity | Positive predictive value equal | Ratio between 0.8-1.25 |
| Calibration | P(Y=1 given score=s) same across groups | Calibration curves overlap |
import pandas as pd
def subgroup_performance(
y_true, y_pred, y_prob,
group_column: np.ndarray,
group_names: dict,
) -> pd.DataFrame:
"""Compute performance metrics per subgroup."""
results = []
for group_val, group_name in group_names.items():
mask = group_column == group_val
if mask.sum() < 10:
continue
# ... (condensed) ...
metrics["auc_roc"] = float(roc_auc_score(y_true[mask], prob))
results.append(metrics)
return pd.DataFrame(results)import hashlib
from dataclasses import dataclass, field
from scipy import stats
@dataclass
class ModelABTest:
"""A/B test framework for comparing models in production."""
name: str
model_a_name: str
model_b_name: str
traffic_split: float = 0.5
results_a: list = field(default_factory=list)
results_b: list = field(default_factory=list)
# ... (condensed) ...
"effect_size": "small" if abs(cohens_d) < 0.5 else "medium" if abs(cohens_d) < 0.8 else "large",
"recommendation": "B" if p_value < 0.05 and mean_b > mean_a else "A" if p_value < 0.05 else "continue_testing",
"samples_a": len(self.results_a),
"samples_b": len(self.results_b),
}from scipy.stats import norm
def required_sample_size(
baseline_metric: float,
minimum_detectable_effect: float,
alpha: float = 0.05,
power: float = 0.8,
metric_std: float = None,
) -> int:
"""Calculate required sample size per group for A/B test."""
if metric_std is None:
metric_std = np.sqrt(baseline_metric * (1 - baseline_metric))
z_alpha = norm.ppf(1 - alpha / 2)
z_beta = norm.ppf(power)
n = (2 * metric_std**2 * (z_alpha + z_beta)**2) / minimum_detectable_effect**2
return int(np.ceil(n))def generate_scoring_report(
model_name: str,
y_true, y_pred, y_prob=None,
sensitive_attrs: dict = None,
) -> dict:
"""Generate comprehensive scoring report."""
report = {
"model": model_name,
"dataset_size": len(y_true),
"class_distribution": {int(k): int(v) for k, v in zip(*np.unique(y_true, return_counts=True))},
}
report["metrics"] = classification_report_full(y_true, y_pred, y_prob)
# ... (condensed) ...
report["fairness"][attr_name] = compute_fairness_metrics(
y_true, y_pred, attr_values
)
return reportSelect metrics appropriate to the task and class distribution
Report multiple metrics (never rely on accuracy alone)
Analyze the confusion matrix for systematic error patterns
Optimize the classification threshold if using probabilities
Use proper cross-validation (stratified for classification, temporal for time series)
Compute confidence intervals on performance estimates
Perform statistical tests when comparing models
Audit for bias across sensitive attributes (gender, race, age)
Check fairness metrics (demographic parity, equal opportunity)
Plan A/B tests with proper sample size calculations
Document all metrics, thresholds, and decisions in a scoring report
Use this skill when:
Designing or implementing model evaluator solutions
Reviewing or improving existing model evaluator approaches
Making architectural or implementation decisions about model evaluator
Learning model evaluator patterns and best practices
Troubleshooting model evaluator-related issues
Do NOT use this skill when:
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
# Model Evaluator Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement model evaluator for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended model evaluator approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When model evaluator must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
523 lines · 1,897 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Model Evaluator bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for