How does this skill handle statistical analysis?

It provides a framework for selecting appropriate tests based on data distribution (e.g., t-tests vs. Mann-Whitney U) and defines reporting standards like standard deviation and confidence intervals.

Can I use this for machine learning projects?

Yes, it is specifically optimized for ML workflows, including sections for GPU-hour estimates, hyperparameter tuning budgets, and data leakage prevention.

What is the primary goal of the Experiment Design Checklist?

The primary goal is to prevent meaningless results by ensuring that experiments are designed with clear hypotheses, appropriate baselines, and rigorous statistical controls before any code is run.

Does it include templates for ablation studies?

Absolutely. It includes a dedicated ablation template that maps specific variants to expected effects, ensuring each model component's value is clearly demonstrated.

Experiment Design Checklist

Name: Experiment Design Checklist
Author: GhostScientist

byGhostScientist

•

Data Science & ML

Generates rigorous experimental frameworks to validate research hypotheses with statistical significance and comprehensive baseline comparisons.

The Experiment Design Checklist skill provides a structured methodology for planning scientific experiments within Claude Code, preventing inconclusive research cycles by enforcing upfront rigor. It guides developers and researchers through a multi-step process including precise hypothesis formulation, variable identification, baseline selection, and ablation study design. By addressing common confounds and providing compute budget estimates, this skill ensures that every experiment is reproducible, statistically sound, and capable of convincing skeptical reviewers.

Key Features

011 GitHub stars

02Statistical rigor planning with sample size and test selection guidance

03Falsifiable hypothesis and null hypothesis generation templates

04Multi-tier baseline hierarchy including SOTA and ablated self-comparisons

05Comprehensive variable mapping (independent, dependent, and control)

06Surgical ablation study planning to isolate component contributions

Use Cases

01Planning ablation studies to determine the specific impact of model components

02Validating a new machine learning architecture against industry standard baselines

03Creating a standardized experimental setup for academic or technical research papers

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add ghostscientist/skills experiment-design-checklist

For use in Claude.ai and ChatGPT

Download Skill