Why is pre-registration mentioned in the skill?

Pre-registration helps prevent 'p-hacking' or moving the goalposts by forcing you to define your success criteria and analysis plan before seeing the experimental results.

Does it include guidance on statistical analysis?

Yes, it provides a roadmap for choosing appropriate statistical tests—such as t-tests, ANOVA, or Mann-Whitney U—based on your data distribution and comparison types.

What is the Experiment Design Checklist?

It is a Claude Code skill that provides a structured framework for planning scientific and machine learning experiments to ensure validity, reproducibility, and statistical significance.

How does this skill help with ML model development?

It helps define clear baselines, ablation studies, and success metrics before you start training, which prevents meaningless results and helps optimize your compute budget.

Can I use this for software performance benchmarking?

Absolutely. The checklist is designed to handle any scenario where you need to isolate variables, such as testing the latency impact of a new software module or database configuration.

Experiment Design Checklist

Name: Experiment Design Checklist
Author: GhostScientist

byGhostScientist

•

Data Science & ML

Generates rigorous experimental frameworks for scientific research and machine learning projects to ensure statistically significant and defensible results.

The Experiment Design Checklist is a specialized Claude Code skill designed to prevent wasted research effort by enforcing rigorous upfront planning. It guides users through the entire scientific process, from crafting falsifiable hypotheses and identifying variables to establishing strong baselines and designing surgical ablations. Ideal for data scientists, ML researchers, and software engineers, it provides a standardized template covering statistical power, confound mitigation, and compute budgeting, ensuring that every experiment produces meaningful results that would satisfy a skeptical reviewer.

Key Features

01Surgical ablation study design to isolate the impact of specific components

02Comprehensive variable mapping for independent, dependent, and control factors

031 GitHub stars

04Baseline hierarchy planning including SOTA and ablated self-comparisons

05Statistical rigor planning including test selection and compute budget estimation

06Falsifiable hypothesis generation with clear null hypothesis definitions

Use Cases

01Designing machine learning model architecture comparisons and benchmarks

02Planning A/B tests for software features or user interface optimizations

03Validating research hypotheses for academic papers or industry technical reports

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add ghostscientist/skills experiment-design-checklist

For use in Claude.ai and ChatGPT

Download Skill