What is the ab-test-setup Claude Code Skill?

It is a specialized capability that provides a structured, gated guide for setting up A/B tests to ensure they are statistically valid and safe before execution.

What are Guardrail Metrics in this context?

Guardrail metrics are secondary data points that must not degrade during a test; they trigger an automatic stop if the variant causes unexpected harm, even if the primary metric is winning.

How does this skill prevent invalid experiments?

The skill implements 'Hard Gates' that stop the process if essential criteria—like a locked hypothesis, defined primary metrics, or sufficient sample size—are not met.

Can I use this for complex multivariate tests?

Yes, while it defaults to simple A/B tests for reliability, it includes selection logic for A/B/n, Multivariate (MVT), and Split URL tests based on your traffic volume.

A/B Test Setup Guide

Name: A/B Test Setup Guide
Author: claudiodearaujo

byclaudiodearaujo

•

Analytics & Monitoring

Standardizes the A/B testing workflow through rigorous hypothesis validation, metric definitions, and statistical power checks.

The A/B Test Setup skill for Claude Code provides a high-integrity framework for designing and executing experiments scientifically. By enforcing mandatory 'hard gates' for hypothesis locking and execution readiness, it prevents common pitfalls such as 'peeking,' underpowered tests, and shifting success criteria. This skill guides users through every critical phase—from defining Minimum Detectable Effects (MDE) to setting guardrail metrics—ensuring that every test provides valid, actionable insights rather than statistical noise.

Key Features

011 GitHub stars

02Automated validity checks for traffic stability, randomization, and user independence.

03Rigorous analysis discipline to prevent causal overreach and generalization errors.

04Statistical Power and MDE calculation guidance for realistic test durations.

05Mandatory Hypothesis Lock to ensure test integrity before implementation.

06Multi-tier metric definitions including Primary, Secondary, and Guardrail metrics.

Use Cases

01Validating new product features using randomized controlled trials.

02Determining sample size requirements for tests with limited traffic volume.

03Designing high-stakes conversion rate optimization (CRO) experiments.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add claudiodearaujo/sistema-de-narra-o-de-livro ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill