Can it help if I don't know my sample size?

Absolutely. It requires you to define baseline rates and MDE to estimate the required sample size and duration before proceeding to implementation.

Does it support multivariate testing (MVT)?

Yes, it guides you through selecting the appropriate test type, including A/B, A/B/n, MVT, and Split URL tests based on your traffic volume.

What are guardrail metrics in this framework?

Guardrail metrics are key indicators that must not degrade; the skill uses them to trigger a test stop if a variant causes unintended harm to the system.

How does this skill prevent 'peeking' at test results?

The skill enforces a pre-defined sample size and duration gate, explicitly instructing users not to stop tests early based on fluctuating early data.

A/B Test Setup Guide

Name: A/B Test Setup Guide
Author: claudiodearaujo

byclaudiodearaujo

•

Analytics & Monitoring

Standardizes the creation and execution of rigorous A/B tests through mandatory validation gates and statistical planning.

The A/B Test Setup skill provides a structured, scientific framework to ensure experimental validity before a single line of code is written. It enforces industry-standard best practices such as hypothesis locking, primary metric definition, and sample size calculations to prevent common pitfalls like 'peeking' or data dredging. By implementing hard gates for readiness and strict refusal conditions for insufficient data, this skill helps teams move from 'guessing' to learning with statistical confidence, ensuring every experiment produces actionable and reliable business insights.

Key Features

01Hierarchical Metric Framework (Primary, Secondary, and Guardrails)

02Statistical Power and Sample Size calculation guidance

03Execution Readiness Gates to block scientifically invalid tests

04Safety Refusal Conditions for low traffic or undefined baselines

051 GitHub stars

06Mandatory Hypothesis Lock to prevent mid-test goal shifting

Use Cases

01Setting up guardrail metrics to prevent negative impacts on core business KPIs

02Designing scientifically valid split tests for new product features

03Establishing baseline metrics and Minimum Detectable Effect (MDE) for CRO

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add claudiodearaujo/sistema-de-narra-o-de-livro ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill