What common A/B testing errors does this skill prevent?

It specifically prevents 'peeking' (checking results early), running underpowered tests with insufficient sample sizes, and 'p-hacking' by forcing you to lock your primary metric before the test starts.

Can I use this for multivariate or split URL tests?

While it defaults to standard A/B tests for simplicity, the skill includes frameworks for A/B/n, Multivariate (MVT), and Split URL testing for more complex structural changes.

When in the development lifecycle should I trigger this skill?

Trigger this during the design or planning phase, before any code is written, to ensure your experiment is valid and your tracking plan is ready.

Does this skill help with statistical calculations?

Yes, it guides you through defining the Minimum Detectable Effect (MDE), statistical power, and significance levels to estimate the required sample size and duration.

Rigorous A/B Test Setup

Name: Rigorous A/B Test Setup
Author: claudiodearaujo

byclaudiodearaujo

•

Analytics & Monitoring

Enforces statistical rigor and mandatory validation gates for designing valid, high-confidence A/B tests before implementation begins.

This skill provides a structured, fail-safe framework for engineering and product teams to design experiments that yield actionable insights. By acting as a methodological gatekeeper, it ensures every A/B test has a locked hypothesis, predefined primary metrics, and calculated sample sizes to prevent common pitfalls like 'peeking' or underpowered results. It is particularly useful for teams that need to maintain high data integrity and avoid the costly mistake of shipping features based on statistically insignificant or biased data.

Key Features

011 GitHub stars

02Hard-gate hypothesis validation to ensure clarity and measurability

03Refusal conditions for insufficient traffic or undefined baselines

04Pre-defined primary, secondary, and guardrail metric tracking

05Sample size and test duration estimation based on MDE

06Standardized documentation templates for experiment learning

Use Cases

01Verifying if a proposed change has enough traffic to reach significance

02Establishing experimental parameters before writing feature flag logic

03Preventing biased result interpretation by freezing metrics before launch

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add claudiodearaujo/sistema-de-narra-o-de-livro ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill