What are guardrail metrics in this context?

Guardrail metrics are secondary data points that must not degrade during a test. If they show a significant negative trend, the skill recommends stopping the test to prevent business harm.

How does this skill prevent common A/B testing errors?

It enforces mandatory 'Hard Gates' such as the Hypothesis Lock and Execution Readiness Gate, ensuring that metrics and sample sizes are frozen before any code is deployed.

Does it provide guidance on analyzing results?

Yes, it includes an Analysis Discipline section that provides an interpretation matrix for significant positive, negative, and inconclusive results.

Can I use this for multivariate or A/B/n testing?

Yes, the skill supports A/B, A/B/n, Multivariate (MVT), and Split URL tests, helping you choose the simplest valid test based on your traffic volume.

Structured A/B Test Setup

Name: Structured A/B Test Setup
Author: claudiodearaujo

byclaudiodearaujo

0•

Analytics & Monitoring

Standardizes the A/B testing process by enforcing statistical rigor and mandatory validation gates before implementation.

This skill provides a systematic framework for designing and executing A/B tests with high statistical confidence. It prevents common pitfalls like 'peeking' at results or testing invalid hypotheses by guiding developers through mandatory gates, including hypothesis locking, sample size estimation, and guardrail metric definition. Ideal for product teams and engineers looking to move beyond 'trial and error' and establish a rigorous data-driven culture with consistent, reliable experimentation results.

Key Features

01Strict Post-Test Analysis Discipline to prevent overgeneralization and causality errors.

02Hard Execution Gates that block implementation until all prerequisites are verified.

03Mandatory Hypothesis Locking to ensure clear directional expectations and success criteria.

04Comprehensive Metric Framework defining primary, secondary, and guardrail metrics.

05Automated Sample Size and Duration Estimation based on MDE and statistical power.

060 GitHub stars

Use Cases

01Standardizing experimentation documentation and records across a multi-developer product team.

02Preventing invalid test results caused by mid-test changes or insufficient sample sizes.

03Setting up a new UI/UX experiment with verified statistical power and traffic estimates.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add claudiodearaujo/izacenter ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill