Can this skill help calculate sample sizes for tests?

Yes, it provides quick-reference tables and guidance for determining the traffic needed per variant to detect specific conversion lifts at various baseline rates.

Does it support multivariate testing (MVT)?

Yes, it covers various test types including standard A/B, A/B/n (multiple variants), MVT (multiple combinations), and Split URL testing.

What is the A/B Test Setup skill?

It is a specialized capability for Claude Code that provides frameworks, best practices, and implementation patterns for designing, running, and analyzing A/B tests and experiments.

How does it ensure statistical validity?

It enforces a strict hypothesis framework, requires pre-defined primary metrics, and provides clear guidelines against common mistakes like 'peeking' at results before reaching sample size.

A/B Test Experiment Setup

Name: A/B Test Experiment Setup
Author: mysticaltech

bymysticaltech

•

Marketing Automation

Facilitates the design, implementation, and analysis of statistically valid A/B tests and marketing experiments.

The A/B Test Setup skill empowers developers and marketers to run rigorous experiments by providing a structured framework for hypothesis generation, sample size calculation, and metric selection. It guides users through the entire experimentation lifecycle—from initial assessment and variant design to technical implementation and post-test analysis—ensuring results are actionable and statistically significant rather than based on random chance. Whether you are testing a single headline change or a complex multivariate layout, this skill ensures your testing methodology is sound and your data is reliable.

Key Features

01Statistical rigor tools including sample size and duration calculators

0259 GitHub stars

03Guidance on primary, secondary, and guardrail metric selection

04Structured hypothesis framework for evidence-based experiment design

05Comprehensive analysis checklists for interpreting significance and effect size

06Technical implementation strategies for client-side and server-side testing

Use Cases

01Designing high-impact split tests for pricing pages or landing pages

02Analyzing experiment results to determine if a variant should be implemented or discarded

03Calculating required traffic volume to achieve statistical significance for a specific lift

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add mysticaltech/marketingskills ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill