How does it help with result analysis?

It provides a structured method for evaluating statistical significance (p-values) and practical significance to determine if a variant should be implemented.

What is the A/B Test Setup skill for Claude Code?

It is a specialized capability that helps users design, implement, and document statistically sound experiments, from initial hypothesis creation to final result interpretation.

Can this skill help calculate sample sizes?

Yes, it provides guidelines and reference tables for determining the required sample size based on your baseline conversion rate and minimum detectable effect (MDE).

Does it support multivariate testing?

Yes, the skill includes frameworks for various experiment types including A/B, A/B/n, Multivariate (MVT), and Split URL tests.

What are guardrail metrics in this context?

Guardrail metrics are key business indicators, such as revenue or retention, that should not be negatively impacted by an experiment, even if the primary metric improves.

A/B Test Setup & Experimentation

Name: A/B Test Setup & Experimentation
Author: aitytech

byaitytech

•

184

•

Marketing Automation

Designs and implements statistically valid A/B tests and experiments to optimize conversion rates and user experience.

This skill transforms Claude into an expert experimentation consultant, helping you design, launch, and analyze A/B tests that produce actionable results. It guides users through creating rigorous hypotheses, calculating required sample sizes, selecting meaningful metrics, and avoiding common pitfalls like 'peeking' at data early. Whether you are testing UI changes, pricing models, or signup flows, this skill provides the frameworks and documentation templates needed to build a data-driven experimentation culture and ensure every test result is both statistically and practically significant.

Key Features

01184 GitHub stars

02Statistical power and sample size calculation guidance

03Structured test plan documentation and post-test analysis templates

04Support for A/B, A/B/n, Multivariate (MVT), and Split URL testing

05Hypothesis generation framework based on data-driven observations

06Guidance on primary, secondary, and guardrail metric selection

Use Cases

01Planning a server-side feature rollout with technical risk mitigation

02Designing a pricing page experiment to increase plan upgrades

03Validating headline changes to improve signup completion rates

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add aitytech/agentkits-marketing ab-test-setup

For use in Claude.ai and ChatGPT

Download Skill