How does this differ from standard code coverage tools?

Standard tools only show if a line was executed. This skill analyzes the quality of the assertion and whether the test would actually catch a bug if the production logic changed.

Why is 'Read Production Code' a mandatory phase?

You cannot determine if a test is effective without understanding the code it claims to exercise. This prevents miscategorizing tests that only exercise mocks or utilities as 'Green'.

What is a 'tautological' test?

A tautological test is one that passes by definition, such as checking if a non-nullable return value is not null or asserting that 2+3 equals 2+3. These provide 'vanity' coverage without testing logic.

Does this skill work with all programming languages?

Yes, it includes detection patterns and mutation testing validation for major ecosystems including Java (Pitest), JavaScript/TypeScript (Stryker), and Python (mutmut).

Test Effectiveness Auditor

Name: Test Effectiveness Auditor
Author: withzombies

bywithzombies

•

Security & Testing

Audits test suites with Google Fellow SRE-level scrutiny to eliminate vanity metrics and identify real logic gaps.

The Test Effectiveness Auditor is a specialized skill designed to move beyond misleading code coverage metrics and evaluate the actual protective value of a test suite. By applying a rigorous 8-phase analysis process, it identifies 'tautological' tests that pass by definition, 'mock-testing' patterns that bypass production logic, and 'line hitters' that execute code without meaningful assertions. The skill mandates a skeptical mindset, requiring Claude to trace production code paths before marking any test as reliable. It concludes by generating a prioritized improvement plan through the 'bd' task tracking system, ensuring that every identified weakness is refined and assigned for remediation.

Key Features

01Detection of tautological tests, mock-testing-mocks, and weak assertions

0258 GitHub stars

03Skeptical RED/YELLOW/GREEN categorization based on SRE standards

04Rigorous 8-phase audit process from inventory to prioritization

05Mandatory production code tracing to verify actual test call paths

06Automated creation of 'bd' epics and SRE-refined tasks for remediation

Use Cases

01Auditing legacy codebases where high coverage masks frequent production bugs

02Onboarding to unfamiliar projects to quickly assess test suite reliability

03Validating the safety net before performing high-risk architectural refactors

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add withzombies/hyperpowers analyzing-test-effectiveness

For use in Claude.ai and ChatGPT

Download Skill