What categories of flakiness can it identify?

It categorizes tests into five research-based groups: Async/Timing, Test Order, Environment, Resource Limits, and Non-deterministic logic (like random values or UUIDs).

How does this skill find flaky tests?

It uses a three-pronged approach: analyzing CI execution history for inconsistent results, scanning source code for known flaky patterns like setTimeout or unhandled promises, and performing automated test re-runs to verify flakiness.

Does it support test quarantine?

Absolutely. It provides workflows to skip flaky tests in CI while tracking them in a dedicated registry until they are verified as stable and ready to be un-skipped.

Can it help fix the tests it finds?

Yes, the skill provides specific code recommendations, such as using mocked time instead of Date.now(), and integrates with the flaky-fix skill for automated repairs.

Flaky Test Detector

Name: Flaky Test Detector
Author: jmagly

byjmagly

•

Security & Testing

Identifies and stabilizes intermittent test failures by analyzing CI history and code patterns to improve build reliability.

About

The flaky-detect skill provides a comprehensive solution for managing non-deterministic test suites that fail and pass inconsistently. By leveraging research-backed taxonomies and CI log analysis, it automatically detects flaky tests, categorizes them into root causes like timing or resource issues, and suggests actionable fixes. It enables developers to audit test reliability, quarantine problematic tests, and maintain a stable CI/CD pipeline, ultimately reducing wasted developer time and infrastructure costs.

Key Features

CI History Analysis: Scans execution logs to calculate pass/fail rates and detect inconsistent results.
Root Cause Taxonomy: Categorizes flakiness into groups such as Async, Test Order, and Environment dependencies.
Code Pattern Recognition: Identifies common flaky anti-patterns like un-awaited promises or hardcoded timeouts.
67 GitHub stars
Automated Re-run Validation: Executes tests multiple times to empirically verify non-deterministic behavior.
Quarantine Management: Streamlines the process of isolating unstable tests while tracking fixes.

Use Cases

Performing a pre-release quality audit to ensure high test suite reliability before major deployments.
Debugging intermittent build failures that disrupt the development workflow and slow down PR merges.
Isolating and tracking non-deterministic tests through a formal quarantine and repair process.

About

Key Features

CI History Analysis: Scans execution logs to calculate pass/fail rates and detect inconsistent results.
Root Cause Taxonomy: Categorizes flakiness into groups such as Async, Test Order, and Environment dependencies.
Code Pattern Recognition: Identifies common flaky anti-patterns like un-awaited promises or hardcoded timeouts.
67 GitHub stars
Automated Re-run Validation: Executes tests multiple times to empirically verify non-deterministic behavior.
Quarantine Management: Streamlines the process of isolating unstable tests while tracking fixes.

Use Cases

Performing a pre-release quality audit to ensure high test suite reliability before major deployments.
Debugging intermittent build failures that disrupt the development workflow and slow down PR merges.
Isolating and tracking non-deterministic tests through a formal quarantine and repair process.