What is Eval-Driven Development (EDD)?

EDD treats evaluations as 'unit tests for AI,' where you define expected behavior and success criteria before implementation to ensure the AI meets specific goals.

Can I use existing test suites with Eval Harness?

Yes, Eval Harness supports code-based graders that can execute any bash command, such as npm test or grep, to verify outcomes deterministically.

How does this skill handle regression testing?

It allows you to set baselines and run existing tests against new AI-generated changes, reporting specifically on whether previously passing functionality has been compromised.

What are the pass@k and pass^k metrics?

Pass@k measures if the AI succeeds at least once in k attempts, while pass^k requires the AI to succeed in every single attempt, providing a higher threshold for critical stability.

Eval Harness

Name: Eval Harness
Author: Nixdorfer

byNixdorfer

•

Security & Testing

Implements an Eval-Driven Development (EDD) framework to ensure reliability and regression testing for AI-generated code.

Eval Harness is a specialized framework designed to bring the rigor of software testing to AI development workflows through Eval-Driven Development (EDD). It allows developers to define clear success criteria before coding begins, facilitating continuous evaluation during the development process. By supporting code-based, model-based, and human graders, the skill enables precise tracking of capability improvements and regression risks. It introduces critical metrics like pass@k to measure AI reliability, ensuring that Claude Code produces production-grade results that remain stable over time.

Key Features

01Regression testing to prevent functional decay in AI sessions

02Standardized evaluation reporting and project-level storage

03Advanced reliability metrics including pass@k and pass^k

04Support for deterministic code-based and model-based graders

05Eval-Driven Development (EDD) workflow integration

061 GitHub stars

Use Cases

01Automating regression checks to ensure new AI changes don't break existing features

02Defining objective success criteria before initiating complex code generation

03Measuring the reliability and consistency of Claude's output for critical system paths

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nixdorfer/claudecodetool eval-harness

For use in Claude.ai and ChatGPT

Download Skill