Where are the evaluation files stored?

All eval definitions, run histories, and baselines are stored locally within your project in the .claude/evals/ directory for easy versioning.

What is Eval-Driven Development (EDD)?

EDD is a methodology where evaluation criteria (evals) are defined before implementation, treating them as the 'unit tests of AI development' to ensure Claude meets specific goals.

How does this skill handle non-deterministic AI outputs?

It utilizes pass@k metrics to measure how often a success occurs within a set number of attempts, and uses model-based graders for evaluating open-ended responses.

Can I use this for regression testing?

Yes, the skill includes specialized regression evals designed to compare current output against baselines (SHA or checkpoints) to ensure new changes don't break existing features.

Eval Harness for Claude Code

Name: Eval Harness for Claude Code
Author: Nixdorfer

byNixdorfer

•

Security & Testing

Implements Eval-Driven Development (EDD) principles to systematically test, evaluate, and validate Claude's coding output through formal frameworks.

The Eval Harness skill transforms Claude Code into a robust, high-reliability development environment by introducing Eval-Driven Development (EDD). It enables developers to define rigorous success criteria before implementation, utilizing capability and regression evals to ensure AI-generated code meets specific functional standards. By incorporating deterministic code-based graders, probabilistic model-based graders, and human review gates, it provides a comprehensive reporting system with pass@k metrics. This allows for continuous tracking of AI reliability, prevents regressions in complex codebases, and provides a structured workflow for shipping production-ready code with confidence.

Key Features

01Standardized Eval-Driven Development (EDD) workflow

02Support for capability and regression testing frameworks

03Reliability tracking with pass@k and pass^k metrics

041 GitHub stars

05Multi-tier grading (Code-based, Model-based, and Human review)

06Automated evaluation reporting and baseline management

Use Cases

01Measuring and improving the consistency of Claude's output for critical code paths

02Preventing functional regressions during large-scale AI refactoring tasks

03Defining feature requirements as executable evals before implementation

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nixdorfer/claudecodetool eval-harness

For use in Claude.ai and ChatGPT

Download Skill