Can I use this for regression testing?

Yes, the skill includes specific templates and workflows for regression evals to ensure new AI-generated changes don't break existing functionality.

What types of graders are supported?

It supports deterministic code-based graders (like grep or test runners), probabilistic model-based graders using AI, and manual human review flags.

What is Eval-Driven Development (EDD)?

EDD is a methodology where you define evaluation criteria and success metrics for an AI agent before starting the actual code implementation, similar to Test-Driven Development (TDD).

How does pass@k measurement work?

It measures the probability that at least one of 'k' generated solutions passes the evaluation, providing a statistical view of an AI's reliability for a specific task.

AI Eval Harness

Name: AI Eval Harness
Author: flatrick

byflatrick

0•

Security & Testing

Implements Eval-Driven Development (EDD) to measure, benchmark, and ensure the reliability of AI-generated code changes.

The AI Eval Harness provides a formal framework for bringing rigorous testing to AI-assisted development by defining success criteria before implementation. It allows developers to track agent reliability using pass@k metrics, manage regression suites, and utilize multi-modal graders—including code-based, model-based, and human review—to ensure that Claude's outputs meet production standards. By treating evals as the unit tests of AI development, this skill helps teams benchmark performance across different models and prevent regressions during complex refactoring tasks.

Key Features

01Automated regression testing for prompt and model changes

020 GitHub stars

03Eval-Driven Development (EDD) workflow integration

04Automated pass@k and pass^k reliability metrics

05Standardized evaluation reporting and session history

06Support for code-based, model-based, and human graders

Use Cases

01Benchmarking AI agent performance across different model versions

02Establishing deterministic pass/fail criteria for complex coding tasks

03Preventing regressions in legacy codebases during AI-led refactoring

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add flatrick/mdt eval-harness

For use in Claude.ai and ChatGPT