Why shouldn't I test against live LLM APIs in my CI pipeline?

Live APIs are non-deterministic, introduce high costs, and create slow feedback loops; this skill provides mocking and VCR patterns to ensure tests are fast, reliable, and free.

How does the skill handle asynchronous AI operations?

It provides standardized async timeout testing patterns to ensure your application correctly handles latency and polling scenarios without hanging indefinitely.

Does this skill help with JSON output validation?

Yes, it includes specific patterns for structured output validation to ensure that LLM-generated JSON strictly adheres to your required schemas.

Which LLM quality metrics can I measure with this skill?

This skill supports industry-standard metrics including Answer Relevancy, Faithfulness, Hallucination rates, and Context Precision using DeepEval and RAGAS.

LLM Testing & Validation

Name: LLM Testing & Validation
Author: yonatangross

byyonatangross

•

Security & Testing

Implements deterministic testing patterns and quality metrics for LLM-based applications using DeepEval and RAGAS.

The llm-testing skill provides a comprehensive framework for ensuring the reliability and quality of AI-powered applications. It enables developers to move beyond flaky, expensive live API tests by implementing deterministic mocking, VCR recordings, and automated quality metrics. By integrating tools like DeepEval and RAGAS, this skill allows for the systematic measurement of answer relevancy, faithfulness, and hallucination rates, while also offering robust patterns for handling async timeouts and validating structured JSON outputs in a production-ready environment.

Key Features

0169 GitHub stars

02DeepEval and RAGAS integration for RAG pipeline evaluation

03Async timeout handling and polling assertion strategies

04Deterministic LLM response mocking and VCR recording patterns

05Automated quality metrics for relevancy, faithfulness, and hallucination

06Structured JSON output and schema validation templates

Use Cases

01Eliminating slow and costly live API calls in CI/CD pipelines

02Testing application resilience against LLM timeouts and malformed JSON

03Validating the accuracy and context-alignment of RAG pipelines

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit llm-testing

For use in Claude.ai and ChatGPT

Download Skill