What is LLM-as-judge in this context?

LLM-as-judge is an automated evaluation pattern where a high-capability model is used to score agent responses based on specific criteria like helpfulness, safety, or tone.

Which observability frameworks are supported?

The skill defines universal span attributes that work with most systems, but provides specific implementation examples for Langfuse and Braintrust.

How does it help with hallucination detection?

The skill provides a structured claim-checking pattern that compares agent responses against source documents to calculate a groundedness score.

Can this skill track human feedback?

Yes, it includes standardized patterns for capturing user ratings, corrections, and comments, and linking that feedback directly to the specific execution trace for debugging.

Agent Quality & Evaluation Instrumentation

Name: Agent Quality & Evaluation Instrumentation
Author: nexus-labs-automation

bynexus-labs-automation

Analytics & Monitoring

Instruments AI agent evaluation systems to track response quality, detect hallucinations, and monitor performance through automated and human feedback loops.

About

The Agent Quality & Evaluation skill empowers developers to build robust observability into AI agents by implementing structured evaluation metrics and feedback loops. It provides standardized patterns for LLM-as-judge scoring, human feedback collection, and ground truth comparisons across dimensions like correctness, helpfulness, and safety. By linking evaluations directly to execution traces, it enables granular debugging of failures and ensures that agent performance is quantitatively measured and continuously improved from development through to production.

Key Features

Integration patterns for Langfuse and Braintrust observability frameworks
Hallucination detection and factual accuracy metrics
Human feedback instrumentation for thumbs-up/down and user comments
0 GitHub stars
Automated LLM-as-judge scoring and heuristic patterns
Standardized span attributes for quality, safety, and relevance

Use Cases

Detecting and preventing hallucinations in RAG-based AI applications
Building a continuous evaluation pipeline to compare agent versions against a baseline
Measuring the impact of prompt engineering changes on agent response quality

About

Key Features

Integration patterns for Langfuse and Braintrust observability frameworks
Hallucination detection and factual accuracy metrics
Human feedback instrumentation for thumbs-up/down and user comments
0 GitHub stars
Automated LLM-as-judge scoring and heuristic patterns
Standardized span attributes for quality, safety, and relevance

Use Cases

Detecting and preventing hallucinations in RAG-based AI applications
Building a continuous evaluation pipeline to compare agent versions against a baseline
Measuring the impact of prompt engineering changes on agent response quality