Does this require specific observability software?

While the skill provides implementation patterns for tools like Langfuse, the underlying logic is adaptable to most modern logging and trace analysis platforms.

How does this skill detect gibberish outputs?

It uses a two-tier approach: first, quick heuristic checks for length and repetitiveness, followed by an 'LLM-as-judge' scoring system to evaluate response quality.

How is tool skipping identified?

The skill compares a predefined list of expected tools for a specific workflow against the actual tool calls recorded in the execution trace, alerting you if required steps are missed.

What is a silent failure in an LLM agent?

A silent failure occurs when an agent completes a task without throwing a technical error, yet the output is incorrect, nonsensical, or fails to utilize the necessary tools.

Can this skill help control LLM API costs?

Yes, it includes loop detection logic that monitors iteration counts and token usage spikes, enabling you to trigger circuit breakers before costs escalate.

Silent Failure Detection

Name: Silent Failure Detection
Author: yonatangross

byyonatangross

•

Analytics & Monitoring

Detects and alerts on hidden LLM agent failures such as tool skipping, infinite loops, and degraded output quality.

This skill provides a comprehensive monitoring framework for AI agents, specifically designed to catch 'silent failures' where an agent appears to function but produces incorrect or low-quality results. It implements sophisticated patterns for identifying skipped tool calls, detecting gibberish through LLM-as-judge scoring, and stopping runaway costs by monitoring token consumption spikes and infinite loops. By establishing statistical baselines and anomaly detection thresholds, this skill ensures that developers can maintain high reliability in production agentic workflows where traditional error codes often fail to trigger.

Key Features

01Production-ready alerting patterns for non-error logic failures

02Infinite loop and token consumption spike monitoring

03Statistical anomaly detection using Z-score baselines

04Quality scoring using LLM-as-judge and heuristic pre-filters

0569 GitHub stars

06Tool skipping detection via execution trace analysis

Use Cases

01Identifying middleware interference that causes agents to bypass required tools

02Monitoring production AI agents for logic regressions and unexpected behavior

03Preventing runaway API costs by detecting stuck agents in infinite loops

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit silent-failure-detection

For use in Claude.ai and ChatGPT

Download Skill