How does this skill help with agent memory management?

It provides standardized methods to monitor short-term conversation history, long-term persistent memory, and episodic summaries, ensuring agents retain and access relevant information correctly.

What is RAG instrumentation?

RAG instrumentation involves tracking the specific steps an AI takes to retrieve external information, including the sources found, the relevance scores of those sources, and how that retrieved context influenced the final response.

Does this work with popular AI frameworks?

Yes, the skill includes specific implementation patterns and decorators for integrating observability into LangChain retrievers and LlamaIndex query engines.

Why should I track context window utilization?

Tracking utilization helps identify when an LLM is nearing its token limit, which prevents silent truncation of critical data and helps developers manage input costs effectively.

Memory and RAG Instrumentation

Name: Memory and RAG Instrumentation
Author: nexus-labs-automation

bynexus-labs-automation

Analytics & Monitoring

Instruments retrieval-augmented generation and memory operations to provide deep visibility into AI agent context and retrieval quality.

About

This skill provides a comprehensive framework for instrumenting RAG (Retrieval-Augmented Generation) pipelines and agent memory systems within your codebase. It enables developers to track exactly what sources were retrieved, evaluate their relevance through granular quality signals, and monitor context window utilization to prevent truncation errors. By implementing standardized spans for query processing, vector searches, and reranking, it transforms the often-opaque RAG process into a transparent, debuggable workflow, allowing for better optimization of LLM responses and precise cost tracking.

Key Features

Comprehensive RAG tracing for vector stores and retrieval sources
Detailed memory tracking across short-term, long-term, and episodic stores
Automated context window management and token utilization tracking
Advanced quality signal monitoring including relevance, coverage, and diversity scores
0 GitHub stars
Seamless integration patterns for LangChain and LlamaIndex frameworks

Use Cases

Debugging why an AI agent provided irrelevant answers by analyzing retrieval scores and sources
Monitoring context window usage to prevent silent truncation and manage high token costs
Optimizing vector search performance and reranking latency in production RAG pipelines

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add nexus-labs-automation/agent-observability memory-rag-instrumentation

For use in Claude.ai and ChatGPT

Download Skill

GitHub