How do I debug failing eval tests?

You can enable verbose output by setting E2E_VERBOSITY=verbose in your terminal, which displays the full conversation between the LLM and your MCP tools.

Can I run individual test files?

Yes, the suite supports standard Jest commands, allowing you to run specific unit tests or eval files by passing the file path to the test command.

How do I mock APIs in these tests?

You can use the USE_MOCK_API=true environment variable and the configureApiClient helper to set up a mock environment before running your tests.

What are MCP eval tests?

Eval tests use an LLM agent to interact with your tools end-to-end, simulating real-world usage scenarios to ensure workflow completion and reliability.

Which SDK does this skill support?

This skill is specifically designed for the @umbraco-cms/mcp-server-sdk, providing patterns for its testing and eval modules.

MCP Testing Suite

Name: MCP Testing Suite
Author: umbraco

byumbraco

0•

Security & Testing

Provides standardized patterns and utilities for unit testing and LLM-driven evaluation of Model Context Protocol (MCP) tools.

The MCP Testing Suite skill streamlines the quality assurance process for Model Context Protocol (MCP) servers using the Umbraco MCP SDK. It offers pre-defined patterns for traditional unit testing—including request handler mocking and environment setup—alongside advanced 'eval' tests that use LLM agents to validate end-to-end tool workflows. This skill is ideal for developers who need to ensure their MCP tools handle complex logic correctly and maintain reliability through both static assertions and dynamic scenario simulations.

Key Features

010 GitHub stars

02Mocking utilities for MCP request contexts and API clients

03Ready-to-use configuration for Jest and environment setups

04LLM-driven eval testing for end-to-end scenario validation

05Verbose debugging support for complex tool interactions

06Standardized unit testing patterns for MCP tool handlers

Use Cases

01Writing unit tests for individual MCP tools with mocked API responses

02Conducting end-to-end workflow validation using LLM agents to simulate user tasks

03Debugging MCP server failures through verbose logging and specialized test environments

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add umbraco/umbraco-mcp-base mcp-testing

For use in Claude.ai and ChatGPT

Key Features

010 GitHub stars

02Mocking utilities for MCP request contexts and API clients

03Ready-to-use configuration for Jest and environment setups

04LLM-driven eval testing for end-to-end scenario validation

05Verbose debugging support for complex tool interactions

06Standardized unit testing patterns for MCP tool handlers

Use Cases

01Writing unit tests for individual MCP tools with mocked API responses

02Conducting end-to-end workflow validation using LLM agents to simulate user tasks

03Debugging MCP server failures through verbose logging and specialized test environments

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add umbraco/umbraco-mcp-base mcp-testing

For use in Claude.ai and ChatGPT