Question Creator

About

Streamline the creation of benchmark test cases for natural language models and agents within the CAC evaluation framework. This skill automates the generation of standardized directory structures and essential files, including metadata, prompts, and reference answers, while enforcing strict numbering rules and formatting guidelines. It is an essential tool for researchers and developers looking to expand evaluation datasets with consistent, machine-readable test questions that include specific scoring indicators and difficulty levels.

Key Features

• Automated metadata configuration with difficulty tiers from base to final+
• Standardized directory and file structure generation for meta.yaml, prompt.md, and reference.md
• Integration with domain-specific scoring metrics such as accuracy, code quality, and robustness
• Support for multiple evaluation domains including code, mathematics, and logical reasoning
• 18 GitHub stars
• Enforcement of sequential numbering and naming conventions for repository consistency

Use Cases

• Automating the boilerplate setup for complex multi-file model performance tests
• Expanding the CAC benchmark suite with new domain-specific evaluation tasks
• Standardizing existing test questions to comply with machine-readable metadata formats