1. Scenario Definition
-- Scenarios are defined as test cases with a prompt, expected outputs, and optional - assertions. They can be created manually or AI-generated. +
Developers Guide: Scenarios
++ Scenarios are TPMJS's AI-powered testing and evaluation system for tool + collections. They provide automated testing, quality tracking, and regression + prevention for your tools.
-+ A Scenario is a test case that simulates how a real user would interact with your + tool collection. Unlike traditional unit tests that verify individual functions, + scenarios exercise your tools end-to-end with realistic prompts and assertions. +
+ +Key Components:
+-
+
- + AI Agent Execution: An ephemeral agent is created with your + collection's tools + +
- + Realistic Prompt: The agent executes your scenario's + prompt against the tools + +
- + LLM Evaluation: An LLM analyzes whether the task was completed + successfully + +
- + Result Recording: All results are stored for quality tracking + and historical analysis + +
+ Think of scenarios as integration tests with AI intelligenceโinstead of hard-coded + assertions, scenarios use natural language evaluation to verify your tools work as + intended. +
+๐งช Continuous Testing
++ Run scenarios automatically on every code change to catch regressions early. +
+๐ฏ AI-Powered Validation
++ Use LLM evaluation to verify your tools actually solve real problems, not just + pass code tests. +
+๐ Quality Metrics
++ Track quality scores over time to identify reliable scenarios and areas for + improvement. +
+๐ CI/CD Ready
++ Integrate seamlessly into your pipeline with JSON output and exit codes. +
+When to Use Scenarios
+-
+
- + Multi-Tool Integration: Testing how multiple tools work + together in complex workflows + +
- + API-First Tools: Validating tools that make HTTP requests or + parse unstructured data + +
- + Quality-Critical Collections: When tool reliability impacts + user experience + +
- + Regression Prevention: Before deploying changes that might + break existing functionality + +
- + Documentation-Driven Testing: Ensuring tools work as described + in their public documentation + +
1. Scenario Definition
++ Scenarios are defined as test cases with a prompt, expected outputs, and optional + assertions. They can be created manually or AI-generated. +
+2. Agent Execution
-- When a scenario runs, an ephemeral AI agent is created with your tool collection. The - agent executes the scenario prompt using only the tools in that collectionโno external - access, no additional context. -
-2. Agent Execution
++ When a scenario runs, an ephemeral AI agent is created with your tool collection. + The agent executes the scenario prompt using only the tools in that collectionโno + external access, no additional context. +
+3. LLM Evaluation
-- After execution, a powerful LLM evaluates the results. This isn't just checking - for errorsโit uses natural language understanding to verify the task was completed - correctly. -
-3. LLM Evaluation
++ After execution, a powerful LLM evaluates the results. This isn't just + checking for errorsโit uses natural language understanding to verify the task was + completed correctly. +
+