+ AI-generate scenarios based on the collection's tools. Requires authentication and
+ collection ownership.
+
+
+
Path Parameters
+
+
+ id - Collection ID
+
+
+
+
Request Body
+
+
+
Example Request
+
+
+
Example Response
+
+
+
+
Similarity Warning
+
+ When similarity.maxSimilarity exceeds 0.7, the scenario is similar to an
+ existing one. The mostSimilar field shows which scenario it overlaps
+ with.
+
+
+
+
+
+ {/* Run Scenario */}
+
+
+ Run Scenario
+ POST /api/scenarios/:id/run
+
+
+
+ Execute a scenario and return the results. Requires authentication. Consumes one run
+ from your daily quota.
+
+
+
Path Parameters
+
+
+ id - Scenario ID
+
+
+
+
Example Request
+
+
+
Example Response
+
+
+
Failure Response
+
+
+
+
+ {/* Get Scenario Runs */}
+
+
+ Get Scenario Run History
+ GET /api/scenarios/:id/runs
+
+
+
+ Retrieve the run history for a scenario. Requires authentication if the scenario belongs
+ to a private collection.
+
+
+
Path Parameters
+
+
+ id - Scenario ID
+
+
+
+
Query Parameters
+
+
+ limit - Number of results (default: 10, max: 50)
+
+ Check if a prompt is similar to existing scenarios in a collection. Useful before
+ creating manual scenarios to avoid duplicates.
+
+
+
Request Body
+
+
+
Example Response
+
+
+
+
Similarity Threshold
+
+ A maxSimilarity above 0.7 indicates significant overlap with existing
+ scenarios. Consider modifying your prompt to test different aspects of your tools.
+
+
+
+
+
+ {/* Featured Scenarios */}
+
+
+ Get Featured Scenarios
+ GET /api/scenarios/featured
+
+
+
+ Retrieve featured scenarios for the homepage showcase. Returns a mix of high-quality,
+ diverse, and recently successful scenarios.
+
+
+
Query Parameters
+
+
+ limit - Number of results (default: 10, max: 20)
+
+ );
+}
diff --git a/apps/web/src/app/docs/scenarios/page.tsx b/apps/web/src/app/docs/scenarios/page.tsx
new file mode 100644
index 0000000..482eab7
--- /dev/null
+++ b/apps/web/src/app/docs/scenarios/page.tsx
@@ -0,0 +1,614 @@
+import { Card, CardContent, CardDescription, CardHeader, CardTitle } from '@tpmjs/ui/Card/Card';
+import { CodeBlock } from '@tpmjs/ui/CodeBlock/CodeBlock';
+import type { Metadata } from 'next';
+import Link from 'next/link';
+
+export const metadata: Metadata = {
+ title: 'Scenarios Guide | TPMJS Docs',
+ description: 'Test your tool collections with AI-generated scenarios and automated evaluation',
+};
+
+export default function ScenariosPage(): React.ReactElement {
+ return (
+
+
+
Scenarios
+
+ Scenarios are AI-generated test cases for your tool collections. They automatically verify
+ that your tools work correctly and provide quality metrics over time.
+
+
+
+ {/* Overview */}
+
+
+ What are Scenarios?
+ Automated testing for tool collections
+
+
+
+ A Scenario is a test case that exercises your tool collection with a realistic user
+ prompt. When you run a scenario:
+
+
+
An ephemeral agent is created with your collection's tools
+
The agent executes the scenario's prompt
+
An LLM evaluates whether the task was completed successfully
+
Results are recorded and quality scores are updated
+
+
+ This enables continuous testing of your tool collections, similar to unit tests for
+ code.
+
+
+
+
+ {/* Use Cases */}
+
+
+ Common Archetypes
+ Typical scenarios for different tool types
+
+
+
+
Web Scraping Tools
+
+ Test data extraction from various webpage structures:
+
+
+
+
+
+
Search Tools
+
+ Verify search accuracy and relevance:
+
+
+
+
+
+
Data Processing Tools
+
+ Test transformations and calculations:
+
+
+
+
+
+
File/Document Tools
+
+ Verify file operations and content extraction:
+
+
+
+
+
+
+ {/* CLI Installation */}
+
+
+ Getting Started
+ Install the CLI and authenticate
+
+
+
Install the TPMJS CLI
+
+
+
Authenticate
+
+ You need a TPMJS API key to run scenarios. Get one from your{' '}
+
+ dashboard settings
+
+ .
+
+ AI-generate test scenarios based on your collection's tools. The generator analyzes
+ your tools and creates realistic prompts.
+
+
+
Basic Usage
+
+
+
Example Output
+
+
+
+
Similarity Detection
+
+ Generated scenarios are checked against existing ones using vector similarity. If a
+ scenario is >70% similar to an existing one, you'll see a warning. This helps
+ maintain diverse test coverage.
+
+
+
+
+
+ {/* List Scenarios */}
+
+
+ List Scenarios
+ tpm scenario list
+
+
+
+ View all scenarios for a collection or browse public scenarios.
+
+
+
Usage
+
+
+
Example Output
+
+
+
+
+ {/* Run Scenarios */}
+
+
+ Run All Scenarios
+ tpm scenario run
+
+
+
+ Execute all scenarios for a collection. This is ideal for CI/CD pipelines or batch
+ testing.
+
+
+
Usage
+
+
+
Example Output
+
+
+
Exit Codes
+
+
+ 0 - All scenarios passed
+
+
+ 1 - One or more scenarios failed
+
+
+
+
+
+ {/* Test Single Scenario */}
+
+
+ Test Single Scenario
+ tpm scenario test
+
+
+
+ Run a single scenario by its ID. Useful for debugging or re-testing specific failures.
+
+ Quality scores help identify reliable scenarios and track improvement over time. Scores
+ range from 0% to 100%.
+
+
+
Streak-Based Scoring
+
+ Scenarios earn bonus points for consecutive passes and lose points for consecutive
+ failures:
+
+
+
+
+
+ On Pass
+
+
+
+5% base score
+
+1% per consecutive pass
+
Maximum score: 100%
+
+
+
+
+ On Failure
+
+
+
-10% base penalty
+
-2% per consecutive fail
+
Minimum score: 0%
+
+
+
+
+
+
Example
+
+ A scenario with 5 consecutive passes would have a quality score of approximately 85%
+ (5% × 5 + 1% × (1+2+3+4+5) = 25% + 15% = 40%, plus base score). High-quality scenarios
+ are featured on the TPMJS homepage showcase.
+