diff --git a/CLAUDE.md b/CLAUDE.md index 6995eac..3e84efc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1167,4 +1167,132 @@ curl https://tpmjs.com/api/health 3. Fetch `/api/health` and compare `commitSha` with your local commit 4. If they match, the deployment is live +**Note:** Vercel provides these values via environment variables (`VERCEL_GIT_COMMIT_SHA`, `VERCEL_GIT_COMMIT_MESSAGE`, `VERCEL_URL`) which are automatically available at runtime. + +--- + +## Health Check Side Effects System + +The health check system tests every registered tool by importing it and executing it with test parameters. For tools that create real resources on external platforms (e.g., `createService`, `createSession`), this creates orphaned resources that are never cleaned up. + +### Problem + +The health check service (`apps/web/src/lib/health-check/health-check-service.ts`) runs two checks per tool: + +1. **Import check** (`/load-and-describe`): Verifies the tool can be loaded and has valid `description` and `inputSchema`. +2. **Execution check** (`/execute-tool`): Executes the tool with auto-generated test parameters via the Railway executor. + +The Railway executor (`apps/railway-executor/server.ts`) injects real API keys from the package's env vars stored in the database before calling `tool.execute()`. There is no dry-run mode. This means tools like `createService` actually create services on the Unsandbox platform. + +### Solution: `healthCheck` Config in tpmjs Spec + +Tool authors can declare per-tool health check behavior via the `healthCheck` field in `package.json`: + +```json +{ + "tpmjs": { + "tools": [ + { + "name": "createService", + "healthCheck": { + "testParams": { "name": "tpmjs-hc-{{timestamp}}" }, + "cleanup": [ + { "tool": "deleteService", "mapping": { "service_id": "service_id" } } + ] + } + }, + { + "name": "getSession", + "healthCheck": { "skipExecution": true } + }, + { + "name": "listJobs" + } + ] + } +} +``` + +### Config Options + +| Field | Type | Description | +|-------|------|-------------| +| `skipExecution` | `boolean` | Skip execution check entirely. Only verify import. Use for tools that require existing external resources (would 404 with fake IDs). | +| `testParams` | `Record` | Override auto-generated test parameters with known-good values. String values support `{{timestamp}}` template variable (replaced with `Date.now()`). | +| `cleanup` | `Array<{ tool, mapping }>` | Ordered cleanup steps to run after execution. Each step calls a tool from the same package with params mapped from the execution result. | + +### Architecture + +**Schema:** `ToolHealthCheckConfigSchema` and `ToolHealthCheckCleanupStepSchema` in `packages/types/src/tpmjs.ts`. + +**Database:** `healthCheckConfig Json? @db.JsonB` column on the `Tool` model in `packages/db/prisma/schema.prisma`. + +**Sync:** The 3 sync routes (`/api/sync/changes`, `/api/sync/keyword`, `/api/sync/package`) store `toolDef.healthCheck` as `healthCheckConfig` during tool upsert. + +**Execution:** `checkExecutionHealth()` in the health check service reads the config from the DB and: +1. If `skipExecution: true` → returns HEALTHY immediately, no execution +2. If `testParams` → uses them (after `{{timestamp}}` template processing) instead of auto-generated params +3. After successful execution, if `cleanup` defined → runs each cleanup step in order via the executor +4. Default behavior unchanged for tools without config + +### Tool Classification Guide + +When adding `healthCheck` config to a new package: + +- **Read-only/list tools** (e.g., `listJobs`, `getLanguages`): No config needed. Safe by default. +- **Sync execution tools** (e.g., `execute`, `run`): Add `testParams` with valid input. No cleanup needed since execution is ephemeral. +- **Async execution tools** (e.g., `executeCodeAsync`): Add `testParams` + `cleanup` to delete the created job. +- **Resource creation tools** (e.g., `createSession`, `createService`): Add `testParams` + `cleanup` to delete the created resource. +- **Operations on existing resources** (e.g., `getSession`, `freezeService`, `deleteSnapshot`): Add `skipExecution: true`. These would 404 with fake IDs. + +### Files Involved + +| File | Role | +|------|------| +| `packages/types/src/tpmjs.ts` | Zod schemas for `ToolHealthCheckConfigSchema` | +| `packages/db/prisma/schema.prisma` | `healthCheckConfig` column on Tool model | +| `apps/web/src/app/api/sync/changes/route.ts` | Stores config during sync | +| `apps/web/src/app/api/sync/keyword/route.ts` | Stores config during sync | +| `apps/web/src/app/api/sync/package/route.ts` | Stores config during sync | +| `apps/web/src/lib/health-check/health-check-service.ts` | Reads config, skip/cleanup logic | +| `packages/tools/official/unsandbox/package.json` | Reference implementation with all tools configured | + +--- + +## Environment Setup + +### Prerequisites + +- **Node.js >= 22** (project enforces this in `package.json` engines field) +- **pnpm** package manager (install via `corepack enable` or `npm install -g pnpm`) + +### First-Time Setup + +```bash +# Clone and install +git clone https://github.com/tpmjs/tpmjs.git +cd tpmjs +pnpm install + +# Generate Prisma client (required before type-checking) +pnpm --filter=@tpmjs/db db:generate + +# Build all packages (needed for cross-package imports) +pnpm build + +# Verify everything works +pnpm type-check +pnpm lint +``` + +### Common Issues + +**`pnpm` not found:** Run `corepack enable` to enable pnpm via Node's corepack. Alternatively, use `npx -y pnpm` as a fallback. + +**Node version mismatch:** The project requires Node >= 22. If you see engine warnings, upgrade Node via nvm: `nvm install 22 && nvm use 22`. + +**Prisma client not generated:** After schema changes or fresh installs, always run `pnpm --filter=@tpmjs/db db:generate`. The postinstall hook does this automatically during `pnpm install`, but manual runs may be needed after schema modifications. + +**Database migrations:** After adding columns to `schema.prisma`, create and apply a migration: `pnpm --filter=@tpmjs/db db:migrate` (creates SQL migration file and applies it). For development, `pnpm --filter=@tpmjs/db db:push` pushes schema changes without creating a migration file. + **Note:** Vercel provides these values via environment variables (`VERCEL_GIT_COMMIT_SHA`, `VERCEL_GIT_COMMIT_MESSAGE`, `VERCEL_URL`) which are automatically available at runtime. \ No newline at end of file diff --git a/apps/web/src/app/api/sync/changes/route.ts b/apps/web/src/app/api/sync/changes/route.ts index e83a83a..132d6a9 100644 --- a/apps/web/src/app/api/sync/changes/route.ts +++ b/apps/web/src/app/api/sync/changes/route.ts @@ -194,6 +194,8 @@ export async function POST(request: NextRequest) { returns: toolDef.returns ? (toolDef.returns as any) : undefined, // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility workaround aiAgent: toolDef.aiAgent ? (toolDef.aiAgent as any) : undefined, + // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility workaround + healthCheckConfig: toolDef.healthCheck ? (toolDef.healthCheck as any) : undefined, qualityScore: null, // Will be calculated by metrics sync // Schema will be extracted below schemaSource: toolDef.parameters ? 'author' : null, @@ -207,6 +209,8 @@ export async function POST(request: NextRequest) { returns: toolDef.returns ? (toolDef.returns as any) : undefined, // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility workaround aiAgent: toolDef.aiAgent ? (toolDef.aiAgent as any) : undefined, + // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility workaround + healthCheckConfig: toolDef.healthCheck ? (toolDef.healthCheck as any) : undefined, toolDiscoverySource, }, }); diff --git a/apps/web/src/app/api/sync/keyword/route.ts b/apps/web/src/app/api/sync/keyword/route.ts index 579aaf7..a81bc39 100644 --- a/apps/web/src/app/api/sync/keyword/route.ts +++ b/apps/web/src/app/api/sync/keyword/route.ts @@ -225,6 +225,8 @@ export async function POST(request: NextRequest) { returns: toolDef.returns ? (toolDef.returns as any) : undefined, // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility workaround aiAgent: toolDef.aiAgent ? (toolDef.aiAgent as any) : undefined, + // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility workaround + healthCheckConfig: toolDef.healthCheck ? (toolDef.healthCheck as any) : undefined, qualityScore: null, // Will be calculated by metrics sync // Schema will be extracted below schemaSource: toolDef.parameters ? 'author' : null, @@ -238,6 +240,8 @@ export async function POST(request: NextRequest) { returns: toolDef.returns ? (toolDef.returns as any) : undefined, // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility workaround aiAgent: toolDef.aiAgent ? (toolDef.aiAgent as any) : undefined, + // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility workaround + healthCheckConfig: toolDef.healthCheck ? (toolDef.healthCheck as any) : undefined, toolDiscoverySource, }, }); diff --git a/apps/web/src/app/api/sync/package/route.ts b/apps/web/src/app/api/sync/package/route.ts index 0975798..2cd7dff 100644 --- a/apps/web/src/app/api/sync/package/route.ts +++ b/apps/web/src/app/api/sync/package/route.ts @@ -202,6 +202,8 @@ export async function POST(request: NextRequest) { returns: toolDef.returns ? (toolDef.returns as any) : undefined, // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility aiAgent: toolDef.aiAgent ? (toolDef.aiAgent as any) : undefined, + // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility + healthCheckConfig: toolDef.healthCheck ? (toolDef.healthCheck as any) : undefined, qualityScore: null, schemaSource: toolDef.parameters ? 'author' : null, toolDiscoverySource, @@ -214,6 +216,8 @@ export async function POST(request: NextRequest) { returns: toolDef.returns ? (toolDef.returns as any) : undefined, // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility aiAgent: toolDef.aiAgent ? (toolDef.aiAgent as any) : undefined, + // biome-ignore lint/suspicious/noExplicitAny: Prisma Json type compatibility + healthCheckConfig: toolDef.healthCheck ? (toolDef.healthCheck as any) : undefined, toolDiscoverySource, }, }); diff --git a/apps/web/src/lib/health-check/health-check-service.ts b/apps/web/src/lib/health-check/health-check-service.ts index e39e16c..01e73c4 100644 --- a/apps/web/src/lib/health-check/health-check-service.ts +++ b/apps/web/src/lib/health-check/health-check-service.ts @@ -4,6 +4,7 @@ */ import { type HealthStatus, type Package, type Prisma, prisma, type Tool } from '@tpmjs/db'; +import type { ToolHealthCheckConfig } from '@tpmjs/types/tpmjs'; import { env } from '~/env'; const RAILWAY_EXECUTOR_URL = env.RAILWAY_EXECUTOR_URL; @@ -19,6 +20,82 @@ interface HealthCheckResult { overallStatus: HealthStatus; } +/** + * Parse healthCheckConfig from DB JSON to typed config. + * Returns null if not present or invalid. + */ +function parseHealthCheckConfig(tool: Tool): ToolHealthCheckConfig | null { + if (!tool.healthCheckConfig || typeof tool.healthCheckConfig !== 'object') return null; + return tool.healthCheckConfig as ToolHealthCheckConfig; +} + +/** + * Process template variables in test parameters. + * Supported: {{timestamp}} - replaced with Date.now() + */ +function processTestParams(params: Record): Record { + const processed: Record = {}; + for (const [key, value] of Object.entries(params)) { + if (typeof value === 'string') { + processed[key] = value.replace(/\{\{timestamp\}\}/g, String(Date.now())); + } else { + processed[key] = value; + } + } + return processed; +} + +/** + * Execute cleanup steps after a health check execution to undo side effects. + * Each step calls a tool from the same package with params mapped from the execution result. + * Best-effort: failures are logged but don't fail the health check. + */ +async function executeCleanup( + tool: Tool & { package: Package }, + cleanupSteps: NonNullable, + executionResult: Record +): Promise { + for (const step of cleanupSteps) { + try { + // Map params from execution result using the mapping config + const cleanupParams: Record = {}; + for (const [paramName, resultField] of Object.entries(step.mapping)) { + cleanupParams[paramName] = executionResult[resultField]; + } + + console.log( + ` Cleanup: calling ${step.tool} with params ${JSON.stringify(cleanupParams)}` + ); + + const response = await fetch(`${RAILWAY_EXECUTOR_URL}/execute-tool`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + packageName: tool.package.npmPackageName, + name: step.tool, + version: tool.package.npmVersion, + params: cleanupParams, + env: tool.package.env || {}, + }), + signal: AbortSignal.timeout(30000), + }); + + if (!response.ok) { + const data = await response.json().catch(() => ({})); + console.warn( + ` Cleanup warning: ${step.tool} returned ${response.status}: ${data.error || 'unknown error'}` + ); + } else { + console.log(` Cleanup: ${step.tool} succeeded`); + } + } catch (error) { + console.warn( + ` Cleanup warning: ${step.tool} failed: ${error instanceof Error ? error.message : 'unknown error'}` + ); + } + } +} + /** * Check if a tool can be imported (load-and-describe) */ @@ -100,6 +177,11 @@ async function checkImportHealth(tool: Tool & { package: Package }): Promise<{ * IMPORTANT: If the tool executes at all (even with errors), it's HEALTHY. * We only mark as BROKEN for infrastructure failures (timeouts, network errors). * Validation errors mean the tool IS working - it's correctly rejecting bad input. + * + * Respects healthCheckConfig from the tool's tpmjs spec: + * - skipExecution: skip execution entirely, only verify import + * - testParams: use author-provided params instead of auto-generated + * - cleanup: run cleanup steps after execution to undo side effects */ async function checkExecutionHealth(tool: Tool & { package: Package }): Promise<{ status: HealthStatus; @@ -108,9 +190,19 @@ async function checkExecutionHealth(tool: Tool & { package: Package }): Promise< testParams: Record; }> { const startTime = Date.now(); + const healthCheckConfig = parseHealthCheckConfig(tool); - // Generate test parameters based on tool schema - const testParams = generateTestParameters(tool); + // If tool declares skipExecution, skip execution health check entirely. + // This is for tools that require existing external resources (would 404 with fake IDs). + if (healthCheckConfig?.skipExecution) { + console.log(` Execution: skipped (healthCheck.skipExecution=true)`); + return { status: 'HEALTHY', error: null, timeMs: 0, testParams: {} }; + } + + // Use author-provided test params if available, otherwise auto-generate + const testParams = healthCheckConfig?.testParams + ? processTestParams(healthCheckConfig.testParams) + : generateTestParameters(tool); try { const response = await fetch(`${RAILWAY_EXECUTOR_URL}/execute-tool`, { @@ -132,7 +224,12 @@ async function checkExecutionHealth(tool: Tool & { package: Package }): Promise< // Any error in the response is from the tool itself (validation, env, etc.) // which means the tool IS working - it's correctly processing/rejecting input if (response.ok) { - // Executor responded - tool executed (success or tool-level error) + // Run cleanup steps if defined (undo side effects from execution) + if (healthCheckConfig?.cleanup?.length) { + const data = await response.json().catch(() => ({})); + const executionResult = data.result ?? data; + await executeCleanup(tool, healthCheckConfig.cleanup, executionResult); + } return { status: 'HEALTHY', error: null, timeMs, testParams }; } diff --git a/packages/db/prisma/schema.prisma b/packages/db/prisma/schema.prisma index 6c9eda0..182224c 100644 --- a/packages/db/prisma/schema.prisma +++ b/packages/db/prisma/schema.prisma @@ -85,6 +85,9 @@ model Tool { qualityScore Decimal? @map("quality_score") @db.Decimal(3, 2) // 0.00 to 1.00 likeCount Int @default(0) @map("like_count") + // Health Check Configuration (from tpmjs spec - declares side effects and cleanup) + healthCheckConfig Json? @map("health_check_config") @db.JsonB + // Health Status Fields importHealth HealthStatus? @default(UNKNOWN) @map("import_health") executionHealth HealthStatus? @default(UNKNOWN) @map("execution_health") diff --git a/packages/tools/official/unsandbox/package.json b/packages/tools/official/unsandbox/package.json index aa3bece..61001cf 100644 --- a/packages/tools/official/unsandbox/package.json +++ b/packages/tools/official/unsandbox/package.json @@ -48,23 +48,38 @@ "tools": [ { "name": "executeCodeAsync", - "description": "Execute code asynchronously in a secure sandbox. Returns a job_id immediately. Use getJob to check status and retrieve results. Supports 42+ languages." + "description": "Execute code asynchronously in a secure sandbox. Returns a job_id immediately. Use getJob to check status and retrieve results. Supports 42+ languages.", + "healthCheck": { + "testParams": { "language": "python", "code": "print('healthcheck')" }, + "cleanup": [{ "tool": "deleteJob", "mapping": { "job_id": "job_id" } }] + } }, { "name": "execute", - "description": "Execute code synchronously in a secure sandbox. Waits for completion and returns results directly. Best for quick scripts." + "description": "Execute code synchronously in a secure sandbox. Waits for completion and returns results directly. Best for quick scripts.", + "healthCheck": { + "testParams": { "language": "python", "code": "print('healthcheck')" } + } }, { "name": "run", - "description": "Execute code synchronously with automatic language detection via shebang (e.g., #!/usr/bin/env python)." + "description": "Execute code synchronously with automatic language detection via shebang (e.g., #!/usr/bin/env python).", + "healthCheck": { + "testParams": { "code": "#!/usr/bin/env python\nprint('healthcheck')" } + } }, { "name": "runAsync", - "description": "Execute code asynchronously with automatic language detection via shebang. Returns job_id for polling." + "description": "Execute code asynchronously with automatic language detection via shebang. Returns job_id for polling.", + "healthCheck": { + "testParams": { "code": "#!/usr/bin/env python\nprint('healthcheck')" }, + "cleanup": [{ "tool": "deleteJob", "mapping": { "job_id": "job_id" } }] + } }, { "name": "getJob", - "description": "Get the status and results of an async code execution job by job_id." + "description": "Get the status and results of an async code execution job by job_id.", + "healthCheck": { "skipExecution": true } }, { "name": "listJobs", @@ -72,7 +87,8 @@ }, { "name": "deleteJob", - "description": "Cancel an active code execution job by job_id." + "description": "Cancel an active code execution job by job_id.", + "healthCheck": { "skipExecution": true } }, { "name": "getLanguages", @@ -84,11 +100,16 @@ }, { "name": "createSession", - "description": "Create a new persistent session with configurable language, resources, and networking. Sessions maintain state across executions." + "description": "Create a new persistent session with configurable language, resources, and networking. Sessions maintain state across executions.", + "healthCheck": { + "testParams": { "shell": "bash" }, + "cleanup": [{ "tool": "deleteSession", "mapping": { "session_id": "session_id" } }] + } }, { "name": "getSession", - "description": "Get details of a session including status, resource usage, and configuration." + "description": "Get details of a session including status, resource usage, and configuration.", + "healthCheck": { "skipExecution": true } }, { "name": "listSessions", @@ -96,43 +117,56 @@ }, { "name": "executeInSession", - "description": "Execute code in an existing session. State and files persist between executions." + "description": "Execute code in an existing session. State and files persist between executions.", + "healthCheck": { "skipExecution": true } }, { "name": "freezeSession", - "description": "Freeze a session to pause execution and reduce resource usage while preserving state." + "description": "Freeze a session to pause execution and reduce resource usage while preserving state.", + "healthCheck": { "skipExecution": true } }, { "name": "unfreezeSession", - "description": "Unfreeze a frozen session to resume execution." + "description": "Unfreeze a frozen session to resume execution.", + "healthCheck": { "skipExecution": true } }, { "name": "lockSession", - "description": "Lock a session to prevent modifications or deletion." + "description": "Lock a session to prevent modifications or deletion.", + "healthCheck": { "skipExecution": true } }, { "name": "unlockSession", - "description": "Unlock a locked session to allow modifications." + "description": "Unlock a locked session to allow modifications.", + "healthCheck": { "skipExecution": true } }, { "name": "createSessionSnapshot", - "description": "Create a snapshot of the current session state for backup or cloning." + "description": "Create a snapshot of the current session state for backup or cloning.", + "healthCheck": { "skipExecution": true } }, { "name": "restoreSession", - "description": "Restore a session from a snapshot." + "description": "Restore a session from a snapshot.", + "healthCheck": { "skipExecution": true } }, { "name": "deleteSession", - "description": "Delete a session and release all associated resources." + "description": "Delete a session and release all associated resources.", + "healthCheck": { "skipExecution": true } }, { "name": "createService", - "description": "Create a long-running service with persistent state, networking, and auto-restart capabilities." + "description": "Create a long-running service with persistent state, networking, and auto-restart capabilities.", + "healthCheck": { + "testParams": { "name": "tpmjs-hc-{{timestamp}}" }, + "cleanup": [{ "tool": "deleteService", "mapping": { "service_id": "service_id" } }] + } }, { "name": "getService", - "description": "Get details of a service including status, endpoints, and resource usage." + "description": "Get details of a service including status, endpoints, and resource usage.", + "healthCheck": { "skipExecution": true } }, { "name": "listServices", @@ -140,59 +174,73 @@ }, { "name": "executeInService", - "description": "Execute a command or code snippet in a running service." + "description": "Execute a command or code snippet in a running service.", + "healthCheck": { "skipExecution": true } }, { "name": "freezeService", - "description": "Freeze a service to pause execution while preserving state." + "description": "Freeze a service to pause execution while preserving state.", + "healthCheck": { "skipExecution": true } }, { "name": "unfreezeService", - "description": "Unfreeze a frozen service to resume execution." + "description": "Unfreeze a frozen service to resume execution.", + "healthCheck": { "skipExecution": true } }, { "name": "lockService", - "description": "Lock a service to prevent modifications or deletion." + "description": "Lock a service to prevent modifications or deletion.", + "healthCheck": { "skipExecution": true } }, { "name": "unlockService", - "description": "Unlock a locked service to allow modifications." + "description": "Unlock a locked service to allow modifications.", + "healthCheck": { "skipExecution": true } }, { "name": "redeployService", - "description": "Redeploy a service with updated configuration or code." + "description": "Redeploy a service with updated configuration or code.", + "healthCheck": { "skipExecution": true } }, { "name": "getServiceLogs", - "description": "Get logs from a service with optional filtering by time range and log level." + "description": "Get logs from a service with optional filtering by time range and log level.", + "healthCheck": { "skipExecution": true } }, { "name": "createServiceSnapshot", - "description": "Create a snapshot of the current service state." + "description": "Create a snapshot of the current service state.", + "healthCheck": { "skipExecution": true } }, { "name": "getServiceEnv", - "description": "Get environment variables configured for a service." + "description": "Get environment variables configured for a service.", + "healthCheck": { "skipExecution": true } }, { "name": "setServiceEnv", - "description": "Set or update environment variables for a service." + "description": "Set or update environment variables for a service.", + "healthCheck": { "skipExecution": true } }, { "name": "deleteServiceEnv", - "description": "Delete an environment variable from a service." + "description": "Delete an environment variable from a service.", + "healthCheck": { "skipExecution": true } }, { "name": "deleteService", - "description": "Delete a service and release all associated resources." + "description": "Delete a service and release all associated resources.", + "healthCheck": { "skipExecution": true } }, { "name": "createSnapshot", - "description": "Create a snapshot from any source (session, service, or existing snapshot)." + "description": "Create a snapshot from any source (session, service, or existing snapshot).", + "healthCheck": { "skipExecution": true } }, { "name": "getSnapshot", - "description": "Get details of a snapshot including metadata and creation info." + "description": "Get details of a snapshot including metadata and creation info.", + "healthCheck": { "skipExecution": true } }, { "name": "listSnapshots", @@ -200,31 +248,38 @@ }, { "name": "lockSnapshot", - "description": "Lock a snapshot to prevent deletion or modification." + "description": "Lock a snapshot to prevent deletion or modification.", + "healthCheck": { "skipExecution": true } }, { "name": "unlockSnapshot", - "description": "Unlock a locked snapshot to allow modifications." + "description": "Unlock a locked snapshot to allow modifications.", + "healthCheck": { "skipExecution": true } }, { "name": "restoreSnapshot", - "description": "Restore a session or service from a snapshot." + "description": "Restore a session or service from a snapshot.", + "healthCheck": { "skipExecution": true } }, { "name": "cloneSnapshot", - "description": "Clone a snapshot to create a new independent copy." + "description": "Clone a snapshot to create a new independent copy.", + "healthCheck": { "skipExecution": true } }, { "name": "deleteSnapshot", - "description": "Delete a snapshot and free associated storage." + "description": "Delete a snapshot and free associated storage.", + "healthCheck": { "skipExecution": true } }, { "name": "publishImage", - "description": "Publish a snapshot as a reusable image for spawning new sessions or services." + "description": "Publish a snapshot as a reusable image for spawning new sessions or services.", + "healthCheck": { "skipExecution": true } }, { "name": "getImage", - "description": "Get details of a published image including metadata and access info." + "description": "Get details of a published image including metadata and access info.", + "healthCheck": { "skipExecution": true } }, { "name": "listImages", @@ -232,39 +287,48 @@ }, { "name": "lockImage", - "description": "Lock an image to prevent modifications or deletion." + "description": "Lock an image to prevent modifications or deletion.", + "healthCheck": { "skipExecution": true } }, { "name": "unlockImage", - "description": "Unlock a locked image to allow modifications." + "description": "Unlock a locked image to allow modifications.", + "healthCheck": { "skipExecution": true } }, { "name": "grantImageAccess", - "description": "Grant access to a private image for specific users or API keys." + "description": "Grant access to a private image for specific users or API keys.", + "healthCheck": { "skipExecution": true } }, { "name": "revokeImageAccess", - "description": "Revoke access to an image from specific users or API keys." + "description": "Revoke access to an image from specific users or API keys.", + "healthCheck": { "skipExecution": true } }, { "name": "transferImage", - "description": "Transfer ownership of an image to another user." + "description": "Transfer ownership of an image to another user.", + "healthCheck": { "skipExecution": true } }, { "name": "setImageVisibility", - "description": "Set image visibility to public or private." + "description": "Set image visibility to public or private.", + "healthCheck": { "skipExecution": true } }, { "name": "spawnFromImage", - "description": "Spawn a new session or service from an image." + "description": "Spawn a new session or service from an image.", + "healthCheck": { "skipExecution": true } }, { "name": "getImageTrustedKeys", - "description": "Get list of API keys that have access to a private image." + "description": "Get list of API keys that have access to a private image.", + "healthCheck": { "skipExecution": true } }, { "name": "deleteImage", - "description": "Delete an image and free associated storage." + "description": "Delete an image and free associated storage.", + "healthCheck": { "skipExecution": true } }, { "name": "healthCheck", diff --git a/packages/types/src/tpmjs.ts b/packages/types/src/tpmjs.ts index dec3ce5..d07fb7b 100644 --- a/packages/types/src/tpmjs.ts +++ b/packages/types/src/tpmjs.ts @@ -93,6 +93,46 @@ export const TpmjsAiAgentSchema = z.object({ export type TpmjsAiAgent = z.infer; +/** + * Health check cleanup step - defines how to undo a side effect after testing. + * The cleanup tool is called from the same package with params mapped from the execution result. + */ +export const ToolHealthCheckCleanupStepSchema = z.object({ + /** Tool export name to call for cleanup (from the same package) */ + tool: z.string().min(1), + /** Maps cleanup tool parameter names to fields in the execution result. + * e.g., { "service_id": "service_id" } means: pass result.service_id as the service_id param */ + mapping: z.record(z.string(), z.string()), +}); + +export type ToolHealthCheckCleanupStep = z.infer; + +/** + * Health check configuration for a tool. + * Tells the health check system how to safely test this tool without leaving orphaned resources. + * + * - Tools with no healthCheck config are assumed idempotent (safe to execute with auto-generated params). + * - Tools with `skipExecution: true` only get import checks (no execution). + * - Tools with `testParams` use those instead of auto-generated minimal values. + * - Tools with `cleanup` steps run those after execution to undo side effects. + * + * Template variables in testParams string values: + * - `{{timestamp}}` - replaced with Date.now() at execution time (for unique resource names) + */ +export const ToolHealthCheckConfigSchema = z.object({ + /** Skip execution health check entirely. Only verify the tool imports correctly. + * Use for tools that require existing external resources (would 404 with fake IDs). */ + skipExecution: z.boolean().optional(), + /** Override auto-generated test parameters with known-good values. + * String values support `{{timestamp}}` template variable. */ + testParams: z.record(z.string(), z.unknown()).optional(), + /** Ordered cleanup steps to run after execution to undo side effects. + * Each step calls a tool from the same package with params mapped from the execution result. */ + cleanup: z.array(ToolHealthCheckCleanupStepSchema).optional(), +}); + +export type ToolHealthCheckConfig = z.infer; + /** * Individual tool definition within a multi-tool package * @@ -101,6 +141,7 @@ export type TpmjsAiAgent = z.infer; * * Optional fields (auto-extracted if not provided): * - description: A description of what the tool does (20-500 chars) - auto-extracted from tool + * - healthCheck: Configuration for safe health checking (side effects, cleanup) * * @deprecated fields (now auto-extracted): * - parameters: Tool input parameters - auto-extracted from inputSchema @@ -118,6 +159,8 @@ export const TpmjsToolDefinitionSchema = z.object({ returns: TpmjsReturnsSchema.optional(), // @deprecated - now auto-extracted from tool aiAgent: TpmjsAiAgentSchema.optional(), + // Health check configuration - declares side effects and cleanup procedures + healthCheck: ToolHealthCheckConfigSchema.optional(), }); export type TpmjsToolDefinition = z.infer;