feat: add dynamic tool loading system with Railway executor

Implements a complete dynamic tool loading system that allows the playground to discover and load tools from the TPMJS registry at runtime.

**Architecture:**
- Search tool package (@tpmjs/search-registry) - Searches registry for tools
- Search API endpoint (/api/tools/search) - Text-based search with scoring
- Pre-flight tool loading - Automatically searches and loads tools on every message
- Railway executor service (Deno) - Loads tools from esm.sh via HTTP imports
- Dynamic tool loader - Calls Railway to load and execute tools remotely

**Key Components:**

1. Railway Executor (apps/railway-executor/)
   - Deno-based service that natively supports HTTP imports
   - Endpoints: /load-and-describe, /execute-tool, /cache/stats, /cache/clear
   - Deploys to Railway with deno run --allow-net --allow-env server.ts

2. Search Tool Package (packages/tools/search-registry/)
   - AI SDK v6 tool for searching TPMJS registry
   - Uses jsonSchema + inputSchema pattern
   - Searches /api/tools/search endpoint

3. Search API (apps/web/src/app/api/tools/search/)
   - Text-based search with composite scoring
   - Scores: text relevance + quality boost + download boost
   - Returns tool metadata with importUrl for dynamic loading

4. Dynamic Tool Loader (apps/playground/src/lib/dynamic-tool-loader.ts)
   - Calls Railway service to load tools from esm.sh
   - Creates tool wrappers that execute remotely
   - Process-level module cache + per-conversation tracking

5. Pre-flight Loading (apps/playground/src/app/api/chat/route.ts)
   - Automatically searches for tools on every user message
   - Loads top 5 matching tools before agent processes request
   - Merges with static tools for seamless experience

**Technical Decisions:**
- Deno over Node.js: Native HTTP import support without flags
- Remote execution: Tools run in Railway sandbox, not Vercel
- Pre-flight loading: Better UX than two-turn search pattern
- Text search: BM25 had dependency issues, simple scoring works well

**Environment Variables:**
- RAILWAY_SERVICE_URL: https://endearing-commitment-production.up.railway.app

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Ajax Davis 2025-12-04 08:25:53 +10:00
parent 141a64d888
commit d597a71eb4
28 changed files with 37043 additions and 27 deletions

View file

@ -759,4 +759,46 @@ export const manualTools: ManualTool[] = [
},
websiteUrl: 'https://valyu.ai',
},
{
npmPackageName: '@tpmjs/search-registry',
category: 'integration',
frameworks: ['vercel-ai'],
exportName: 'searchTpmjsToolsTool',
description: 'Search the TPMJS tool registry to find AI SDK tools by keyword, category, or description. Returns tool metadata for dynamic loading.',
parameters: [
{
name: 'query',
type: 'string',
description: 'Search query (keywords, tool names, descriptions)',
required: true,
},
{
name: 'category',
type: 'string',
description: 'Filter by category (optional)',
required: false,
},
{
name: 'limit',
type: 'number',
description: 'Maximum results (1-20, default 10)',
required: false,
},
],
returns: {
type: 'object',
description: 'Search results with tool metadata including packageName, exportName, version, importUrl',
},
aiAgent: {
useCase: "Use when you need a tool that isn't currently available. Search for tools by keyword or domain.",
examples: [
'Search for "weather" when asked about weather',
'Search for "wikipedia" when asked to search Wikipedia',
'Search for "database" when working with SQL',
],
},
tags: ['search', 'tool-discovery', 'registry'],
websiteUrl: 'https://tpmjs.com',
docsUrl: 'https://tpmjs.com/spec',
},
];