diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 5bbf39e..b758495 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -23,7 +23,7 @@ jobs: - uses: actions/setup-node@v4 with: - node-version: 21 + node-version: 22 cache: 'pnpm' registry-url: 'https://registry.npmjs.org' diff --git a/2025-BEST-PRACTICES.md b/2025-BEST-PRACTICES.md deleted file mode 100644 index 59676be..0000000 --- a/2025-BEST-PRACTICES.md +++ /dev/null @@ -1,275 +0,0 @@ -# 2025 Best Practices for TPMJS Monorepo - -This document outlines recommendations to make TPMJS a cutting-edge 2025 monorepo optimized for both human and agentic development (Claude Code, Cursor, etc.). - -## High-Impact Additions - -### 1. Agent-First Documentation - -``` -packages/docs/ -├── architecture-decisions/ # ADRs in markdown -├── patterns/ # Common patterns with examples -├── schemas/ # JSON schemas for all data structures -└── examples/ # Working code examples per feature -``` - -**Why:** Claude Code and other agents work better with: -- Explicit decision documentation (ADRs) -- Pattern libraries showing "the right way" -- Machine-readable schemas -- Real working examples to reference - -### 2. Automated Testing Pyramid - -```bash -# Add to package.json scripts -"test:unit": "vitest" # ✅ Already have this -"test:integration": "vitest -c vitest.integration.config.ts" # Add -"test:e2e": "playwright test" # Add -"test:visual": "playwright test --grep @visual" # Add -"test:contracts": "pactum" # Add for API testing -``` - -**Packages to add:** -- `@playwright/test` - E2E testing -- `@playwright/experimental-ct-react` - Component testing -- `pactum` or `msw` integration tests (you have mocks setup) -- `chromatic` or `percy` - Visual regression - -### 3. Type Coverage & Quality Gates - -```json -// Add to root package.json -{ - "scripts": { - "type-check": "tsc --noEmit", - "type-coverage": "type-coverage --at-least 95", - "find-deadcode": "knip", - "check-architecture": "depcruiser --validate" - } -} -``` - -**Add packages:** -- `type-coverage` - Ensure no implicit `any` -- `knip` - Find unused files/exports/dependencies -- `dependency-cruiser` - Enforce architecture rules -- `@total-typescript/ts-reset` - Better built-in types - -### 4. Development Containers - -```json -// .devcontainer/devcontainer.json -{ - "name": "TPMJS Dev", - "dockerComposeFile": "docker-compose.yml", - "service": "dev", - "features": { - "ghcr.io/devcontainers/features/node:1": {}, - "ghcr.io/devcontainers-contrib/features/pnpm:2": {} - }, - "customizations": { - "vscode": { - "extensions": [ - "biomejs.biome", - "bradlc.vscode-tailwindcss", - "lokalise.i18n-ally" - ] - } - } -} -``` - -**Why:** Agents like Claude Code work better when environment is reproducible. This also helps human developers. - -### 5. Code Generation & Scaffolding - -```typescript -// packages/cli/ - Internal dev tool -import { scaffold } from '@tpmjs/cli'; - -// Commands: -pnpm gen:component ButtonGroup -pnpm gen:package @tpmjs/new-package -pnpm gen:app marketing-site -``` - -**Create:** -- `plop` or `hygen` templates -- Component scaffolding (with tests, stories, exports) -- Package scaffolding (with tsconfig, package.json, exports) -- Consistent file structure generation - -**Why:** Agents can use these commands to create new code following your exact patterns. - -### 6. Enhanced Strict Mode TypeScript - -```json -// packages/tsconfig/base.json - Add these -{ - "compilerOptions": { - "exactOptionalPropertyTypes": true, - "noUncheckedIndexedAccess": true, - "noPropertyAccessFromIndexSignature": true, - "allowUnusedLabels": false, - "allowUnreachableCode": false, - "noImplicitOverride": true - } -} -``` - -### 7. Bundle Analysis & Performance - -```json -{ - "scripts": { - "analyze": "turbo run build --filter=@tpmjs/web -- --analyze", - "lighthouse": "lhci autorun", - "bundle-size": "size-limit" - } -} -``` - -**Add:** -- `@next/bundle-analyzer` -- `@lhci/cli` - Lighthouse CI -- `size-limit` - Bundle size tracking in CI - -### 8. Smart Dependency Management - -```json -// .github/renovate.json -{ - "extends": ["config:base"], - "packageRules": [ - { - "matchPackagePatterns": ["*"], - "matchUpdateTypes": ["minor", "patch"], - "groupName": "all non-major dependencies", - "groupSlug": "all-minor-patch" - } - ] -} -``` - -**Use:** Renovate or Dependabot with auto-merge for passing tests - -### 9. API Documentation Generation - -```bash -pnpm add -D -w typedoc typedoc-plugin-markdown -``` - -Auto-generate API docs from TSDoc comments that both humans and agents can read. - -### 10. Schema-First Development - -```typescript -// packages/schemas/ - Central schema definitions -export * from './tool-schema'; -export * from './registry-api-schema'; -export * from './event-schema'; - -// Use Zod for runtime + type generation -// Agents can read schemas to understand contracts -``` - -## Monorepo-Specific Improvements - -### 11. Better Local Development - -```typescript -// turbo.json -{ - "pipeline": { - "dev": { - "cache": false, - "persistent": true, - "dependsOn": ["^build"] - }, - "build": { - "dependsOn": ["^build"], - "outputs": ["dist/**", ".next/**"] - } - } -} -``` - -### 12. Workspace Protocols & Constraints - -```yaml -# .pnpm-workspace.yaml -packages: - - 'apps/*' - - 'packages/*' - -# Add constraints -pnpm-workspace-constraints: - dependencies: - '@tpmjs/ui': 'workspace:*' - '@tpmjs/utils': 'workspace:*' -``` - -## Recommended Final Structure - -``` -. -├── .devcontainer/ # Dev containers config -├── .github/ -│ ├── workflows/ # CI/CD -│ └── renovate.json # Dependency automation -├── apps/ -│ └── web/ -├── packages/ -│ ├── cli/ # ⭐ NEW: Dev tooling -│ ├── schemas/ # ⭐ NEW: Central schemas -│ └── ...existing -├── docs/ -│ ├── adr/ # ⭐ NEW: Architecture decisions -│ ├── patterns/ # ⭐ NEW: Code patterns -│ └── examples/ # ⭐ NEW: Working examples -├── scripts/ -│ ├── scaffold.ts # ⭐ NEW: Code generation -│ └── validate-deps.ts # ⭐ NEW: Architecture validation -├── playwright.config.ts # ⭐ NEW: E2E testing -├── .lighthouserc.json # ⭐ NEW: Performance -└── knip.json # ⭐ NEW: Dead code detection -``` - -## Priority Order for Implementation - -### Phase 1 (Foundation) -1. **Knip + type-coverage** - Catch issues early -2. **Code generation scripts** - Ensure consistency -3. **ADR documentation structure** - Decision tracking - -### Phase 2 (Quality) -4. **E2E testing with Playwright** - Full user flow coverage -5. **Bundle analysis + performance budgets** - Keep app fast -6. **Stricter TypeScript settings** - Catch more bugs at compile time - -### Phase 3 (DX) -7. **Dev containers** - Reproducible environments -8. **API documentation generation** - Auto-generated from code -9. **Renovate automation** - Keep dependencies fresh - -## Benefits for Agent-Driven Development - -1. **Explicit Patterns** - Agents can reference documented patterns instead of guessing -2. **Code Generation** - Consistent scaffolding commands agents can use -3. **Machine-Readable Schemas** - JSON schemas help agents understand data structures -4. **Quality Gates** - Automated checks catch agent mistakes early -5. **Working Examples** - Agents can copy-paste-adapt proven patterns -6. **Architecture Enforcement** - Dependency rules prevent agents from creating invalid imports - -## Next Steps - -Start with the highest ROI items: -1. Install Knip to find dead code -2. Set up code generation for components/packages -3. Create docs/patterns/ with common examples -4. Add stricter TypeScript compiler options -5. Set up Playwright for E2E testing - -These changes will make the codebase more maintainable and significantly improve the experience of working with AI coding agents. diff --git a/API_ROUTES_TIMEOUT_INVESTIGATION.md b/API_ROUTES_TIMEOUT_INVESTIGATION.md deleted file mode 100644 index 575a6bd..0000000 --- a/API_ROUTES_TIMEOUT_INVESTIGATION.md +++ /dev/null @@ -1,932 +0,0 @@ -# API Routes Timeout Issue - Complete Investigation Report - -## Problem Statement - -API routes deployed to Vercel are timing out with no response. The Next.js application pages work perfectly, but all API endpoints at `/api/*` return timeouts or "Redirecting..." messages. - -**Affected URLs:** -- `https://tpmjs.com/api/health` - Returns "Redirecting..." -- `https://tpmjs.com/api/tools` - Returns "Redirecting..." -- `https://tpmjs-1chh44d1u-tpmjs.vercel.app/api/health` - Timeouts (exit code 28) -- `https://tpmjs-1chh44d1u-tpmjs.vercel.app/api/tools` - Timeouts (exit code 28) - -**Working:** -- All page routes work correctly (e.g., `/`, `/tool/[slug]`) -- UI navigation and client-side routing function normally -- Local development API routes work perfectly - -## Environment Details - -### Project Structure -- **Monorepo:** Turborepo setup with pnpm workspaces -- **Framework:** Next.js 16.0.4 (App Router) -- **Node Version:** 24.x (on Vercel) -- **Deployment Platform:** Vercel -- **Custom Domains:** tpmjs.com, www.tpmjs.com - -### Repository Structure -``` -tpmjs/ -├── apps/ -│ └── web/ # Next.js 16 App Router application -│ ├── src/ -│ │ └── app/ -│ │ ├── api/ -│ │ │ ├── health/route.ts -│ │ │ ├── stats/route.ts -│ │ │ ├── tools/ -│ │ │ │ ├── route.ts -│ │ │ │ ├── [id]/route.ts -│ │ │ │ ├── [slug]/route.ts -│ │ │ │ └── validate/route.ts -│ │ │ └── sync/ -│ │ │ ├── changes/route.ts -│ │ │ ├── keyword/route.ts -│ │ │ └── metrics/route.ts -│ │ ├── page.tsx -│ │ └── tool/[slug]/page.tsx -│ ├── next.config.ts -│ └── vercel.json -├── packages/ -│ ├── db/ # Prisma client -│ ├── types/ # Shared TypeScript types -│ ├── utils/ # Utility functions -│ ├── env/ # Environment validation -│ └── ui/ # React component library -├── vercel.json # Root Vercel configuration -└── turbo.json -``` - -## API Route Examples - -### `/apps/web/src/app/api/health/route.ts` -```typescript -import { NextResponse } from 'next/server'; - -export const runtime = 'nodejs'; -export const dynamic = 'force-dynamic'; -export const maxDuration = 60; - -/** - * GET /api/health - * Simple health check endpoint that doesn't touch the database - */ -export async function GET() { - return NextResponse.json({ - status: 'ok', - timestamp: new Date().toISOString(), - env: { - hasDatabase: !!process.env.DATABASE_URL, - nodeEnv: process.env.NODE_ENV, - }, - }); -} -``` - -### `/apps/web/src/app/api/tools/route.ts` -```typescript -import { NextResponse } from 'next/server'; -import { prisma } from '@tpmjs/db/client'; - -export const runtime = 'nodejs'; -export const dynamic = 'force-dynamic'; - -export async function GET(request: Request) { - // ... query string parsing - - const [tools, totalCount] = await Promise.all([ - prisma.tool.findMany({ - where, - orderBy: [ - { qualityScore: 'desc' }, - { npmDownloadsLastMonth: 'desc' }, - { createdAt: 'desc' }, - ], - take: limit, - skip: offset, - }), - prisma.tool.count({ where }), - ]); - - return NextResponse.json({ - data: tools, - pagination: { - page, - limit, - total: totalCount, - totalPages: Math.ceil(totalCount / limit), - }, - }); -} -``` - -## Configuration Files - -### `/apps/web/next.config.ts` (Current) -```typescript -import type { NextConfig } from 'next'; - -const nextConfig: NextConfig = { - transpilePackages: ['@tpmjs/ui', '@tpmjs/utils', '@tpmjs/db', '@tpmjs/types', '@tpmjs/env'], - reactStrictMode: true, -}; - -export default nextConfig; -``` - -### `/apps/web/vercel.json` (Current) -```json -{ - "$schema": "https://openapi.vercel.sh/vercel.json", - "buildCommand": "cd ../.. && pnpm --filter=@tpmjs/web build", - "installCommand": "pnpm install" -} -``` - -### `/vercel.json` (Root) -```json -{ - "$schema": "https://openapi.vercel.sh/vercel.json", - "git": { - "deploymentEnabled": { - "main": true - } - }, - "github": { - "silent": false, - "autoJobCancelation": true - }, - "crons": [ - { - "path": "/api/sync/changes", - "schedule": "*/2 * * * *" - }, - { - "path": "/api/sync/keyword", - "schedule": "*/15 * * * *" - }, - { - "path": "/api/sync/metrics", - "schedule": "0 * * * *" - } - ] -} -``` - -## Local Build Verification - -### Local Build Output Structure -```bash -$ ls -R /Users/ajaxdavis/repos/tpmjs/tpmjs/apps/web/.next/server/app/api/ - -health/ -stats/ -sync/ -tools/ - -/apps/web/.next/server/app/api/health: -route -route.js -route.js.map -route.js.nft.json -route_client-reference-manifest.js - -/apps/web/.next/server/app/api/stats: -route -route.js -route.js.map -route.js.nft.json -route_client-reference-manifest.js - -/apps/web/.next/server/app/api/tools: -[id]/ -[slug]/ -validate/ -route -route.js -route.js.map -route.js.nft.json -route_client-reference-manifest.js -``` - -### Routes Manifest Confirmation -```bash -$ cat /apps/web/.next/routes-manifest.json | jq '.staticRoutes[] | select(.page | contains("api"))' - -{ - "page": "/api/health", - "regex": "^/api/health(?:/)?$", - "routeKeys": {}, - "namedRegex": "^/api/health(?:/)?$" -} -{ - "page": "/api/stats", - "regex": "^/api/stats(?:/)?$", - "routeKeys": {}, - "namedRegex": "^/api/stats(?:/)?$" -} -{ - "page": "/api/sync/changes", - "regex": "^/api/sync/changes(?:/)?$", - "routeKeys": {}, - "namedRegex": "^/api/sync/changes(?:/)?$" -} -{ - "page": "/api/sync/keyword", - "regex": "^/api/sync/keyword(?:/)?$", - "routeKeys": {}, - "namedRegex": "^/api/sync/keyword(?:/)?$" -} -{ - "page": "/api/sync/metrics", - "regex": "^/api/sync/metrics(?:/)?$", - "routeKeys": {}, - "namedRegex": "^/api/sync/metrics(?:/)?$" -} -``` - -### Node File Trace (NFT) Verification -```bash -$ cat /apps/web/.next/server/app/api/health/route.js.nft.json - -{ - "version": 1, - "files": [ - "../../../../../../../node_modules/.pnpm/next@16.0.4_@babel+core@7.28.5_react-dom@19.2.0_react@19.2.0__react@19.2.0/node_modules/next/dist/client/components/app-router-headers.js", - "../../../../../../../node_modules/.pnpm/next@16.0.4_@babel+core@7.28.5_react-dom@19.2.0_react@19.2.0__react@19.2.0/node_modules/next/dist/compiled/@opentelemetry/api/index.js", - // ... many more dependencies - ] -} -``` - -**Conclusion:** API routes build correctly locally with all dependencies properly traced. - -## Vercel Deployment Analysis - -### Deployment Inspection Output -```bash -$ vercel inspect https://tpmjs-1chh44d1u-tpmjs.vercel.app - -General - id dpl_2FHyiTWtBZzvT8EcohwYEZmdb8rb - name tpmjs-web - target production - status ● Ready - url https://tpmjs-1chh44d1u-tpmjs.vercel.app - created Fri Nov 28 2025 20:44:03 GMT+1000 - -Aliases - ╶ https://www.tpmjs.com - ╶ https://tpmjs-web.vercel.app - ╶ https://tpmjs-web-tpmjs.vercel.app - ╶ https://tpmjs-web-git-main-tpmjs.vercel.app - ╶ https://tpmjs.com - -Builds - ┌ . [0ms] - ├── λ tool/[slug] (562.92KB) [iad1] - ├── λ tool/[slug].rsc (562.92KB) [iad1] - ├── λ _global-error (642.48KB) [iad1] - ├── λ _global-error.rsc (642.48KB) [iad1] - ├── λ _global-error.segments/__PAGE__.segment.rsc (642.48KB) [iad1] - └── 56 output items hidden -``` - -**CRITICAL FINDING:** No API routes are listed in the build output. Only pages (`tool/[slug]`, `_global-error`, etc.) appear as serverless functions (`λ`). - -Expected API routes that should appear: -- `λ api/health` -- `λ api/tools` -- `λ api/tools/[id]` -- `λ api/tools/[slug]` -- `λ api/sync/changes` -- etc. - -### Testing Results -```bash -# Direct Vercel URL - Timeouts -$ curl -s -m 10 https://tpmjs-1chh44d1u-tpmjs.vercel.app/api/health -# Exit code 28 (timeout) - -# Custom Domain - Returns "Redirecting..." -$ curl -s -m 10 https://tpmjs.com/api/health -Redirecting... - -# Custom Domain - Returns "Redirecting..." -$ curl -s -m 10 https://tpmjs.com/api/tools -Redirecting... - -# Check redirect headers -$ curl -I https://tpmjs.com -HTTP/2 307 -cache-control: public, max-age=0, must-revalidate -content-type: text/plain -date: Fri, 28 Nov 2025 10:38:12 GMT -location: https://www.tpmjs.com/ -server: Vercel -``` - -## Vercel Configuration Details - -### User-Confirmed Settings -- **Root Directory:** `apps/web` (set in Vercel dashboard) -- **DATABASE_URL:** Configured in Vercel environment variables (Production) -- **Framework Preset:** (Unknown - needs verification) -- **Build Output Directory:** (Unknown - using default `.next`) - -### Project List -```bash -$ vercel project ls | grep -i tpmjs - -tpmjs -- 16h 24.x -v0-tool-registry-page https://tpmjs.com 3d 22.x -``` - -**NOTE:** Two projects exist: -1. `tpmjs` - Current project (Node 24.x) -2. `v0-tool-registry-page` - Also has tpmjs.com domain (Node 22.x) - -This could indicate a domain routing conflict or outdated project. - -## Investigation Timeline & Attempts - -### Attempt 1: Remove Root-Level Redirects -**Hypothesis:** The redirect in `/vercel.json` was intercepting API requests. - -**Original `/vercel.json`:** -```json -{ - "redirects": [ - { - "source": "/:path*", - "has": [ - { - "type": "host", - "value": "www.tpmjs.com" - } - ], - "destination": "https://tpmjs.com/:path*", - "permanent": true - } - ] -} -``` - -**Action:** Removed the `redirects` array from root `/vercel.json`. - -**Result:** ❌ API routes still timeout. Redirect rule was not the root cause. - -**Commit:** `9be3af7 fix(routing): move www redirect from vercel.json to Next.js config` - -### Attempt 2: Move Redirects to Next.js Config -**Hypothesis:** Next.js should handle redirects after routing. - -**Action:** Added `async redirects()` to `apps/web/next.config.ts`: -```typescript -async redirects() { - return [ - { - source: '/:path*', - has: [ - { - type: 'host', - value: 'www.tpmjs.com', - }, - ], - destination: 'https://tpmjs.com/:path*', - permanent: true, - }, - ]; -} -``` - -**Result:** ❌ API routes returned "Redirecting..." instead of executing. Next.js `async redirects()` applies to ALL routes including API routes. - -**Commit:** `9be3af7 fix(routing): move www redirect from vercel.json to Next.js config` - -### Attempt 3: Exclude API Routes from Redirect -**Hypothesis:** Use regex to exclude `/api/*` from redirects. - -**Action:** Modified redirect pattern: -```typescript -async redirects() { - return [ - { - source: '/((?!api).*)', // Negative lookahead to exclude /api/* - has: [ - { - type: 'host', - value: 'www.tpmjs.com', - }, - ], - destination: 'https://tpmjs.com/$1', - permanent: true, - }, - ]; -} -``` - -**Result:** ❌ API routes back to timing out (not redirecting anymore, but still not working). - -**Commit:** `c42bfb6 fix(redirects): exclude API routes from www redirect` - -### Attempt 4: Remove All Redirects -**Hypothesis:** Eliminate redirect loop causing ERR_TOO_MANY_REDIRECTS. - -**Action:** Removed `async redirects()` entirely from `next.config.ts`. - -**Result:** ✅ Redirect loop fixed. ❌ API routes still timeout. - -**Commit:** `b93cd42 fix: remove redirects to resolve redirect loop` - -### Attempt 5: Add Vercel Functions Configuration -**Hypothesis:** Vercel needs explicit configuration to detect API routes. - -**Action:** Added to `apps/web/vercel.json`: -```json -{ - "functions": { - "app/api/**/*.ts": { - "maxDuration": 60 - } - } -} -``` - -**Result:** ❌ No change. API routes still timeout. - -**Commit:** `748ca4d fix(api): configure Vercel functions for API routes with maxDuration` - -### Attempt 6: Add maxDuration to Route Files -**Hypothesis:** Export configuration directly in route handlers. - -**Action:** Added to `apps/web/src/app/api/health/route.ts`: -```typescript -export const maxDuration = 60; -``` - -**Result:** ❌ No change. API routes still timeout. - -**Commit:** `8281f8f fix(build): disable Turbopack for Vercel deployment` - -### Attempt 7: Disable Turbopack -**Hypothesis:** Turbopack (Next.js 16 default) has compatibility issues with Vercel. - -**Action:** Added `--webpack` flag to build command: -```json -{ - "buildCommand": "cd ../.. && turbo build --filter=@tpmjs/web -- --webpack" -} -``` - -**Result:** ❌ Build failed completely. Invalid flag syntax. - -**Commit:** `8281f8f fix(build): disable Turbopack for Vercel deployment` - -### Attempt 8: Simplify Build Command -**Hypothesis:** Use direct pnpm build instead of Turbo wrapper. - -**Action:** Changed to: -```json -{ - "buildCommand": "cd ../.. && pnpm --filter=@tpmjs/web build" -} -``` - -**Result:** ⏳ Pending deployment test. - -**Commit:** `065196d fix(build): simplify Vercel build command` - -## Root Cause Analysis - -### What We Know FOR SURE - -1. ✅ **API routes build correctly locally** - - All 9 API routes compile to `.next/server/app/api/` - - NFT (Node File Trace) files are generated with proper dependencies - - Routes manifest includes all API routes - -2. ✅ **Next.js configuration is correct** - - `export const runtime = 'nodejs'` set correctly - - `export const dynamic = 'force-dynamic'` set correctly - - `transpilePackages` includes all workspace packages - -3. ✅ **Pages deploy and work perfectly** - - `/tool/[slug]` renders correctly - - Homepage loads - - Client-side navigation works - -4. ❌ **API routes are NOT deployed as serverless functions** - - `vercel inspect` shows NO API routes in build output - - Only pages appear as `λ` (lambda) functions - - This is the PRIMARY issue - -5. ❌ **Direct Vercel URLs timeout** - - Not just a custom domain issue - - Affects `*.vercel.app` URLs - - Exit code 28 (timeout) - no response at all - -6. ❌ **Custom domain shows "Redirecting..."** - - Even with all redirects removed from config - - Suggests a redirect at Vercel platform level OR DNS level - - Could be from the `v0-tool-registry-page` project conflict - -### Possible Root Causes - -#### Theory 1: Vercel Project Misconfiguration -**Likelihood:** HIGH - -**Evidence:** -- Two projects with same domain (`tpmjs` and `v0-tool-registry-page`) -- Framework Preset might not be set to "Next.js" -- Root Directory is `apps/web` but Vercel might not be detecting Next.js properly - -**What to Check:** -1. Vercel Dashboard → Project Settings → General - - Framework Preset: Should be "Next.js" - - Root Directory: Should be "apps/web" - - Build Command: Should match vercel.json - - Output Directory: Should be blank (default `.next`) - -2. Vercel Dashboard → Domains - - Check if both projects have tpmjs.com - - Remove domain from `v0-tool-registry-page` if present - -3. Vercel Dashboard → Deployments → Build Logs - - Search for "API" or "route" - - Look for errors about missing functions - - Check if Next.js is detected correctly - -#### Theory 2: Monorepo Detection Issue -**Likelihood:** MEDIUM - -**Evidence:** -- Build command uses `cd ../.. && pnpm --filter=@tpmjs/web build` -- Vercel might not be correctly detecting workspace structure -- `transpilePackages` includes workspace packages - -**What to Check:** -1. Build logs for workspace resolution errors -2. Check if `node_modules` is being created in correct location -3. Verify pnpm workspace configuration - -**Potential Fix:** -Try setting `installCommand` to: -```json -{ - "installCommand": "pnpm install --shamefully-hoist" -} -``` - -#### Theory 3: Next.js 16 + Vercel Incompatibility -**Likelihood:** MEDIUM - -**Evidence:** -- Next.js 16 released recently (November 2024) -- Turbopack is default (might have Vercel issues) -- App Router API routes behave differently than Pages Router - -**What to Check:** -1. Vercel build logs for Next.js version detection -2. Any warnings about incompatible features -3. Check Vercel's Next.js 16 support status - -**Potential Fix:** -Downgrade to Next.js 15.x temporarily to test: -```json -{ - "dependencies": { - "next": "^15.0.0" - } -} -``` - -#### Theory 4: Environment Variable Issue -**Likelihood:** LOW - -**Evidence:** -- DATABASE_URL is configured -- Pages work (they might not need env vars) -- API routes use Prisma (requires DATABASE_URL) - -**What to Check:** -1. Vercel Dashboard → Settings → Environment Variables - - Verify DATABASE_URL is set for Production - - Verify it's not blocked or empty - - Check if other vars are needed - -2. Build logs for Prisma generation errors - -**Potential Fix:** -None - user confirmed DATABASE_URL is set. - -#### Theory 5: Build Output Issue -**Likelihood:** MEDIUM-HIGH - -**Evidence:** -- `vercel inspect` doesn't show API routes -- Only pages are listed as functions -- Build completes successfully (38-41 seconds) - -**What to Check:** -1. Build logs: Does Next.js report building API routes? - - Look for "Route (app)" or "λ" indicators for API routes - - Compare to local build output - -2. Check if Vercel is using correct build output structure - - App Router uses `.next/server/app/` - - Pages Router uses `.next/server/pages/` - -**Potential Fix:** -Try forcing Vercel to recognize the build: -```json -{ - "builds": [ - { - "src": "package.json", - "use": "@vercel/next" - } - ] -} -``` - -(Note: `builds` is legacy, modern Next.js should auto-detect) - -## Recommended Next Steps - -### Immediate Actions (High Priority) - -1. **Check Vercel Project Settings** - - Go to Vercel Dashboard → tpmjs-web project - - Verify Framework Preset is "Next.js" - - Verify Root Directory is `apps/web` - - Screenshot settings for reference - -2. **Review Build Logs** - - Go to latest deployment - - Download complete build logs - - Search for: - - "Route (app)" - should show API routes - - "λ" - should show API functions - - "api" - any mentions - - Errors or warnings - -3. **Check Domain Configuration** - - Verify only ONE project has tpmjs.com domain - - Remove domain from `v0-tool-registry-page` project if present - - Check DNS settings aren't redirecting - -4. **Test Simple API Route** - - Create minimal API route: - ```typescript - // apps/web/src/app/api/test/route.ts - export async function GET() { - return new Response('Hello from API', { status: 200 }); - } - ``` - - Deploy and test - - If this doesn't work, confirms platform issue - -### Investigation Actions (Medium Priority) - -5. **Compare Working vs Non-Working** - - Find a deployment where pages DO work - - Compare build output between page routes and API routes - - Look for differences in how they're compiled - -6. **Test Vercel CLI Deploy** - - Deploy directly via CLI: `vercel --prod` - - Check if behavior differs from Git-based deploy - - Might reveal configuration issues - -7. **Check Vercel Function Logs** - - Even though functions aren't in build output, try: - - `vercel logs --since 1h` - - Look for any API route invocations or errors - -8. **Review Turbo Configuration** - ```bash - # Check turbo.json for Next.js build config - cat /turbo.json - - # Verify build runs correctly locally - pnpm --filter=@tpmjs/web build - ``` - -### Alternative Approaches (If Above Fails) - -9. **Create New Vercel Project** - - Import from Git fresh - - Use identical settings - - Test if fresh project works - -10. **Contact Vercel Support** - - This may be a platform bug with Next.js 16 - - Provide this document as context - - Ask specifically why API routes aren't in build output - -11. **Temporary Workaround** - - Deploy API routes separately (different service) - - Use Vercel proxy to route `/api/*` to separate deployment - - Not ideal but unblocks development - -## Environment Variables Needed - -```bash -# Required for API routes -DATABASE_URL="postgresql://..." - -# Optional (check if needed) -NODE_ENV="production" -NEXT_PUBLIC_* # Any public env vars -``` - -## Build Commands Reference - -### Local Development -```bash -# Install dependencies -pnpm install - -# Generate Prisma client -pnpm --filter=@tpmjs/db db:generate - -# Run development server -pnpm --filter=@tpmjs/web dev - -# Build for production -pnpm --filter=@tpmjs/web build - -# Test build locally -pnpm --filter=@tpmjs/web start -``` - -### Vercel Configuration -**Current:** -```json -{ - "buildCommand": "cd ../.. && pnpm --filter=@tpmjs/web build", - "installCommand": "pnpm install" -} -``` - -**Alternative to try:** -```json -{ - "buildCommand": "cd ../.. && turbo build --filter=@tpmjs/web", - "installCommand": "pnpm install", - "framework": "nextjs" -} -``` - -## Key Files to Review - -1. `/apps/web/next.config.ts` - Next.js configuration -2. `/apps/web/vercel.json` - Vercel app-level config -3. `/vercel.json` - Vercel root config -4. `/turbo.json` - Turborepo configuration -5. `/apps/web/.next/routes-manifest.json` - Route definitions -6. `/apps/web/.next/build-manifest.json` - Build output -7. Vercel build logs (from dashboard) - -## Questions for Vercel Support - -If escalating to Vercel support, ask: - -1. Why are API routes not appearing in the build output (`vercel inspect`) when pages are deploying correctly? - -2. Is there a known issue with Next.js 16 App Router API routes in Turborepo monorepos? - -3. What's the correct way to configure `vercel.json` for a Next.js 16 app in a monorepo with custom build commands? - -4. Could having two projects (`tpmjs` and `v0-tool-registry-page`) with the same domain cause routing issues? - -5. Are there any specific requirements for deploying Next.js 16 API routes that differ from Next.js 15? - -## Related Documentation - -- [Next.js 16 Upgrade Guide](https://nextjs.org/docs/app/guides/upgrading/version-16) -- [Vercel Next.js Deployment](https://vercel.com/docs/frameworks/nextjs) -- [Vercel Functions Configuration](https://vercel.com/docs/functions/configuring-functions) -- [Turborepo with Vercel](https://vercel.com/docs/monorepos/turborepo) -- [Next.js App Router API Routes](https://nextjs.org/docs/app/api-reference/file-conventions/route) - -## Recent Commits Related to This Issue - -``` -065196d fix(build): simplify Vercel build command -8281f8f fix(build): disable Turbopack for Vercel deployment -748ca4d fix(api): configure Vercel functions for API routes with maxDuration -b93cd42 fix: remove redirects to resolve redirect loop -c42bfb6 fix(redirects): exclude API routes from www redirect -9be3af7 fix(routing): move www redirect from vercel.json to Next.js config -a92dfff fix(build): add workspace packages to Next.js transpilePackages -cc6c824 fix(vercel): configure Turborepo monorepo build for apps/web -``` - -## ✅ CONCLUSION - ROOT CAUSE IDENTIFIED - -### The Real Problem - -**Vercel is NOT detecting this project as a Next.js application.** - -When Vercel doesn't detect Next.js, it: -- Uses `@vercel/static-builder` instead of `@vercel/next` -- Treats the deployment as a static site -- Deploys pages (static HTML) successfully -- **Completely drops all App Router API routes** -- Never generates serverless functions for `/api/*` routes - -This explains EVERY symptom: -- ✅ Pages work (they're static files) -- ❌ API routes timeout (they were never deployed) -- ❌ No `λ api/*` in build output (functions don't exist) -- ❌ Direct Vercel URLs timeout (not a DNS issue) -- ❌ "Redirecting..." on custom domain (wrong project owns the domain) - -### Why Vercel Doesn't Detect Next.js - -**1. Wrong Root Directory** -- Vercel project likely has Root Directory set to `.` or empty -- Should be exactly: `apps/web` -- A single character difference breaks Next.js detection - -**2. Wrong Framework Preset** -- When Vercel can't auto-detect Next.js (due to monorepo + wrong root) -- It defaults to Framework Preset = "Other" -- "Other" uses static builder, not Next.js builder - -**3. Domain Conflict** -- Two projects exist: `tpmjs` and `v0-tool-registry-page` -- Both have `tpmjs.com` domain assigned -- Requests randomly route to wrong project -- "Redirecting..." comes from the old project, not your code - -**4. Custom Build Commands** -- Custom build commands can bypass Vercel's auto-detection -- Should let Vercel auto-detect and use default commands - -### Evidence - -**Local build works:** -```bash -$ ls apps/web/.next/server/app/api/ -health/ stats/ sync/ tools/ -# All API routes compile correctly -``` - -**Vercel deployment missing API routes:** -```bash -$ vercel inspect https://tpmjs-1chh44d1u-tpmjs.vercel.app -Builds - ├── λ tool/[slug] # Pages present - ├── λ _global-error # Pages present - └── 56 output items hidden -# NO api/health, NO api/tools, NO api/* at all -``` - -This is **not a Next.js bug, not a Turborepo bug, not a code issue**. -This is **pure Vercel project misconfiguration**. - -### The Fix (4 Steps, 5 Minutes) - -See `VERCEL_FIX_CHECKLIST.md` for detailed instructions. - -**Quick version:** -1. Set Root Directory to `apps/web` (exactly) -2. Set Framework Preset to "Next.js" -3. Remove `tpmjs.com` domain from `v0-tool-registry-page` project -4. Clear custom build commands (let Vercel auto-detect) - -**No code changes needed. Just fix Vercel dashboard settings.** - -### After the Fix - -Next deployment will show: -```bash -$ vercel inspect -Builds - ├── λ api/health - ├── λ api/tools - ├── λ api/sync/changes - ├── λ api/tools/[id] - ├── λ api/tools/[slug] - ├── λ tool/[slug] - ... -``` - -And API routes will work instantly: -```bash -$ curl https://tpmjs.com/api/health -{"status":"ok","timestamp":"..."} -``` - ---- - -## Summary for ChatGPT - -The core issue is that **Vercel is not detecting the project as Next.js** due to incorrect Root Directory, Framework Preset set to "Other", domain conflicts between two projects, and custom build commands bypassing auto-detection. - -When Vercel doesn't detect Next.js, it uses a static site builder which deploys pages but **completely drops all App Router API routes** from the build. This is why pages work but all `/api/*` requests timeout - the API route serverless functions were never created. - -**The fix is purely configuration** - no code changes needed. Set Root Directory to `apps/web`, Framework Preset to "Next.js", remove the domain from the old project, and clear custom build commands. See `VERCEL_FIX_CHECKLIST.md` for step-by-step instructions. diff --git a/DENO_NODE_PACKAGE_ISSUE.md b/DENO_NODE_PACKAGE_ISSUE.md deleted file mode 100644 index f8fbc5f..0000000 --- a/DENO_NODE_PACKAGE_ISSUE.md +++ /dev/null @@ -1,306 +0,0 @@ -# Running `ai-sdk-tool-code-execution` in Deno - Compatibility Issue - -## Problem Summary - -We need to run the npm package `ai-sdk-tool-code-execution` in a Deno runtime environment on Railway. The package requires Node.js built-ins (`node:sqlite`, `undici`) that don't exist in Deno, and we're looking for a solution to make it work. - -## Environment - -- **Runtime:** Deno 1.39.0 on Railway -- **Package:** `ai-sdk-tool-code-execution@0.0.2` -- **Import Method:** Dynamic imports via esm.sh CDN -- **Use Case:** Remote code execution for AI SDK tools - -## What We're Trying to Do - -We have a Deno server that dynamically imports npm packages at runtime to provide AI SDK tools. The workflow is: - -1. User requests a tool (e.g., `executeCode`) -2. Deno server fetches the package from esm.sh or npm -3. Server loads the tool's schema and execution function -4. Server executes the tool with user-provided parameters - -## The Package We Need - -**Package:** `ai-sdk-tool-code-execution` -**Version:** `0.0.2` -**Description:** Execute Python code in a sandboxed environment using Vercel Sandbox -**npm URL:** https://www.npmjs.com/package/ai-sdk-tool-code-execution -**CDN URLs:** -- esm.sh: `https://esm.sh/ai-sdk-tool-code-execution@0.0.2` -- jsdelivr: `https://cdn.jsdelivr.net/npm/ai-sdk-tool-code-execution@0.0.2/+esm` - -**Dependencies (from package.json):** -```json -{ - "dependencies": { - "ai": "^4.0.18", - "better-sqlite3": "^11.8.1", - "undici": "^7.16.0" - } -} -``` - -**Key Issue:** The package depends on: -- `better-sqlite3` → which requires `node:sqlite` (Node.js built-in) -- `undici` → HTTP client that uses Node.js internals - -## What We've Tried - -### Attempt 1: Deno npm: Specifier (Node.js Compatibility Mode) - -**Code:** -```typescript -const npmUrl = `npm:ai-sdk-tool-code-execution@0.0.2`; -const module = await import(npmUrl); -``` - -**Error:** -``` -Loading unprepared module: npm:ai-sdk-tool-code-execution@0.0.2 -``` - -**Why it failed:** Deno's npm compatibility requires the package to be "prepared" (downloaded/cached) before import. Dynamic imports of unprepared npm packages fail. - -### Attempt 2: esm.sh with Node.js Target - -**Code:** -```typescript -const esmUrl = `https://esm.sh/ai-sdk-tool-code-execution@0.0.2?target=esnext`; -const module = await import(esmUrl); -``` - -**Error:** -``` -Module not found "https://esm.sh/node:sqlite?target=esnext" - at https://esm.sh/undici@^7.16.0?target=esnext:25:8 -``` - -**Why it failed:** The package code imports `node:sqlite` which esm.sh tries to load from `https://esm.sh/node:sqlite?target=esnext`, but `node:sqlite` is a Node.js built-in, not an npm package. - -### Attempt 3: Multi-Strategy with Fallback - -**Code:** -```typescript -let module; -let importError; - -// Strategy 1: npm: specifier -try { - const npmUrl = `npm:${packageName}@${version}`; - module = await import(npmUrl); -} catch (error) { - importError = error; - - // Strategy 2: esm.sh with esnext target - try { - const esmUrl = `https://esm.sh/${packageName}@${version}?target=esnext`; - module = await import(esmUrl); - } catch (esmError) { - return { success: false, error: esmError.message }; - } -} -``` - -**Result:** Both strategies fail with the same errors as above. - -## Current Deno Configuration - -**`deno.json`:** -```json -{ - "compilerOptions": { - "allowJs": true, - "lib": ["deno.window"], - "strict": true - }, - "nodeModulesDir": true, - "unstable": ["byonm"], - "imports": { - "zod-to-json-schema": "https://esm.sh/zod-to-json-schema@3.25.0" - } -} -``` - -**Key Settings:** -- `nodeModulesDir: true` - Creates `node_modules` directory for npm packages -- `unstable: ["byonm"]` - Enables "Bring Your Own Node Modules" mode - -## Full Error Details - -### npm: Strategy Error -```json -{ - "success": false, - "error": "Failed to import package: ...", - "details": { - "npmError": "Loading unprepared module: npm:ai-sdk-tool-code-execution@0.0.2, imported from: file:///app/server.ts" - } -} -``` - -### esm.sh Strategy Error -```json -{ - "success": false, - "error": "Failed to import package: Module not found \"https://esm.sh/node:sqlite?target=esnext\"", - "details": { - "esmError": "Module not found \"https://esm.sh/node:sqlite?target=esnext\".\n at https://esm.sh/undici@^7.16.0?target=esnext:25:8" - } -} -``` - -## Technical Deep Dive - -### Why This Package Needs Node.js - -1. **better-sqlite3** - Native Node.js addon for SQLite - - Uses `node:sqlite` built-in - - Compiled C++ bindings - - Not available in Deno without Node compatibility layer - -2. **undici** - Modern HTTP client for Node.js - - Uses Node.js streams and buffer APIs - - Optimized for Node.js internals - - May work in Deno with polyfills, but blocked by sqlite dependency - -### Deno's Node.js Compatibility - -Deno supports many Node.js built-ins via `node:*` imports: -- `node:fs`, `node:path`, `node:http`, `node:crypto`, etc. - -**BUT** it does NOT support: -- `node:sqlite` (not a standard Node.js built-in) -- Native addons (`.node` files) -- Some advanced internal APIs - -### The Import Flow - -1. **Deno tries to import** `npm:ai-sdk-tool-code-execution@0.0.2` -2. **Package resolves to** esm.sh or npm registry -3. **Package imports** `better-sqlite3` -4. **better-sqlite3 imports** `node:sqlite` -5. **FAILURE:** `node:sqlite` doesn't exist in Deno or esm.sh - -## Questions for ChatGPT - -1. **Can Deno's npm compatibility layer handle `better-sqlite3` or `node:sqlite`?** - - Is there a Deno-compatible SQLite library we could alias? - - Can we use import maps to redirect `node:sqlite` to a Deno polyfill? - -2. **Can we "prepare" the npm module in Deno before dynamic import?** - - Is there a way to pre-cache npm packages in Deno? - - Can we use `deno vendor` or similar to prepare the package? - -3. **Can esm.sh or other CDNs provide Node.js built-in polyfills?** - - Does esm.sh have a mode that bundles Node.js built-ins? - - Are there CDN parameters we're missing? - -4. **Could we use Deno's `--node-modules-dir` flag differently?** - - Should we install the package via npm/pnpm first? - - Can we point Deno to pre-installed node_modules? - -5. **Is there a way to patch/bundle the package to remove Node.js dependencies?** - - Could we create a Deno-compatible fork? - - Are there tools to transpile Node.js packages to Deno? - -6. **Alternative: Different code execution package?** - - Are there Deno-native code execution tools? - - Could we use WebAssembly or browser-based sandboxing? - -## What Would Success Look Like - -**Ideal outcome:** -```typescript -// This should work in Deno: -const module = await import('npm:ai-sdk-tool-code-execution@0.0.2'); -const { executeCode } = module; - -// And this should execute: -const result = await executeCode.execute({ - code: 'print(fibonacci(10))', - language: 'python' -}); -``` - -**Acceptable outcome:** -```typescript -// Some preparation step, then: -const module = await import('https://esm.sh/ai-sdk-tool-code-execution@0.0.2'); -// Works without errors -``` - -## Repository Context - -**Project:** TPMJS - Tool Package Manager for AI SDK -**Server:** `apps/railway-executor/server.ts` -**Config:** `apps/railway-executor/deno.json` -**Deployment:** Railway with Deno runtime - -**Server Code (Simplified):** -```typescript -async function loadAndDescribe(req: Request): Promise { - const { packageName, exportName, version, importUrl } = await req.json(); - - // Try npm: specifier first - try { - const npmUrl = `npm:${packageName}@${version}`; - const module = await import(npmUrl); - const tool = module[exportName]; - return Response.json({ success: true, tool }); - } catch (error) { - // Try esm.sh fallback - const esmUrl = `https://esm.sh/${packageName}@${version}?target=esnext`; - const module = await import(esmUrl); - const tool = module[exportName]; - return Response.json({ success: true, tool }); - } -} - -Deno.serve({ port: 3001 }, handler); -``` - -## Live Error Logs - -**Request:** -```bash -curl -X POST https://endearing-commitment-production.up.railway.app/load-and-describe \ - -H "Content-Type: application/json" \ - -d '{ - "packageName": "ai-sdk-tool-code-execution", - "exportName": "executeCode", - "version": "0.0.2", - "importUrl": "https://esm.sh/ai-sdk-tool-code-execution@0.0.2" - }' -``` - -**Response:** -```json -{ - "success": false, - "error": "Failed to import package: Module not found \"https://esm.sh/node:sqlite?target=esnext\"", - "details": { - "npmError": "Loading unprepared module: npm:ai-sdk-tool-code-execution@0.0.2", - "esmError": "Module not found \"https://esm.sh/node:sqlite?target=esnext\"" - } -} -``` - -## Additional Context - -- We successfully load other packages (e.g., `@tpmjs/hello`, `zod-to-json-schema`) -- Only packages with Node.js built-in dependencies fail -- Switching to Node.js would work, but we prefer Deno's security model -- This is for a production tool registry serving AI SDK tools to users - -## Related Resources - -- **Deno npm compatibility:** https://deno.com/manual/node/npm_specifiers -- **Deno Node built-ins:** https://deno.com/manual/node/node_specifiers -- **esm.sh documentation:** https://esm.sh/ -- **Package source:** https://www.npmjs.com/package/ai-sdk-tool-code-execution -- **Deno SQLite libraries:** https://deno.land/x/sqlite@v3.8 - ---- - -**Question for ChatGPT:** Is there any way to make `ai-sdk-tool-code-execution` work in Deno, given these constraints? If not, what's the closest alternative that would work in Deno's runtime? diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 914560e..dd4d51c 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -167,7 +167,7 @@ Not needed for Option 1 (Deployment Protection). Add to README.md to show CI status: ```markdown -[![CI](https://github.com/YOUR_ORG/YOUR_REPO/actions/workflows/ci.yml/badge.svg)](https://github.com/YOUR_ORG/YOUR_REPO/actions/workflows/ci.yml) +[![CI](https://github.com/tpmjs/tpmjs/actions/workflows/ci.yml/badge.svg)](https://github.com/tpmjs/tpmjs/actions/workflows/ci.yml) ``` ## Summary diff --git a/DYNAMIC_IMPORT_ISSUE.md b/DYNAMIC_IMPORT_ISSUE.md deleted file mode 100644 index 9523485..0000000 --- a/DYNAMIC_IMPORT_ISSUE.md +++ /dev/null @@ -1,864 +0,0 @@ -# Dynamic Import Issue: Cannot Import ESM Modules from CDN in Next.js Server-Side API Route - -## Executive Summary - -We're building a dynamic tool loading system where AI agents can discover and load tools at runtime from npm packages via esm.sh CDN. The system successfully searches and finds relevant tools, but fails when trying to dynamically import them using `import()` in a Next.js App Router API route. - -**Error**: `Error: Cannot find module 'unknown'` with code `MODULE_NOT_FOUND` - -**Critical Question**: How can we dynamically import ESM modules from external URLs (like esm.sh) in Next.js 16 App Router API routes running in Node.js runtime? - ---- - -## System Architecture - -### High-Level Flow - -``` -1. User sends message → "use firecrawl to search for ajax davis" -2. Chat API extracts query → "use firecrawl to search for ajax davis" -3. Pre-flight search → Calls searchTpmjsToolsTool.execute({ query, limit: 5 }) -4. Search API returns → Top 5 matching tools from database (BM25-like scoring) -5. Dynamic loading → Tries to import tools from esm.sh URLs ❌ FAILS HERE -6. Agent uses tools → Would pass loaded tools to AI model -``` - -### Tech Stack - -- **Framework**: Next.js 16.0.4 -- **Build Tool**: Turbopack (default in Next.js 15+) -- **Runtime**: Node.js (not edge) -- **Package Manager**: pnpm (monorepo with workspaces) -- **Deployment Target**: Vercel (eventually, currently local dev) -- **AI SDK**: Vercel AI SDK v6.0.0-beta.124 -- **Model**: OpenAI GPT-4o-mini via `streamText()` - -### Monorepo Structure - -``` -tpmjs/ -├── apps/ -│ ├── playground/ # Next.js app with chat interface -│ │ └── src/ -│ │ ├── app/api/chat/route.ts # Where dynamic import fails -│ │ └── lib/dynamic-tool-loader.ts -│ └── web/ # Tool registry website -│ └── src/app/api/tools/search/route.ts -└── packages/ - └── tools/ - ├── hello/ # Static tool (works fine) - └── search-registry/ # Meta-tool for searching registry -``` - ---- - -## Detailed Code Implementation - -### File 1: `apps/playground/src/lib/dynamic-tool-loader.ts` - -**Purpose**: Load tools dynamically from esm.sh CDN - -```typescript -// Cache for imported tool modules (process-level) -const moduleCache = new Map(); - -// Cache for per-conversation active tools -const conversationTools = new Map>(); - -/** - * Generate cache key for a tool - */ -function getCacheKey(packageName: string, exportName: string): string { - return `${packageName}::${exportName}`; -} - -/** - * Validate that an import is a valid AI SDK tool - */ -function isValidTool(value: any): boolean { - return ( - value && - typeof value === 'object' && - typeof value.description === 'string' && - typeof value.execute === 'function' - ); -} - -/** - * Dynamically import a tool from ESM CDN - * - * THIS IS WHERE IT FAILS ❌ - */ -export async function loadToolDynamically( - packageName: string, - exportName: string, - version: string, - importUrl?: string -): Promise { - const cacheKey = getCacheKey(packageName, exportName); - - // Check cache first - if (moduleCache.has(cacheKey)) { - console.log(`✅ Cache hit: ${cacheKey}`); - return moduleCache.get(cacheKey); - } - - // Build import URL - const url = importUrl || `https://esm.sh/${packageName}@${version}`; - - try { - console.log(`📦 Importing: ${url}`); - // Example: https://esm.sh/firecrawl-aisdk@0.7.2 - - // Dynamic import with @vite-ignore to bypass bundler - const module = await import(/* @vite-ignore */ url); - - console.log(`🔍 Module imported successfully`); - console.log(`🔍 Module type: ${typeof module}`); - console.log(`🔍 Module keys: ${Object.keys(module).join(', ')}`); - console.log(`🔍 Looking for export: "${exportName}"`); - console.log(`🔍 Export exists: ${exportName in module}`); - console.log(`🔍 Export type: ${typeof module[exportName]}`); - - // Get the specific export - const tool = module[exportName]; - - if (!tool) { - console.error(`❌ Export "${exportName}" not found in module. Available exports:`, Object.keys(module)); - return null; - } - - console.log(`🔍 Tool structure:`, { - hasDescription: 'description' in tool, - hasExecute: 'execute' in tool, - hasInputSchema: 'inputSchema' in tool, - keys: Object.keys(tool), - }); - - if (!isValidTool(tool)) { - console.error(`❌ Invalid tool structure: ${exportName} from ${packageName}`); - console.error(` Tool:`, tool); - return null; - } - - // Cache successful import - moduleCache.set(cacheKey, tool); - console.log(`✅ Loaded: ${cacheKey}`); - - return tool; - } catch (error) { - console.error(`❌ Failed to load ${packageName}#${exportName}:`, error); - console.error(` URL: ${url}`); - console.error(` Stack:`, error instanceof Error ? error.stack : 'No stack trace'); - return null; - } -} - -/** - * Load multiple tools in parallel - */ -export async function loadToolsBatch( - toolMetadata: Array<{ - packageName: string; - exportName: string; - version: string; - importUrl?: string; - }> -): Promise> { - const promises = toolMetadata.map((meta) => - loadToolDynamically( - meta.packageName, - meta.exportName, - meta.version, - meta.importUrl - ).then((tool) => ({ - key: getCacheKey(meta.packageName, meta.exportName), - tool, - })) - ); - - const results = await Promise.all(promises); - - const tools: Record = {}; - for (const { key, tool } of results) { - if (tool) { - tools[key] = tool; - } - } - - return tools; -} -``` - -### File 2: `apps/playground/src/app/api/chat/route.ts` - -**Purpose**: Main chat API that orchestrates tool discovery and loading - -```typescript -import { createOpenAI } from '@ai-sdk/openai'; -import { type UIMessage, convertToModelMessages, stepCountIs, streamText } from 'ai'; -import type { NextRequest } from 'next/server'; -import { NextResponse } from 'next/server'; -import { env } from '~/env'; -import { loadAllTools, sanitizeToolName } from '~/lib/tool-loader'; -import { searchTpmjsToolsTool } from '@tpmjs/search-registry'; -import { - loadToolsBatch, - addConversationTools, -} from '~/lib/dynamic-tool-loader'; - -export const runtime = 'nodejs'; // ⚠️ Important: We're using Node.js runtime, not edge -export const dynamic = 'force-dynamic'; -export const maxDuration = 60; - -// Initialize OpenAI provider -const openai = createOpenAI({ - apiKey: env.OPENAI_API_KEY, -}); - -// Add conversation state tracking (in-memory for MVP) -const conversationStates = new Map }>(); - -/** - * POST /api/chat - * Chat with AI agent that can execute TPMJS tools - */ -export async function POST(request: NextRequest) { - try { - const body = await request.json(); - console.log('📥 Request body:', JSON.stringify(body, null, 2)); - - const messages: UIMessage[] = body.messages || []; - const conversationId: string = body.conversationId || 'default'; - - console.log(`🔑 Conversation ID: ${conversationId}`); - - // Get or create conversation state - if (!conversationStates.has(conversationId)) { - console.log('✨ Creating new conversation state'); - conversationStates.set(conversationId, { loadedTools: {} }); - } - const state = conversationStates.get(conversationId)!; - console.log(`📊 Current loaded tools in conversation: ${Object.keys(state.loadedTools).length}`); - - // 1. Load static tools + search tool - const staticTools = await loadAllTools(); - console.log(`🔧 Loaded ${Object.keys(staticTools).length} static tools`); - - staticTools.searchTpmjsTools = searchTpmjsToolsTool; - console.log('✅ Added searchTpmjsTools to static tools'); - - // 2. Extract user query from last message for tool search - const lastMessage = messages[messages.length - 1]; - let userQuery = ''; - if (lastMessage?.role === 'user') { - const parts = (lastMessage as any).parts || []; - for (const part of parts) { - if (part.type === 'text') { - userQuery = part.text; - break; - } - } - } - - console.log(`💬 User query: "${userQuery}"`); - - // 3. Automatically search for relevant tools based on the user's message - if (userQuery && userQuery.trim().length > 0) { - console.log('🔎 Searching for relevant tools...'); - - try { - const searchResult = await searchTpmjsToolsTool.execute({ - query: userQuery, - limit: 5, // Get top 5 relevant tools - }, {} as any); - - console.log(`📦 Found ${searchResult.matchCount} matching tools`); - - if (searchResult.tools && searchResult.tools.length > 0) { - console.log(`🔧 Tools found:`, searchResult.tools.map((t: any) => `${t.packageName}/${t.exportName}`)); - - // Dynamically load tools from esm.sh - console.log(`📥 Loading ${searchResult.tools.length} tools dynamically...`); - - const toolsToLoad = searchResult.tools.map((meta: any) => ({ - packageName: meta.packageName, - exportName: meta.exportName, - version: meta.version, - importUrl: meta.importUrl, - })); - - try { - // ❌ THIS IS WHERE IT FAILS - const loadedTools = await loadToolsBatch(toolsToLoad); - console.log(`✅ Successfully loaded ${Object.keys(loadedTools).length} tools`); - - // Add sanitized tools to conversation state - for (const [key, tool] of Object.entries(loadedTools)) { - const [pkg, exp] = key.split('::'); - const sanitizedKey = sanitizeToolName(`${pkg}-${exp}`); - state.loadedTools[sanitizedKey] = tool; - console.log(`✅ Added to conversation: ${sanitizedKey}`); - } - - // Track for this conversation - addConversationTools(conversationId, Object.keys(state.loadedTools)); - } catch (error) { - console.error('❌ Error loading tools:', error); - } - } else { - console.log('ℹ️ No matching tools found for this query'); - } - } catch (error) { - console.error('❌ Error searching for tools:', error); - } - } - - // 4. Merge with conversation's dynamically loaded tools - const allTools: Record = { ...staticTools, ...state.loadedTools }; - - // 5. Build system prompt with available tools - const toolsList = Object.keys(allTools) - .map((name) => { - const tool = allTools[name] as { description?: string } | undefined; - return `- ${name}: ${tool?.description || 'No description'}`; - }) - .join('\n'); - - const system = `You are a helpful AI assistant that can use TPMJS tools to help users. - -Available tools: -${toolsList} - -When you use a tool, you MUST always follow up with a natural language answer to the user summarizing the result.`; - - // 6. Stream response with all available tools - const result = streamText({ - model: openai('gpt-4o-mini'), - system, - messages: convertToModelMessages(messages), - tools: allTools, - stopWhen: stepCountIs(5), - }); - - return result.toUIMessageStreamResponse(); - } catch (error) { - console.error('Chat API error:', error); - - return new Response( - JSON.stringify({ - success: false, - error: error instanceof Error ? error.message : 'Unknown error', - }), - { - status: 500, - headers: { 'Content-Type': 'application/json' }, - } - ); - } -} -``` - -### File 3: Example Tool Metadata (from search API) - -When we search for "firecrawl", the search API returns: - -```json -{ - "success": true, - "query": "firecrawl ajax davis", - "results": { - "total": 29, - "returned": 5, - "tools": [ - { - "id": "cm4abc123", - "exportName": "searchTool", - "description": "Search the web using Firecrawl's search API", - "qualityScore": 0.85, - "package": { - "npmPackageName": "firecrawl-aisdk", - "npmVersion": "0.7.2", - "category": "web-scraping", - "frameworks": ["vercel-ai"], - "env": "server" - }, - "importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2", - "cdnUrl": "https://cdn.jsdelivr.net/npm/firecrawl-aisdk@0.7.2/+esm" - } - ] - } -} -``` - -So we're trying to: -```typescript -const module = await import('https://esm.sh/firecrawl-aisdk@0.7.2'); -const tool = module.searchTool; // Get the exported tool -``` - ---- - -## The Error - -### Console Output - -``` -📥 Loading 5 tools dynamically... -📦 Importing: https://esm.sh/firecrawl-aisdk@0.7.2 -❌ Failed to load firecrawl-aisdk#searchTool: Error: Cannot find module 'unknown' - at (.next/dev/server/chunks/[root-of-the-server]__746deca2._.js:357:23) - at loadToolDynamically (.next/dev/server/chunks/[root-of-the-server]__746deca2._.js:360:11) - at (src/lib/dynamic-tool-loader.ts:108:5) - at Array.map () - at loadToolsBatch (src/lib/dynamic-tool-loader.ts:107:33) - at POST (src/app/api/chat/route.ts:104:53) - { - code: 'MODULE_NOT_FOUND' - } - URL: https://esm.sh/firecrawl-aisdk@0.7.2 - Stack: Error: Cannot find module 'unknown' - at /Users/ajaxdavis/repos/tpmjs/tpmjs/apps/playground/.next/dev/server/chunks/[root-of-the-server]__746deca2._.js:357:23 - at loadToolDynamically (/Users/ajaxdavis/repos/tpmjs/tpmjs/apps/playground/.next/dev/server/chunks/[root-of-the-server]__746deca2._.js:360:11) -``` - -### Key Observations - -1. **Error happens immediately** - Never gets to our debug logs after `await import()` -2. **Error is MODULE_NOT_FOUND** - Treating URL as a module path -3. **Error says "unknown"** - Not even using the actual module name -4. **Code is in .next/dev/server/chunks/** - Next.js/Turbopack transformed our code -5. **Same error for all packages** - firecrawl-aisdk, @exalabs/ai-sdk, etc. - ---- - -## Verification: The URL Works - -### Manual Test 1: Browser - -``` -Visit: https://esm.sh/firecrawl-aisdk@0.7.2 -``` - -Returns valid ESM module: -```javascript -/* esm.sh - firecrawl-aisdk@0.7.2 */ -import * as __1$ from "/v135/@ai-sdk/provider-utils@2.0.8/..."; -// ... rest of module code -export { searchTool, scrapeTool, crawlTool }; -``` - -### Manual Test 2: Plain Node.js Script - -Create `test-import.mjs`: -```javascript -const module = await import('https://esm.sh/firecrawl-aisdk@0.7.2'); -console.log('Module:', module); -console.log('Exports:', Object.keys(module)); -``` - -Run: `node test-import.mjs` - -**Expected**: Would work in plain Node.js with `--experimental-network-imports` flag -**In Next.js**: Can't even get this far - ---- - -## What We've Tried - -### Attempt 1: `/* @vite-ignore */` Comment -```typescript -const module = await import(/* @vite-ignore */ url); -``` -**Result**: Still fails with MODULE_NOT_FOUND - -### Attempt 2: `/* webpackIgnore: true */` Comment -```typescript -const module = await import(/* webpackIgnore: true */ url); -``` -**Result**: Still fails with MODULE_NOT_FOUND - -### Attempt 3: Force Dynamic Runtime -```typescript -export const runtime = 'nodejs'; -export const dynamic = 'force-dynamic'; -``` -**Result**: Still fails (we're already using this) - -### Attempt 4: Verify esm.sh Works -- Tested URLs in browser: ✅ Works -- All packages return valid ESM: ✅ Valid -- esm.sh is accessible: ✅ Reachable - -### Attempt 5: Check Static Imports -```typescript -import { helloWorldTool } from '@tpmjs/hello'; -``` -**Result**: Works perfectly (but bundled at build time) - ---- - -## Configuration Files - -### `apps/playground/next.config.ts` - -```typescript -import type { NextConfig } from 'next'; - -const config: NextConfig = { - reactStrictMode: true, - transpilePackages: ['@tpmjs/ui'], - experimental: { - turbo: { - // Using Turbopack (Next.js 15+ default) - }, - }, -}; - -export default config; -``` - -### `apps/playground/package.json` (relevant parts) - -```json -{ - "name": "@tpmjs/playground", - "version": "0.0.0", - "private": true, - "scripts": { - "dev": "next dev --port 3001", - "build": "next build", - "start": "next start" - }, - "dependencies": { - "@ai-sdk/openai": "^1.0.15", - "@tpmjs/hello": "workspace:*", - "@tpmjs/search-registry": "workspace:*", - "ai": "6.0.0-beta.124", - "next": "16.0.4", - "react": "19.0.0" - } -} -``` - -### `turbo.json` (monorepo config) - -```json -{ - "$schema": "https://turbo.build/schema.json", - "tasks": { - "dev": { - "cache": false, - "persistent": true - }, - "build": { - "dependsOn": ["^build"], - "outputs": [".next/**", "dist/**"] - } - } -} -``` - ---- - -## Why This Matters - -### The Bigger Picture - -We're building a **self-referential tool discovery system**: - -1. **Tool Registry** (tpmjs.com) - Indexes all TPMJS-compatible tools from npm -2. **Search Tool** - AI SDK tool that searches the registry -3. **Dynamic Loader** - Loads found tools at runtime -4. **AI Agent** - Uses dynamically loaded tools - -This creates infinite extensibility: -- No need to bundle all possible tools -- Tools can be published to npm independently -- System discovers and loads tools as needed -- Bundle size stays small - -### Use Case Example - -``` -User: "Search Wikipedia for quantum computing" - ↓ -System searches registry: Finds "wikipedia-aisdk" tool - ↓ -System loads tool: import('https://esm.sh/wikipedia-aisdk@1.0.0') - ↓ -AI uses tool: wikipediaSearchTool.execute({ query: "quantum computing" }) - ↓ -User gets answer with Wikipedia citations -``` - ---- - -## Possible Root Causes - -### Hypothesis 1: Turbopack Doesn't Support Dynamic Import URLs -- Turbopack intercepts all `import()` calls -- Transforms them to module resolution -- Doesn't handle external URLs - -### Hypothesis 2: Next.js Security Restriction -- Next.js blocks dynamic imports from external URLs for security -- Prevents arbitrary code execution -- No way to whitelist esm.sh - -### Hypothesis 3: Dev Mode Only Issue -- Turbopack dev mode has more restrictions -- Production webpack build might work -- But we need dev mode to work too - -### Hypothesis 4: Node.js Runtime Limitation in Next.js -- Next.js Node.js runtime is sandboxed -- Dynamic imports are intercepted before reaching Node.js -- Plain Node.js would work with --experimental-network-imports - ---- - -## Alternative Approaches We're Considering - -### Option A: Fetch + VM Module -```typescript -import { SourceTextModule } from 'vm'; - -const response = await fetch(url); -const code = await response.text(); -const module = new SourceTextModule(code); -await module.link(() => {}); -await module.evaluate(); -const exports = module.namespace; -``` - -**Pros**: Bypasses import() entirely -**Cons**: Complex, security concerns, might not work in Next.js - -### Option B: Separate Microservice -```typescript -// New service: tool-loader-service (Express or Fastify) -POST /load-tool -Body: { packageName, exportName, version } -Response: { tool: } -``` - -**Pros**: Full control, definitely works -**Cons**: Extra infrastructure, latency, complexity - -### Option C: Switch to Edge Runtime -```typescript -export const runtime = 'edge'; // Instead of 'nodejs' -``` - -**Pros**: Edge might have different import behavior -**Cons**: Edge has limitations (no Node.js APIs), might still not work - -### Option D: Pre-bundle Common Tools -```typescript -// Generate static imports for top 100 tools -import { tool1 } from 'package1'; -import { tool2 } from 'package2'; -// ... etc -``` - -**Pros**: Definitely works -**Cons**: Defeats the purpose, huge bundle size - -### Option E: Use unpkg or jsdelivr with Different Strategy -```typescript -// Fetch raw code, eval in isolated context -const response = await fetch(`https://unpkg.com/${pkg}@${ver}/dist/index.mjs`); -const code = await response.text(); -const exports = evalInContext(code); -``` - -**Pros**: More control -**Cons**: Same security/execution issues - ---- - -## Specific Questions for ChatGPT - -### Question 1: Is This Possible? -**Can Next.js 16 App Router API routes (Node.js runtime) dynamically import ESM modules from external URLs using `import()`?** - -If yes: -- What configuration is needed? -- Are there security allowlists? -- Does it work in both dev and production? - -If no: -- Why not? -- What's the recommended alternative? -- Is this a Turbopack limitation or Next.js design? - -### Question 2: Turbopack Behavior -**Does Turbopack intercept all `import()` calls, even with magic comments?** - -We've tried: -- `/* @vite-ignore */` -- `/* webpackIgnore: true */` - -None work. Is there a Turbopack-specific comment or config? - -### Question 3: Edge vs Node Runtime -**Would switching to edge runtime change import behavior?** - -```typescript -export const runtime = 'edge'; // vs 'nodejs' -``` - -Does edge runtime allow dynamic imports from URLs? - -### Question 4: Best Practice -**What's the recommended way to implement dynamic tool loading in Next.js?** - -Given constraints: -- Need to load arbitrary npm packages at runtime -- Packages are ESM modules from CDN -- Can't pre-bundle all possibilities -- Need to work in production on Vercel - -### Question 5: Security Model -**Is Next.js intentionally blocking this for security?** - -- Is there a whitelist for allowed CDNs? -- Can we configure allowed import sources? -- Is this related to CSP or other security headers? - ---- - -## Environment Details - -### Versions -```json -{ - "next": "16.0.4", - "react": "19.0.0", - "turbo": "2.6.1", - "pnpm": "9.15.0", - "node": "v20.11.0", - "ai": "6.0.0-beta.124" -} -``` - -### Operating System -- **OS**: macOS (Darwin 23.5.0) -- **Architecture**: arm64 (Apple Silicon) - -### Development Commands -```bash -# Start dev server -pnpm dev --filter=@tpmjs/playground - -# Output -▲ Next.js 16.0.4 (Turbopack) -- Local: http://localhost:3001 -- Network: http://192.168.0.25:3001 -✓ Ready in 2.5s -``` - -### Build Output Structure -``` -apps/playground/.next/ -├── dev/ -│ └── server/ -│ └── chunks/ -│ └── [root-of-the-server]__746deca2._.js # ← Error originates here -``` - ---- - -## Success Criteria - -### What We Need Working - -```typescript -// In Next.js API route (Node.js runtime) -const url = 'https://esm.sh/firecrawl-aisdk@0.7.2'; -const module = await import(url); -const tool = module.searchTool; - -console.log(tool.description); // "Search the web using Firecrawl's search API" -console.log(typeof tool.execute); // "function" - -// Tool is ready to use with AI SDK -const result = await tool.execute({ query: "test" }, context); -``` - -### Acceptable Outcomes - -1. ✅ **Best**: Dynamic `import()` works with configuration change -2. ✅ **Good**: Alternative approach that doesn't require microservice -3. ✅ **Acceptable**: Workaround that works in production even if dev is tricky -4. ❌ **Unacceptable**: "You can't do this in Next.js" without alternative - ---- - -## Additional Context - -### Why Not Just Bundle Everything? - -Currently have ~30 tools in registry, growing to 100s or 1000s: -- Bundle size would be massive (10+ MB) -- Most tools won't be used in most conversations -- Tools are published independently by community -- Want instant availability of new tools without redeploying - -### Why esm.sh Specifically? - -- ✅ Converts any npm package to ESM -- ✅ Handles dependencies automatically -- ✅ Fast CDN with caching -- ✅ No build step required -- ✅ Version pinning built-in - -But we're flexible - if jsdelivr, unpkg, or another approach works better, we'll use it. - -### Static Imports Work Fine - -This works perfectly (but defeats the purpose): -```typescript -import { searchTool } from 'firecrawl-aisdk'; -``` - -The tools themselves are fine. We just can't load them dynamically. - ---- - -## What We're Hoping For - -### Ideal Answer Format - -1. **Root cause**: Why it's failing -2. **Solution**: How to fix it (with code example) -3. **Configuration**: Any Next.js config needed -4. **Limitations**: What won't work / tradeoffs -5. **Alternatives**: If dynamic import truly impossible - -### We're Happy to Try - -- Different CDN (unpkg, jsdelivr, etc.) -- Different import strategy (fetch + eval, vm module, etc.) -- Different runtime (edge if it works) -- Different Next.js version (if specific version supports this) -- Webpack instead of Turbopack (if webpack handles this better) - -We just need a path forward that enables runtime tool loading in a production Next.js app on Vercel. - ---- - -## Files to Reference - -All code is in this monorepo: -- `apps/playground/src/lib/dynamic-tool-loader.ts` - Import logic -- `apps/playground/src/app/api/chat/route.ts` - API route -- `apps/playground/next.config.ts` - Next.js config -- `DYNAMIC_IMPORT_ISSUE.md` - This document - ---- - -## Thank You - -This is a critical blocker for our dynamic tool loading system. Any insights, workarounds, or alternative approaches would be immensely helpful! diff --git a/DYNAMIC_TOOL_LOADING_PRD.md b/DYNAMIC_TOOL_LOADING_PRD.md deleted file mode 100644 index f777951..0000000 --- a/DYNAMIC_TOOL_LOADING_PRD.md +++ /dev/null @@ -1,988 +0,0 @@ -# Dynamic Tool Loading System - Product Requirements Document - -## Executive Summary - -Build a self-referential tool discovery system where AI agents can search the TPMJS registry, find relevant tools, and dynamically import them during conversation. This creates a "meta-tool" that makes the entire TPMJS ecosystem available to any agent at runtime. - -**Core Innovation:** An AI agent can discover and load tools on-demand by searching the registry, rather than having all tools pre-loaded. This enables infinite tool extensibility without bundle size concerns. - ---- - -## Problem Statement - -### Current Limitations - -1. **Static Tool Loading**: Playground requires all tools to be hardcoded in `tool-loader.ts` -2. **Bundle Size**: Loading many tools increases bundle size and initialization time -3. **Discovery Gap**: Agents can't discover new tools that match their current task -4. **Manual Updates**: Adding tools requires code changes and redeployment - -### User Pain Points - -- Users want agents to access the full TPMJS registry without manual configuration -- Developers want to publish tools that are immediately available to all agents -- Agents need context-aware tool selection based on the conversation - ---- - -## Solution Overview - -### The Meta-Tool: `searchTpmjsTools` - -A TPMJS tool that searches the TPMJS registry and returns tool metadata needed for dynamic import. - -**Flow:** -``` -User: "Search Wikipedia for quantum computing" - ↓ -Agent: Calls searchTpmjsTools("wikipedia search") - ↓ -API: Returns tools matching "wikipedia" (BM25 search) - ↓ -Playground: Dynamically imports matching tools - ↓ -Agent: Now has Wikipedia tools available, uses them -``` - -### Key Components - -1. **`@tpmjs/search-registry`** - NPM package exporting `searchTpmjsToolsTool` -2. **`/api/tools/search`** - New API endpoint with BM25 full-text search -3. **Playground Dynamic Loader** - Runtime tool import system -4. **Tool Import Strategy** - ESM CDN imports or bundled approach - ---- - -## Technical Architecture - -### Component 1: Search Tool Package - -**Package:** `packages/tools/search-registry/` - -```typescript -// packages/tools/search-registry/src/index.ts -import { tool } from 'ai'; -import { z } from 'zod'; - -export const searchTpmjsToolsTool = tool({ - description: 'Search the TPMJS tool registry to find AI SDK tools. Use this when you need a tool that isn\'t currently available. Returns tool metadata including package names and descriptions.', - parameters: z.object({ - query: z.string().describe('Search query (e.g., "weather", "database", "wikipedia")'), - category: z.enum([ - 'text-analysis', - 'code-generation', - 'data-processing', - 'image-generation', - 'audio-processing', - 'search', - 'integration', - 'other' - ]).optional().describe('Filter by tool category'), - limit: z.number().min(1).max(20).default(10).describe('Max number of tools to return'), - }), - execute: async ({ query, category, limit }) => { - // Call TPMJS search API - const params = new URLSearchParams({ - q: query, - limit: String(limit), - ...(category && { category }), - }); - - const response = await fetch( - `https://tpmjs.com/api/tools/search?${params}` - ); - - if (!response.ok) { - throw new Error(`Search failed: ${response.statusText}`); - } - - const data = await response.json(); - - // Return structured tool metadata - return { - query, - matchCount: data.tools.length, - tools: data.tools.map((tool: any) => ({ - packageName: tool.package.npmPackageName, - exportName: tool.exportName, - description: tool.description, - category: tool.package.category, - qualityScore: tool.qualityScore, - frameworks: tool.package.frameworks, - env: tool.package.env, - })), - }; - }, -}); -``` - -**Package Metadata:** - -```json -{ - "name": "@tpmjs/search-registry", - "version": "0.1.0", - "description": "AI SDK tool for searching the TPMJS tool registry", - "keywords": ["tpmjs-tool", "ai", "search"], - "tpmjs": { - "category": "search", - "frameworks": ["vercel-ai"], - "tools": [ - { - "exportName": "searchTpmjsToolsTool", - "description": "Search the TPMJS tool registry to find AI SDK tools by keyword, category, or description. Returns tool metadata for dynamic loading.", - "parameters": [ - { - "name": "query", - "type": "string", - "description": "Search query (keywords, tool names, descriptions)", - "required": true - }, - { - "name": "category", - "type": "string", - "description": "Filter by category (text-analysis, search, etc.)", - "required": false - }, - { - "name": "limit", - "type": "number", - "description": "Maximum number of results (1-20, default 10)", - "required": false - } - ], - "returns": { - "type": "object", - "description": "Search results with tool metadata for dynamic import" - }, - "aiAgent": { - "useCase": "Use this tool when you need a tool that isn't currently available. For example, if asked to search Wikipedia but you don't have a Wikipedia tool, search for 'wikipedia' to find and load it.", - "examples": [ - "Search for 'weather' tools when asked about weather", - "Search for 'database' tools when working with data", - "Search for 'code' tools when generating code" - ], - "limitations": "Returns metadata only - the playground handles actual tool loading" - } - } - ] - } -} -``` - ---- - -### Component 2: BM25 Search API Endpoint - -**File:** `apps/web/src/app/api/tools/search/route.ts` - -**Requirements:** - -1. **Full-Text Search with BM25** - - Search across: tool description, package name, npm description, npm keywords - - BM25 scoring for relevance ranking - - Category filtering - - Quality score boosting (rich tier tools rank higher) - -2. **Search Implementation Options** - - **Option A: PostgreSQL Full-Text Search** - ```sql - -- Add tsvector column to tools table - ALTER TABLE tools ADD COLUMN search_vector tsvector; - - -- Create GIN index for fast full-text search - CREATE INDEX tools_search_idx ON tools USING GIN(search_vector); - - -- Update search vector on insert/update - CREATE TRIGGER tools_search_update - BEFORE INSERT OR UPDATE ON tools - FOR EACH ROW EXECUTE FUNCTION - tsvector_update_trigger(search_vector, 'pg_catalog.english', - description); - ``` - - **Option B: JavaScript BM25 Library** - ```typescript - import { BM25 } from 'bm25'; - - // Load all tools into memory (cached) - const tools = await prisma.tool.findMany({ - include: { package: true }, - }); - - // Build BM25 index - const documents = tools.map(tool => ({ - id: tool.id, - text: `${tool.description} ${tool.package.npmPackageName} ${tool.package.npmDescription} ${tool.package.npmKeywords.join(' ')}`, - })); - - const bm25 = new BM25(documents); - const results = bm25.search(query); - ``` - - **Option C: Hybrid Approach** - - Use PostgreSQL `LIKE` for exact matches (fastest) - - Fall back to BM25 for fuzzy/semantic search - - Cache search results in Redis - -3. **API Response Format** - -```typescript -// GET /api/tools/search?q=weather&category=integration&limit=10 - -{ - "success": true, - "query": "weather", - "filters": { - "category": "integration" - }, - "results": { - "total": 23, - "returned": 10, - "tools": [ - { - "id": "clx...", - "exportName": "getWeatherTool", - "description": "Get current weather data for any location using OpenWeatherMap API", - "qualityScore": 0.85, - "package": { - "npmPackageName": "@tpmjs/weather", - "npmVersion": "1.2.0", - "category": "integration", - "frameworks": ["vercel-ai"], - "env": [ - { - "name": "OPENWEATHER_API_KEY", - "description": "OpenWeatherMap API key", - "required": true - } - ], - "npmRepository": { - "type": "git", - "url": "https://github.com/user/weather-tool" - }, - "isOfficial": false - }, - // Include everything needed for dynamic import - "importUrl": "https://esm.sh/@tpmjs/weather@1.2.0", - "cdnUrl": "https://cdn.jsdelivr.net/npm/@tpmjs/weather@1.2.0/+esm" - } - // ... more tools - ] - } -} -``` - ---- - -### Component 3: Dynamic Tool Loader (Playground) - -**File:** `apps/playground/src/lib/dynamic-tool-loader.ts` - -**Requirements:** - -1. **Runtime ESM Import** - ```typescript - async function loadToolDynamically( - packageName: string, - exportName: string, - version: string - ) { - // Option 1: ESM CDN (esm.sh, unpkg, jsdelivr) - const cdnUrl = `https://esm.sh/${packageName}@${version}`; - - try { - const module = await import(/* @vite-ignore */ cdnUrl); - const tool = module[exportName]; - - if (!isValidTool(tool)) { - throw new Error(`Invalid tool: ${exportName}`); - } - - return tool; - } catch (error) { - console.error(`Failed to load ${packageName}:`, error); - return null; - } - } - ``` - -2. **Tool Caching Strategy** - ```typescript - // Cache loaded tools to avoid redundant imports - const toolCache = new Map(); - - function getCacheKey(packageName: string, exportName: string): string { - return `${packageName}::${exportName}`; - } - - async function loadToolWithCache( - packageName: string, - exportName: string, - version: string - ) { - const key = getCacheKey(packageName, exportName); - - if (toolCache.has(key)) { - return toolCache.get(key); - } - - const tool = await loadToolDynamically(packageName, exportName, version); - - if (tool) { - toolCache.set(key, tool); - } - - return tool; - } - ``` - -3. **Tool Registry Integration** - ```typescript - // Merge static tools + dynamically loaded tools - async function getAllAvailableTools( - staticTools: Record, - searchResults: SearchResult[] - ): Promise> { - const allTools = { ...staticTools }; - - // Load tools from search results - for (const result of searchResults) { - const tool = await loadToolWithCache( - result.package.npmPackageName, - result.exportName, - result.package.npmVersion - ); - - if (tool) { - const key = sanitizeToolName( - `${result.package.npmPackageName}-${result.exportName}` - ); - allTools[key] = tool; - } - } - - return allTools; - } - ``` - ---- - -### Component 4: Playground Chat Integration - -**File:** `apps/playground/src/app/api/chat/route.ts` - -**Flow:** - -1. **Initial Tool Set** - - Load static tools (hardcoded in tool-loader) - - Always include `searchTpmjsToolsTool` in initial set - -2. **Agent Invokes Search** - - Agent calls `searchTpmjsToolsTool` with query - - Search API returns matching tool metadata - - Response includes tool metadata - -3. **Dynamic Loading Trigger** - - Detect when agent successfully calls `searchTpmjsToolsTool` - - Extract tool metadata from response - - Load tools dynamically before next agent turn - -4. **Tool Availability Update** - - Merge dynamically loaded tools into available tool set - - Agent can now use newly loaded tools in subsequent turns - -**Implementation:** - -```typescript -// apps/playground/src/app/api/chat/route.ts -export async function POST(req: Request) { - const { messages } = await req.json(); - - // 1. Load static tools + search tool - let availableTools = await loadAllTools(); // static - availableTools['searchTpmjsTools'] = searchTpmjsToolsTool; // meta-tool - - // 2. Create streamText with current tools - const result = streamText({ - model: openai('gpt-4'), - messages, - tools: availableTools, - maxSteps: 10, // Allow multiple tool call rounds - - onStepFinish: async (step) => { - // 3. Check if agent called searchTpmjsToolsTool - for (const toolCall of step.toolCalls) { - if (toolCall.toolName === 'searchTpmjsTools') { - const searchResults = toolCall.result?.tools || []; - - // 4. Dynamically load tools from search results - console.log(`Loading ${searchResults.length} tools dynamically...`); - - for (const toolMeta of searchResults) { - const tool = await loadToolWithCache( - toolMeta.packageName, - toolMeta.exportName, - 'latest' // or toolMeta.version - ); - - if (tool) { - const key = sanitizeToolName( - `${toolMeta.packageName}-${toolMeta.exportName}` - ); - availableTools[key] = tool; - console.log(`✅ Loaded: ${key}`); - } - } - - // 5. Update tool registry for subsequent steps - // Note: This requires AI SDK to support dynamic tool updates - // May need to restart the streamText with updated tools - } - } - }, - }); - - return result.toDataStreamResponse(); -} -``` - ---- - -## Technical Challenges & Solutions - -### Challenge 1: AI SDK Doesn't Support Dynamic Tool Updates Mid-Stream - -**Problem:** Vercel AI SDK's `streamText` sets tools at initialization. Can't add tools after streaming starts. - -**Solutions:** - -**Option A: Multi-Turn Pattern** -```typescript -// Turn 1: Agent searches for tools -// Turn 2: Agent uses loaded tools - -// Detect search tool call, return early -if (hasSearchToolCall) { - return new Response(JSON.stringify({ - type: 'tools_loaded', - tools: searchResults, - message: 'Tools loaded. Please continue your request.', - })); -} -``` - -**Option B: Pre-Flight Search (Recommended)** -```typescript -// Before calling streamText, analyze user message -const needsTools = await analyzeMessageForToolNeeds(userMessage); - -if (needsTools.length > 0) { - // Pre-load tools based on intent - const searchResults = await searchTools(needsTools); - const dynamicTools = await loadToolsFromResults(searchResults); - availableTools = { ...staticTools, ...dynamicTools }; -} - -// Now call streamText with full tool set -const result = streamText({ - model, - messages, - tools: availableTools, -}); -``` - -**Option C: Agent-Driven Two-Phase** -```typescript -// Phase 1: Planning -const planResult = await generateText({ - model, - messages: [ - { role: 'system', content: 'Analyze this request and determine what tools are needed. Call searchTpmjsTools if needed.' }, - ...messages, - ], - tools: { searchTpmjsTools }, -}); - -// Phase 2: Execution with loaded tools -const executionResult = await streamText({ - model, - messages, - tools: { ...staticTools, ...loadedTools }, -}); -``` - ---- - -### Challenge 2: ESM Dynamic Import in Browser vs Node.js - -**Problem:** Dynamic `import()` works differently in browser vs server environments. - -**Solutions:** - -**Server-Side (Recommended):** -```typescript -// Use Node.js dynamic import -// Works with esm.sh CDN -const tool = await import(`https://esm.sh/${pkg}@${version}`); -``` - -**Client-Side (Avoid):** -```typescript -// Browser import() has CORS and CSP restrictions -// Would require: -// 1. CDN supports CORS -// 2. CSP allows script-src from CDN -// 3. Tools are browser-compatible (no Node.js APIs) -``` - -**Hybrid Approach:** -```typescript -// Load tools server-side, serialize to client -// Client displays available tools -// Server executes tool calls -``` - ---- - -### Challenge 3: Tool Dependencies & Environment Variables - -**Problem:** Dynamically loaded tools may require: -- Environment variables (API keys) -- npm dependencies not in bundle -- Node.js-specific APIs - -**Solutions:** - -**Option A: Require Pre-Configuration** -```typescript -// Before loading, check if tool requirements are met -async function canLoadTool(toolMeta: ToolMetadata): Promise { - // Check required env vars - for (const env of toolMeta.package.env || []) { - if (env.required && !process.env[env.name]) { - console.warn(`Missing required env: ${env.name}`); - return false; - } - } - - return true; -} -``` - -**Option B: Graceful Degradation** -```typescript -// Load tool, catch errors, inform agent -try { - const tool = await loadTool(packageName, exportName); - return tool; -} catch (error) { - return createStubTool(packageName, exportName, error); -} - -function createStubTool(pkg: string, exp: string, error: Error) { - return tool({ - description: `[UNAVAILABLE] ${exp} from ${pkg}: ${error.message}`, - parameters: z.object({}), - execute: async () => { - throw new Error(`Cannot execute ${exp}: ${error.message}`); - }, - }); -} -``` - -**Option C: Proxy Through Server** -```typescript -// All tools execute server-side where env vars exist -// Client just displays tool calls, server handles execution -``` - ---- - -### Challenge 4: Security & Sandboxing - -**Problem:** Dynamically importing arbitrary npm packages is a security risk. - -**Solutions:** - -**Option A: Allowlist Only** -```typescript -// Only load tools from TPMJS registry (already vetted) -const allowedPackages = await prisma.package.findMany({ - select: { npmPackageName: true } -}); - -if (!allowedPackages.includes(packageName)) { - throw new Error('Package not in TPMJS registry'); -} -``` - -**Option B: Version Pinning** -```typescript -// Only load specific versions from registry -// Don't use 'latest' to avoid supply chain attacks -const version = toolMeta.package.npmVersion; // e.g., "1.2.0" -const url = `https://esm.sh/${pkg}@${version}`; -``` - -**Option C: VM Sandbox (Advanced)** -```typescript -// Execute tools in isolated VM context -import { VM } from 'vm2'; - -const vm = new VM({ - timeout: 5000, - sandbox: { - fetch: safeFetch, // Wrapped fetch with rate limits - console: safeConsole, - }, -}); - -const tool = vm.run(toolCode); -``` - ---- - -### Challenge 5: Performance & Bundle Size - -**Problem:** Loading many tools dynamically could be slow. - -**Solutions:** - -**Option A: Lazy Loading** -```typescript -// Only load tools when agent decides to use them -// Not when they're discovered -``` - -**Option B: Parallel Loading** -```typescript -// Load multiple tools concurrently -const toolPromises = searchResults.map(result => - loadToolWithCache(result.package.npmPackageName, result.exportName, result.package.npmVersion) -); - -const tools = await Promise.all(toolPromises); -``` - -**Option C: CDN Caching** -```typescript -// Use CDN with aggressive caching -// esm.sh has built-in caching -const url = `https://esm.sh/${pkg}@${version}?target=es2022&bundle`; -``` - ---- - -## Implementation Plan - -### Phase 1: MVP (Week 1-2) - -**Goal:** Prove dynamic loading works with simple prototype - -1. **Create `@tpmjs/search-registry` package** - - Implement `searchTpmjsToolsTool` - - Publish to npm - - Add to manual-tools registry - -2. **Build `/api/tools/search` endpoint** - - Start with simple PostgreSQL `LIKE` search - - Return tool metadata with package info - - Test with curl - -3. **Implement basic dynamic loader** - - Use esm.sh CDN for imports - - Load tools server-side only - - Cache in memory - -4. **Playground integration - Two-Turn Pattern** - - User asks question - - Agent calls `searchTpmjsToolsTool` - - Backend loads tools - - Agent uses tools in next turn - -**Success Criteria:** -- Agent can search registry -- Agent can use dynamically loaded tools -- End-to-end flow works for 1-2 example tools - ---- - -### Phase 2: BM25 Search (Week 3) - -**Goal:** Improve search relevance with BM25 - -1. **Research BM25 implementation options** - - Test PostgreSQL full-text search - - Test JavaScript BM25 libraries - - Benchmark performance - -2. **Implement chosen approach** - - Add search vector column if using PostgreSQL - - Create search index - - Update search endpoint - -3. **Test search quality** - - Create test queries - - Measure precision/recall - - Compare to baseline `LIKE` search - -**Success Criteria:** -- BM25 search returns more relevant results than LIKE -- Search latency < 100ms for 95th percentile -- Agent can find tools for diverse queries - ---- - -### Phase 3: Production Hardening (Week 4) - -**Goal:** Make system production-ready - -1. **Error Handling** - - Handle import failures gracefully - - Validate tool schemas - - Return helpful error messages to agent - -2. **Security** - - Implement package allowlist - - Pin versions from registry - - Add rate limiting to search API - -3. **Performance** - - Implement Redis caching for search results - - Add CDN caching headers - - Optimize tool loading parallelism - -4. **Monitoring** - - Log all dynamic tool loads - - Track search queries and results - - Monitor import success/failure rates - -**Success Criteria:** -- System handles errors without crashing -- Security review passes -- Latency and reliability SLOs met - ---- - -### Phase 4: Advanced Features (Week 5+) - -**Goal:** Enhance UX and capabilities - -1. **Pre-flight Search** - - Analyze user message for intent - - Proactively load tools before agent call - - Reduce total turns needed - -2. **Tool Recommendations** - - "You might also need..." suggestions - - Based on tool co-occurrence data - - Help agent discover related tools - -3. **Client-Side Tool Display** - - Show which tools are available - - Indicate dynamically loaded tools - - Allow user to manually load tools - -4. **Tool Versioning** - - Support multiple versions of same tool - - Let agent choose version - - Handle breaking changes gracefully - ---- - -## Success Metrics - -### Technical Metrics - -1. **Search Quality** - - Precision@10 > 0.8 (80% of top 10 results are relevant) - - Mean Reciprocal Rank (MRR) > 0.7 - - Search latency p95 < 100ms - -2. **Tool Loading** - - Import success rate > 95% - - Tool load time p95 < 2 seconds - - Cache hit rate > 70% after warmup - -3. **End-to-End Performance** - - Total conversation latency < 5 seconds (including tool search + load + execution) - - Agent uses correct tools > 90% of time - -### User Metrics - -1. **Adoption** - - % of playground sessions using dynamic tools > 30% - - Number of unique tools loaded dynamically per week > 50 - -2. **Tool Coverage** - - % of user queries satisfied with available tools > 80% - - Tool search leading to successful task completion > 70% - ---- - -## Open Questions - -### 1. CDN Choice for ESM Imports - -**Options:** -- **esm.sh** - Purpose-built for ESM imports, fast, reliable -- **unpkg** - Popular, simple, but slower -- **jsdelivr** - Fast CDN, good for production -- **Custom bundler** - Pre-bundle tools, serve from our CDN - -**Recommendation:** Start with esm.sh for MVP, evaluate custom bundler for production. - ---- - -### 2. When to Load Tools? - -**Options:** -- **On-demand**: Load when agent calls search tool (current plan) -- **Pre-flight**: Analyze user message, load proactively -- **Lazy**: Load when agent tries to use tool (not when discovered) -- **Eager**: Load all tools from search results immediately - -**Recommendation:** Start with on-demand (Phase 1), add pre-flight in Phase 4. - ---- - -### 3. How to Handle Environment Variables? - -**Problem:** Dynamically loaded tools may need API keys (e.g., OpenWeather API). - -**Options:** -- **User provides**: UI for users to enter API keys (like playground settings) -- **Server-managed**: Admin pre-configures keys in .env -- **Graceful fail**: Load tool, but execution fails if env missing -- **Hybrid**: Some tools work without keys (free tier), others require keys - -**Recommendation:** Start with graceful fail (Phase 1), add user-provided keys (Phase 4). - ---- - -### 4. Should Tools Load Client-Side or Server-Side? - -**Client-Side Pros:** -- Reduces server load -- Faster for subsequent uses -- Better for browser-compatible tools - -**Client-Side Cons:** -- Requires CORS-enabled CDN -- CSP restrictions -- Many tools need Node.js APIs -- Exposing API keys in browser is insecure - -**Server-Side Pros:** -- Access to Node.js APIs -- Secure environment variable access -- No CORS issues -- Easier to implement - -**Server-Side Cons:** -- Requires server memory for caching -- Increases server load -- Cold starts for new tools - -**Recommendation:** Server-side for MVP (Phase 1), evaluate client-side for browser-compatible tools (Phase 4+). - ---- - -### 5. How to Handle Tool Dependencies? - -**Problem:** Some tools depend on other npm packages (e.g., `axios`, `cheerio`). - -**Options:** -- **Bundled**: CDN bundles dependencies (esm.sh does this) -- **Peer deps**: Require dependencies in playground package.json -- **Dynamic install**: npm install on-the-fly (slow, risky) -- **Pre-vetted**: Only allow tools with no/minimal dependencies - -**Recommendation:** Use esm.sh bundling (Phase 1), bundle size limits if issues arise. - ---- - -## Risk Assessment - -### High Risk - -1. **Security Vulnerability** - - **Risk**: Malicious package in registry executes code - - **Mitigation**: Allowlist registry packages, version pinning, VM sandboxing - - **Owner**: Security team - -2. **Performance Degradation** - - **Risk**: Loading many tools causes timeout/slow response - - **Mitigation**: Parallel loading, caching, lazy loading, timeouts - - **Owner**: Backend team - -### Medium Risk - -3. **Import Failures** - - **Risk**: CDN down, package incompatible, missing dependencies - - **Mitigation**: Fallback CDNs, error handling, stub tools - - **Owner**: Frontend team - -4. **AI SDK Limitations** - - **Risk**: Can't dynamically update tools mid-stream - - **Mitigation**: Two-turn pattern, pre-flight search - - **Owner**: AI team - -### Low Risk - -5. **Search Quality** - - **Risk**: BM25 doesn't return relevant tools - - **Mitigation**: A/B test search algorithms, collect feedback - - **Owner**: Search team - ---- - -## Future Enhancements - -### 1. Tool Composition -- Agent can combine multiple tools -- Example: `searchTool` + `summarizeTool` = search and summarize - -### 2. Tool Learning -- Track which tools are used together -- Recommend tool combinations -- "Users who used X also used Y" - -### 3. Custom Tool Registry -- Users can add private tools -- Organization-specific tool registry -- Access control and permissions - -### 4. Tool Marketplace -- Developers promote their tools -- Usage analytics and ratings -- Paid/premium tools - -### 5. Agent Templates -- Pre-configured agents with tool sets -- "Research Agent" has search + summarize tools -- "Code Agent" has code generation tools - ---- - -## Conclusion - -This dynamic tool loading system represents a paradigm shift in how AI agents discover and use tools. By making the TPMJS registry itself searchable, we enable infinite extensibility without the limitations of static bundling. - -**Key Innovation:** Self-referential tool discovery - a tool that searches for tools. - -**Next Steps:** -1. Review this PRD with team -2. Validate technical feasibility with ChatGPT/Claude -3. Spike on BM25 search implementation -4. Spike on dynamic ESM import -5. Begin Phase 1 implementation - -**Success Looks Like:** -- User: "Search Wikipedia for quantum computing" -- Agent: *searches registry, finds Wikipedia tool, loads it, uses it* -- User: Gets Wikipedia results without any manual tool configuration - -This is a novel approach that could define how AI agents discover and use tools. Let's build it. 🚀 diff --git a/ENV_VAR_TRANSPORT_ISSUE.md b/ENV_VAR_TRANSPORT_ISSUE.md deleted file mode 100644 index 3a04693..0000000 --- a/ENV_VAR_TRANSPORT_ISSUE.md +++ /dev/null @@ -1,86 +0,0 @@ -# Environment Variables Not Sent to API - Frontend Transport Issue - -## Problem - -Environment variables saved in localStorage are NOT being sent to `/api/chat` endpoint. - -**Evidence from logs:** -``` -📥 Request body: { - "conversationId": "7xur5hf1GDSOQMgFYF-l7", - "env": {}, // ❌ EMPTY - should have FIRECRAWL_API_KEY - ... -} -``` - -## Root Cause - -The issue is in `apps/playground/src/hooks/useChat.ts`: - -```typescript -export function useChat() { - const [conversationId] = useState(() => nanoid()); - const envVars = useEnvVars(); // ❌ Empty on first render (useEffect loads async) - - const envObject = envVars.reduce( - (acc, { key, value }) => { - acc[key] = value; - return acc; - }, - {} as Record - ); - - const chat = useAISDKChat({ - transport: new DefaultChatTransport({ // ❌ Created ONCE with empty envObject - api: '/api/chat', - body: { - conversationId, - env: envObject, // ❌ This is {} on first render, never updates - }, - }), - }); - - return { ...chat, conversationId }; -} -``` - -**Why it fails:** - -1. `useEnvVars()` loads from localStorage inside a `useEffect` (async) -2. On first render, `envVars = []`, so `envObject = {}` -3. `DefaultChatTransport` is created with `body: { env: {} }` -4. Even when `envVars` updates later, the transport is already created and doesn't re-create - -## Attempted Solutions That Don't Work - -❌ **Just updating state** - Transport is created once and cached -❌ **Using useEffect** - Transport is already created before effect runs - -## What We Need - -The `body` field in `DefaultChatTransport` needs to be **dynamic** and read the latest env vars on each request, not just once during component mount. - -## Questions for ChatGPT - -1. **How do we make `DefaultChatTransport` body dynamic?** Can we pass a function instead of an object? - -2. **Does AI SDK have a way to update transport body between messages?** The env vars might change while the chat is open. - -3. **Should we use a custom transport instead?** Can we implement our own transport that reads env vars fresh on each request? - -4. **Alternative: Can we manually add env to each message?** Is there a way to inject extra data per-request instead of per-transport? - -## Current Code Files - -- `apps/playground/src/hooks/useChat.ts` - The broken hook -- `apps/playground/src/components/sidebar/SettingsSidebar.tsx` - Where env vars are stored (works fine) -- `apps/playground/src/app/api/chat/route.ts` - Server expects `body.env` but receives `{}` - -## What We Know Works - -✅ Saving env vars to localStorage - working -✅ Reading env vars from localStorage - working -✅ Server accepting and using env vars - working -❌ **Sending env vars from client to server - BROKEN** - -The ONLY broken part is the transport not sending the latest env object. diff --git a/IMPLEMENTATION_CHECKLIST.md b/IMPLEMENTATION_CHECKLIST.md deleted file mode 100644 index f5afd62..0000000 --- a/IMPLEMENTATION_CHECKLIST.md +++ /dev/null @@ -1,782 +0,0 @@ -# TPMJS NPM Registry - Implementation Checklist - -> **Reference:** See [NPM_MIRROR.md](./NPM_MIRROR.md) for complete architecture details - -**Stack Decision:** Vercel + Neon + Vercel Cron (polling-based sync) - ---- - -## 🎯 Implementation Strategy - -### Architecture Simplification - -**Original Plan (NPM_MIRROR.md):** -- Separate Node.js sync service with persistent changes feed connection -- Self-hosted PostgreSQL -- More complex deployment - -**Revised Plan (This Checklist):** -- All-in-one Next.js app on Vercel -- Neon Postgres (serverless) -- Vercel Cron for sync jobs (polling-based, no persistent connections) -- Simpler, faster to ship - -### Why This Approach? - -✅ **Simpler Infrastructure** -- One deployment (Vercel) -- Managed database (Neon) -- Built-in cron (Vercel Cron) - -✅ **Lower Cost** -- No separate sync service hosting -- Neon free tier generous -- Vercel free/hobby tier sufficient for MVP - -✅ **Same Functionality** -- Poll NPM changes feed every 1-2 minutes (effectively real-time) -- All discovery features from NPM_MIRROR.md maintained -- Quality scoring, validation, etc. all work the same - ---- - -## 📋 Phase 1: Foundation (Week 1) - -**Goal:** Set up database, types, and NPM client - -### 1.1 Database Setup - -- [ ] **Create Neon project** - - Go to https://neon.tech/ - - Create new project - - Save connection string - -- [ ] **Create `packages/db` package** - ```bash - mkdir -p packages/db - cd packages/db - pnpm init - pnpm add prisma @prisma/client - pnpm add -D typescript @types/node - ``` - -- [ ] **Initialize Prisma** - ```bash - npx prisma init - ``` - -- [ ] **Create Prisma schema** - - Copy schema from NPM_MIRROR.md Database Design section - - File: `packages/db/prisma/schema.prisma` - - Include all three models: `Tool`, `SyncCheckpoint`, `SyncLog` - -- [ ] **Add database URL to `.env`** - ```env - DATABASE_URL="postgresql://..." - ``` - -- [ ] **Run first migration** - ```bash - npx prisma migrate dev --name init - npx prisma generate - ``` - -- [ ] **Create Prisma client singleton** - - File: `packages/db/src/client.ts` - ```typescript - import { PrismaClient } from '@prisma/client'; - - const globalForPrisma = globalThis as unknown as { - prisma: PrismaClient | undefined; - }; - - export const prisma = globalForPrisma.prisma ?? new PrismaClient(); - - if (process.env.NODE_ENV !== 'production') { - globalForPrisma.prisma = prisma; - } - ``` - -- [ ] **Export from package** - - File: `packages/db/src/index.ts` - ```typescript - export { prisma } from './client'; - export * from '@prisma/client'; - ``` - -- [ ] **Update package.json** - ```json - { - "name": "@tpmjs/db", - "main": "./src/index.ts", - "types": "./src/index.ts" - } - ``` - -- [ ] **Seed initial sync state** - ```sql - INSERT INTO sync_checkpoints (source, checkpoint) - VALUES - ('changes-feed', '{"sequence": 0}'::jsonb), - ('keyword-search', '{"lastRun": null}'::jsonb), - ('metrics', '{"lastRun": null}'::jsonb) - ON CONFLICT (source) DO NOTHING; - ``` - -**Verification:** -```bash -cd packages/db -npx prisma studio # Should open DB browser with empty tables -``` - ---- - -### 1.2 Types Package - -**Reference:** See NPM_MIRROR.md "The 'tpmjs' Field Schema" section - -- [ ] **Update `packages/types/src/tool.ts`** - - Add `TpmjsMinimalSchema` with Zod - - Add `TpmjsRichSchema` extending minimal - - Export both schemas and inferred types - -- [ ] **Create validation helper** - - File: `packages/types/src/validator.ts` - ```typescript - export function validateTpmjsField(tpmjs: unknown): { - valid: boolean; - tier: 'minimal' | 'rich' | null; - data?: unknown; - errors?: ZodError[]; - } - ``` - -- [ ] **Update exports** - - File: `packages/types/src/index.ts` - - Export all schemas and validators - -**Verification:** -```typescript -import { validateTpmjsField } from '@tpmjs/types'; - -const result = validateTpmjsField({ - category: 'web-scraping', - description: 'Test description that is long enough', - example: 'const x = await tool.test()' -}); - -console.log(result); // Should be { valid: true, tier: 'minimal', ... } -``` - ---- - -### 1.3 NPM Client Package - -**Reference:** See NPM_MIRROR.md "NPM Integration Strategy" section - -- [ ] **Create `packages/npm-client`** - ```bash - mkdir -p packages/npm-client/src - cd packages/npm-client - pnpm init - pnpm add zod - pnpm add -D typescript @types/node - ``` - -- [ ] **Implement changes feed client** - - File: `packages/npm-client/src/changes.ts` - ```typescript - export async function fetchChanges(since: string, limit = 100): Promise<{ - results: Array<{ id: string; seq: string }>; - lastSeq: string; - }> - ``` - - Use endpoint: `https://replicate.npmjs.com/registry/_changes` - - Poll-based (no EventSource needed) - -- [ ] **Implement keyword search** - - File: `packages/npm-client/src/search.ts` - ```typescript - export async function searchByKeyword( - keyword: string, - size = 250, - from = 0 - ): Promise> - ``` - - Use endpoint: `/-/v1/search?text=keywords:${keyword}` - -- [ ] **Implement package metadata fetcher** - - File: `packages/npm-client/src/package.ts` - ```typescript - export async function fetchPackageMetadata(packageName: string): Promise<{ - name: string; - 'dist-tags': { latest: string }; - versions: Record; - time: Record; - } | null> - ``` - - Use endpoint: `https://registry.npmjs.org/${packageName}` - -- [ ] **Implement download stats** - - File: `packages/npm-client/src/stats.ts` - ```typescript - export async function fetchDownloadStats( - packageName: string - ): Promise - ``` - - Use endpoint: `https://api.npmjs.org/downloads/point/last-month/${packageName}` - -- [ ] **Implement GitHub stats** (optional Phase 4) - - File: `packages/npm-client/src/github.ts` - ```typescript - export async function fetchGithubStars( - repoUrl: string - ): Promise - ``` - -- [ ] **Add rate limiting helper** - - File: `packages/npm-client/src/rate-limiter.ts` - - Simple delay between requests - - Exponential backoff on 429 - -- [ ] **Export all functions** - - File: `packages/npm-client/src/index.ts` - -**Verification:** -```typescript -import { fetchPackageMetadata } from '@tpmjs/npm-client'; - -const pkg = await fetchPackageMetadata('express'); -console.log(pkg?.name); // Should print 'express' -``` - ---- - -## 📋 Phase 2: Core API Routes (Week 2) - -**Goal:** Build public API for searching/listing tools - -### 2.1 Tool Search/List API - -**Reference:** See NPM_MIRROR.md "API Routes" section - -- [ ] **Create `apps/web/src/app/api/tools/route.ts`** - - Implement `GET` handler - - Query params: `q`, `category`, `official`, `limit`, `offset` - - Use Prisma to query `tools` table - - Return paginated results with metadata - -- [ ] **Add full-text search** - - Use Postgres `ts_vector` for search - - Or simple `ILIKE` for MVP - - Search across: `npmPackageName`, `description`, `tags` - -- [ ] **Add filtering** - - By `category` - - By `isOfficial` - - By `tier` (optional) - -- [ ] **Add sorting** - - Default: `qualityScore DESC`, `npmDownloadsLastMonth DESC` - - Optional: `createdAt DESC`, `npmPackageName ASC` - -**Verification:** -```bash -curl "http://localhost:3001/api/tools?q=web&limit=5" -# Should return JSON with tools array and pagination -``` - ---- - -### 2.2 Tool Detail API - -- [ ] **Create `apps/web/src/app/api/tools/[id]/route.ts`** - - Implement `GET` handler - - Accept ID or package name - - Return full tool details - -**Verification:** -```bash -curl "http://localhost:3001/api/tools/1" -# Should return single tool object -``` - ---- - -### 2.3 Validation API - -- [ ] **Create `apps/web/src/app/api/tools/validate/route.ts`** - - Implement `POST` handler - - Accept JSON body with `tpmjs` field - - Use `@tpmjs/types` validator - - Return validation result with errors - -**Verification:** -```bash -curl -X POST http://localhost:3001/api/tools/validate \ - -H "Content-Type: application/json" \ - -d '{"category":"web-scraping","description":"Test tool for validation","example":"const x = await tool.test()"}' -# Should return { valid: true, tier: "minimal" } -``` - ---- - -### 2.4 Stats API - -- [ ] **Create `apps/web/src/app/api/stats/route.ts`** - - Implement `GET` handler - - Aggregate counts by category - - Total tools, official tools, etc. - -**Verification:** -```bash -curl "http://localhost:3001/api/stats" -# Should return { totalTools: 0, officialTools: 0, categories: {} } -``` - ---- - -## 📋 Phase 3: Sync Workers (Week 2-3) - -**Goal:** Implement automatic NPM package discovery - -**Reference:** See NPM_MIRROR.md "NPM Integration Strategy" section - -### 3.1 Changes Feed Sync - -- [ ] **Create `apps/web/src/app/api/sync/changes/route.ts`** - -- [ ] **Implement POST handler** - ```typescript - export async function POST(request: Request) { - // 1. Verify CRON_SECRET header - // 2. Get last sequence from sync_checkpoints - // 3. Fetch changes from NPM (limit 100-500) - // 4. For each change: - // - Fetch package metadata - // - Check for tpmjs field - // - Validate with @tpmjs/types - // - Upsert to tools table - // - Log to sync_logs - // 5. Update checkpoint with new sequence - // 6. Return summary (processed, skipped, errors) - } - ``` - -- [ ] **Add secret protection** - ```typescript - const secret = request.headers.get('x-cron-secret'); - if (secret !== process.env.CRON_SECRET) { - return new Response('Unauthorized', { status: 401 }); - } - ``` - -- [ ] **Add timeout protection** - - Limit processing to 50 packages per run - - Or 50 seconds max execution time - - Save checkpoint frequently - -- [ ] **Add error handling** - - Try/catch around each package - - Log errors to `sync_logs` - - Continue processing other packages - -**Verification:** -```bash -curl -X POST http://localhost:3001/api/sync/changes \ - -H "x-cron-secret: your-secret" -# Should process changes and return summary -``` - ---- - -### 3.2 Keyword Search Sync - -- [ ] **Create `apps/web/src/app/api/sync/keyword/route.ts`** - -- [ ] **Implement POST handler** - ```typescript - export async function POST(request: Request) { - // 1. Verify CRON_SECRET header - // 2. Search NPM for keyword 'tpmjs-tool' - // 3. For each result: - // - Fetch package metadata - // - Validate tpmjs field - // - Upsert with isOfficial=true - // - Log to sync_logs - // 4. Update checkpoint - // 5. Return summary - } - ``` - -- [ ] **Handle pagination** - - NPM allows `size` up to 250 - - May need multiple requests for all results - -**Verification:** -```bash -curl -X POST http://localhost:3001/api/sync/keyword \ - -H "x-cron-secret: your-secret" -# Should search and process keyword packages -``` - ---- - -### 3.3 Metrics Sync (Phase 4) - -- [ ] **Create `apps/web/src/app/api/sync/metrics/route.ts`** - -- [ ] **Implement POST handler** - ```typescript - export async function POST(request: Request) { - // 1. Verify CRON_SECRET - // 2. Select tools to update (recent, popular, or sample) - // 3. For each tool: - // - Fetch NPM download stats - // - Fetch GitHub stars (if repo exists) - // - Calculate quality score - // - Update tools table - // 4. Update checkpoint - // 5. Return summary - } - ``` - -**Verification:** -```bash -curl -X POST http://localhost:3001/api/sync/metrics \ - -H "x-cron-secret: your-secret" -# Should update metrics for tools -``` - ---- - -### 3.4 Vercel Cron Configuration - -- [ ] **Add to `vercel.json`** - ```json - { - "crons": [ - { - "path": "/api/sync/changes", - "schedule": "*/2 * * * *" - }, - { - "path": "/api/sync/keyword", - "schedule": "*/15 * * * *" - }, - { - "path": "/api/sync/metrics", - "schedule": "0 * * * *" - } - ] - } - ``` - -- [ ] **Set up environment variables in Vercel** - - `DATABASE_URL` - Neon connection string - - `CRON_SECRET` - Generate random secret - - `NPM_REGISTRY_URL` - https://registry.npmjs.org - - `NPM_CHANGES_URL` - https://replicate.npmjs.com/registry - ---- - -## 📋 Phase 4: Frontend Integration (Week 3) - -**Goal:** Replace mock data with real API calls - -### 4.1 Update Tool Listing Page - -- [ ] **Update `apps/web/src/app/tools/page.tsx`** - - Remove mock data import - - Fetch from `/api/tools` - - Add loading state - - Add error handling - -- [ ] **Add search functionality** - - Search input component - - Debounced API calls - - Update URL with search params - -- [ ] **Add category filter** - - Category dropdown/pills - - Filter API calls by category - -- [ ] **Add pagination** - - Next/previous buttons - - Or infinite scroll - -**Verification:** -- Visit http://localhost:3001/tools -- Should show real tools from database -- Search should work -- Filters should work - ---- - -### 4.2 Update Tool Detail Page - -- [ ] **Update `apps/web/src/app/tools/[id]/page.tsx`** - - Fetch from `/api/tools/[id]` - - Display all tool metadata - - Show rich tier fields if available - -- [ ] **Add install instructions** - - npm install command - - Usage example from `tpmjs.example` - -- [ ] **Add links** - - NPM package page - - GitHub repository - - Documentation - - Playground (if available) - -**Verification:** -- Visit http://localhost:3001/tools/some-package -- Should show full tool details - ---- - -### 4.3 Update Homepage - -**Reference:** See NPM_MIRROR.md for stats display - -- [ ] **Update stats in hero section** - - Fetch from `/api/stats` - - Show real tool count - - Show category breakdown - -- [ ] **Update live metrics** - - Real download counts - - Real tool counts - - Update frequently (client-side polling or static) - -**Verification:** -- Visit http://localhost:3001 -- Stats should be real, not mock - ---- - -## 📋 Phase 5: Testing & Polish (Week 4) - -### 5.1 Create Test Packages - -- [ ] **Publish 3-5 real NPM packages with `tpmjs` field** - - At least one with minimal tier - - At least one with rich tier - - Use `tpmjs-tool` keyword for official listing - -- [ ] **Verify automatic discovery** - - Wait for next sync run - - Check they appear in database - - Check they appear on website - ---- - -### 5.2 Documentation - -- [ ] **Create docs section** - - `apps/web/src/app/docs/page.tsx` - - Getting started guide - - Schema reference - - Examples - -- [ ] **Add validation playground** - - `apps/web/src/app/docs/validate/page.tsx` - - Form to test `tpmjs` field - - Real-time validation feedback - - Uses `/api/tools/validate` - ---- - -### 5.3 CLI Tool (Optional) - -- [ ] **Create `packages/cli`** - - Command: `tpmjs validate` - - Reads local `package.json` - - Validates `tpmjs` field - - Calls `/api/tools/validate` - ---- - -### 5.4 Monitoring - -- [ ] **Add health endpoint** - - `apps/web/src/app/api/health/route.ts` - - Check database connectivity - - Check sync status (last run times) - -- [ ] **Set up uptime monitoring** - - Use UptimeRobot or Better Stack - - Monitor `/api/health` - - Alert if down or sync stale - -- [ ] **Add error tracking** - - Set up Sentry for Next.js - - Track API errors - - Track sync errors - -**Verification:** -```bash -curl http://localhost:3001/api/health -# Should return { status: "ok", db: "ok", sync: { ... } } -``` - ---- - -## 📋 Phase 6: Launch (Week 5) - -### 6.1 Pre-Launch Checklist - -- [ ] **Database** - - ✓ Prisma schema deployed - - ✓ Indexes created - - ✓ Backups enabled in Neon - -- [ ] **Environment Variables** - - ✓ All secrets in Vercel - - ✓ `CRON_SECRET` set - - ✓ `DATABASE_URL` set - -- [ ] **API Routes** - - ✓ All endpoints working - - ✓ Rate limiting added (optional) - - ✓ Error handling complete - -- [ ] **Sync Workers** - - ✓ Changes feed running every 2 min - - ✓ Keyword search running every 15 min - - ✓ Checkpoints updating correctly - -- [ ] **Frontend** - - ✓ All pages loading real data - - ✓ Search working - - ✓ Mobile responsive - -- [ ] **Monitoring** - - ✓ Health check endpoint live - - ✓ Uptime monitoring active - - ✓ Error tracking active - ---- - -### 6.2 Launch Steps - -- [ ] **Deploy to production** - ```bash - git push origin main - # Vercel auto-deploys - ``` - -- [ ] **Verify deployment** - - Check all pages load - - Check API endpoints work - - Check cron jobs run - -- [ ] **Publish announcement** - - Tweet/post about TPMJS - - Explain how to add `tpmjs` field - - Share validation endpoint - -- [ ] **Monitor for 24 hours** - - Watch error logs - - Check sync is working - - Fix any issues - ---- - -## 📋 Phase 7: Post-Launch (Ongoing) - -### Enhancements - -- [ ] **Semantic search** - - Add embeddings to tools table - - Use OpenAI/Cohere for semantic search - -- [ ] **Usage analytics** - - Track tool views - - Track search queries - - Popular tools widget - -- [ ] **Tool recommendations** - - "Similar tools" section - - "You might also like" - -- [ ] **GitHub Actions** - - Validate `tpmjs` field in CI - - Auto-comment validation results - -- [ ] **NPM webhooks** - - Listen for package updates - - Immediate sync (instead of polling) - ---- - -## 🎯 Success Criteria - -Check these metrics after launch: - -### Week 1 -- [ ] 10+ official tools listed -- [ ] All sync jobs running successfully -- [ ] Zero API errors - -### Month 1 -- [ ] 50+ official tools -- [ ] 5+ package authors using TPMJS -- [ ] <200ms API response time (p95) - -### Month 3 -- [ ] 200+ tools -- [ ] 20+ package authors -- [ ] Community contributions - -### Month 6 -- [ ] 1000+ tools -- [ ] 50+ active package authors -- [ ] Established as go-to AI tool registry - ---- - -## 🔄 Ongoing Maintenance - -Weekly: -- [ ] Check sync logs for errors -- [ ] Review new tools for quality -- [ ] Update documentation - -Monthly: -- [ ] Database optimization (indexes, vacuum) -- [ ] Review and adjust quality scoring -- [ ] Update NPM_MIRROR.md with learnings - ---- - -## 📚 Key Documents - -**Read frequently during implementation:** - -1. **NPM_MIRROR.md** - Complete architecture reference - - Database schema - - API specifications - - Validation rules - - Quality scoring - - All examples - -2. **This checklist** - Implementation order and verification steps - -3. **Plan file** - `.claude/plans/goofy-inventing-stearns.md` - Detailed planning notes - ---- - -## 🚀 Ready to Build - -This checklist is your complete implementation guide. Work through it phase by phase, checking off items as you go. - -**Start with Phase 1, Step 1.1** and work sequentially. Each step has verification instructions to ensure it's working before moving on. - -Good luck! 🎉 diff --git a/IMPLEMENTATION_STATUS.md b/IMPLEMENTATION_STATUS.md deleted file mode 100644 index 50bf6c1..0000000 --- a/IMPLEMENTATION_STATUS.md +++ /dev/null @@ -1,220 +0,0 @@ -# Dynamic Tool Loading - Implementation Status - -## ✅ Completed - -### 1. Search Tool Package (`@tpmjs/search-registry`) -- ✅ Created package with AI SDK v6 JSON Schema format -- ✅ Connects to search API endpoint -- ✅ Returns tool metadata (packageName, exportName, version, importUrl) -- ✅ Fixed schema format (was using Zod, now uses jsonSchema) -- ✅ Location: `packages/tools/search-registry/` - -### 2. Search API Endpoint (`/api/tools/search`) -- ✅ Implemented simple text-based search (BM25 had dependency issues) -- ✅ Searches by keywords in description, package name, keywords -- ✅ Returns tools with import URLs for esm.sh -- ✅ Location: `apps/web/src/app/api/tools/search/route.ts` - -### 3. Pre-flight Tool Loading in Playground -- ✅ Automatic search on every user message -- ✅ Extracts user query from last message -- ✅ Calls searchTpmjsTools automatically -- ✅ Attempts to load discovered tools dynamically -- ✅ Location: `apps/playground/src/app/api/chat/route.ts` - -### 4. Dynamic Tool Loader (Railway Service Approach) -- ✅ Updated to call Railway service instead of local imports -- ✅ Calls `/load-and-describe` endpoint to get tool schema -- ✅ Wraps tool with remote execution via `/execute-tool` endpoint -- ✅ Caches tool wrappers locally -- ✅ Location: `apps/playground/src/lib/dynamic-tool-loader.ts` - -### 5. Documentation -- ✅ DYNAMIC_IMPORT_ISSUE.md - Comprehensive problem analysis -- ✅ RAILWAY_DYNAMIC_TOOL_LOADER.md - Railway implementation guide -- ✅ This file - Implementation status - -## 🚧 Pending (Railway Service Implementation) - -### Railway Service Endpoints Needed - -You need to add these two endpoints to your existing Railway service: - -#### 1. `POST /load-and-describe` - -**Purpose**: Load a tool from esm.sh and return its schema - -**Request**: -```json -{ - "packageName": "firecrawl-aisdk", - "exportName": "webSearchTool", - "version": "0.7.2", - "importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2" -} -``` - -**Response**: -```json -{ - "success": true, - "tool": { - "exportName": "webSearchTool", - "description": "Search the web using Firecrawl", - "inputSchema": { - "type": "object", - "properties": { - "query": { "type": "string" } - } - } - } -} -``` - -**Implementation Reference**: See `RAILWAY_DYNAMIC_TOOL_LOADER.md` for full code - -#### 2. `POST /execute-tool` - -**Purpose**: Execute a dynamically loaded tool with parameters - -**Request**: -```json -{ - "packageName": "firecrawl-aisdk", - "exportName": "webSearchTool", - "version": "0.7.2", - "importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2", - "params": { - "query": "latest AI news" - } -} -``` - -**Response**: -```json -{ - "success": true, - "output": { "results": [...] }, - "executionTimeMs": 1234 -} -``` - -**Implementation Reference**: See `RAILWAY_DYNAMIC_TOOL_LOADER.md` for full code - -### Deployment Requirements - -1. **Railway Service**: - - Must run with `--experimental-network-imports` flag - - Add to start command: `node --experimental-network-imports server.js` - -2. **Environment Variables** (Vercel): - ```bash - RAILWAY_SERVICE_URL=https://your-railway-service.up.railway.app - # or reuse existing: - SANDBOX_EXECUTOR_URL=https://your-railway-service.up.railway.app - ``` - -3. **Local Testing** (Railway service on port 3001): - ```bash - RAILWAY_SERVICE_URL=http://localhost:3001 - ``` - -## 🎯 Testing Checklist - -Once Railway endpoints are deployed: - -- [ ] Test `/load-and-describe` endpoint directly with curl -- [ ] Test `/execute-tool` endpoint directly with curl -- [ ] Test full flow in playground: - - [ ] Ask: "search the web for latest AI news" - - [ ] Verify pre-flight search finds tools - - [ ] Verify tools load via Railway - - [ ] Verify tool execution works - - [ ] Check console logs for debugging info - -## 📊 Current Flow - -``` -User: "search the web for latest AI news" - ↓ - Chat API extracts query - ↓ - Automatically calls searchTpmjsTools - ↓ - Search API returns matching tools - (packageName, exportName, version) - ↓ - loadToolsBatch() called for each tool - ↓ - For each tool: - 1. Check local cache - 2. If not cached: - → POST to Railway: /load-and-describe - ← Get back: description + inputSchema - 3. Create wrapper tool with: - - description from Railway - - inputSchema from Railway - - execute() → calls Railway /execute-tool - 4. Cache wrapper locally - ↓ - All tools available to agent - ↓ - Agent calls tool (wrapper) - ↓ - Wrapper → POST to Railway: /execute-tool - ↓ - Railway imports from esm.sh and executes - ↓ - Result returned to agent - ↓ - Agent uses result to answer user -``` - -## 🔍 Debugging - -Check console logs for: -- `📦 Loading from Railway` - Tool loading initiated -- `✅ Tool loaded from Railway` - Tool schema received -- `🚀 Executing ... remotely` - Tool execution initiated -- `✅ Tool executed successfully` - Tool execution complete -- `❌ Railway service error` - Connection failed -- `❌ Failed to load tool` - Import failed - -## 📁 Files Modified - -1. `packages/tools/search-registry/src/index.ts` - Search tool -2. `packages/tools/search-registry/package.json` - AI SDK version -3. `apps/web/src/app/api/tools/search/route.ts` - Search endpoint -4. `apps/playground/src/app/api/chat/route.ts` - Pre-flight search -5. `apps/playground/src/lib/dynamic-tool-loader.ts` - Railway integration -6. `apps/playground/next.config.ts` - Added urlImports (unused) -7. `apps/playground/src/lib/tool-loader.ts` - Removed firecrawl - -## 🚀 Next Steps - -1. **Deploy Railway endpoints** using code from `RAILWAY_DYNAMIC_TOOL_LOADER.md` -2. **Set environment variables** in Vercel -3. **Test locally** with Railway service running on localhost:3001 -4. **Deploy to production** and test with real tools -5. **Monitor logs** for any issues - -## 💡 Key Insights - -- **Next.js Limitation**: Cannot do dynamic HTTP imports due to bundler -- **Railway Solution**: Plain Node.js with `--experimental-network-imports` -- **Caching Strategy**: Two-level cache (local wrapper + Railway module) -- **Execution Model**: Remote execution in Railway, not Next.js -- **Security**: Tools execute in Railway sandbox, not Vercel -- **Performance**: First load ~1-2s (import), cached loads <10ms - -## 📚 Related Documentation - -- `DYNAMIC_IMPORT_ISSUE.md` - Problem analysis and ChatGPT response -- `RAILWAY_DYNAMIC_TOOL_LOADER.md` - Full Railway implementation guide -- Plan file: `~/.claude/plans/jiggly-inventing-dragon.md` - ---- - -**Status**: Ready for Railway deployment -**Blocker**: Railway `/load-and-describe` and `/execute-tool` endpoints need implementation -**ETA**: 30-60 minutes to implement Railway endpoints + test diff --git a/LAUNCH_REVIEW.md b/LAUNCH_REVIEW.md new file mode 100644 index 0000000..edb20e4 --- /dev/null +++ b/LAUNCH_REVIEW.md @@ -0,0 +1,331 @@ +# TPMJS Launch Review & Checklist + +**STATUS: COMPLETED** - All critical issues have been fixed. + +A comprehensive review of all public-facing content for Hacker News launch readiness. + +--- + +## Executive Summary + +**Overall Readiness: 7/10 - Needs Work Before Launch** + +The website has excellent technical content and professional design, but fails the "5-second test" - a first-time visitor cannot quickly understand what TPMJS is or why they need it. The documentation is strong for existing users but assumes too much prior knowledge about AI agents and tooling. + +### Critical Issues (Must Fix) +1. **Landing page doesn't explain what TPMJS is** - Hero section uses jargon without definition +2. **"Tool" vs "Package" never defined** - Core concepts assumed, not explained +3. **Knowledge gaps** - Assumes familiarity with AI agents, Zod, semantic search +4. **Category inconsistency** - HOW_TO_PUBLISH and NPM_MIRROR have different category lists +5. **NPM_MIRROR.md conflicts with other docs** - Appears outdated, creates confusion + +### What's Working Well +- Publishing guide (HOW_TO_PUBLISH_A_TOOL.md) is excellent +- How It Works page has great technical depth +- Developer testimonials are concrete with real metrics +- No obvious AI-generated language on website +- Code examples are practical and well-placed + +--- + +## The 5-Second Test: FAILED + +**Question:** Can a developer understand what TPMJS is within 5 seconds of landing on the homepage? + +**Answer:** No. + +### What They See First +``` +TOOL REGISTRY FOR AI AGENTS +Discover, share, and integrate tools that give your agents superpowers +``` + +### What's Missing +- What is a "tool" in this context? +- What is an "AI agent"? +- Why would I use this vs npm directly? +- Is this a package manager? A marketplace? An SDK? + +### The "Aha Moment" is Unclear +A visitor still doesn't know: +- WHO should use TPMJS (tool builders? agent developers? both?) +- WHEN they would use it (at development time? runtime?) +- HOW it differs from regular npm packages +- WHY they can't just install packages normally + +--- + +## Page-by-Page Clarity Ratings + +| Page | Clarity | Human Feel | Issues | +|------|---------|------------|--------| +| **Landing Page** | 5/10 | Yes | No 5-second explanation, jargon-heavy | +| **Hero Section** | 3/10 | Yes | "Tool registry" undefined, circular language | +| **Problem Section** | 7/10 | Yes | Best section - concrete pain points | +| **Vision Section** | 5/10 | Yes | "Semantic search" unexplained | +| **Developer Stories** | 7/10 | Yes | Good metrics, but code unexplained | +| **Publish Section** | 6/10 | Yes | Assumes visitor is a tool builder | +| **How It Works** | 9/10 | Excellent | Minor density issues | +| **FAQ** | 8/10 | Yes | Missing some common questions | +| **Publish Guide** | 8.5/10 | Yes | Tier system could be clearer upfront | +| **Spec Page** | 8.5/10 | Yes | Assumes Zod/AI SDK knowledge | +| **Docs Page** | 9/10 | Excellent | Overwhelming length | +| **SDK Page** | 8.5/10 | Yes | Assumes Vercel AI SDK familiarity | +| **Privacy** | 8/10 | Yes | Hardcoded email address | +| **Terms** | 8/10 | Yes | Hardcoded date | + +--- + +## Documentation Clarity Ratings + +| Document | Clarity | Necessary | Critical Issues | +|----------|---------|-----------|-----------------| +| README.md | 8/10 | YES | Missing "what is TPMJS" explanation | +| HOW_TO_PUBLISH_A_TOOL.md | 9/10 | YES | Minor - excellent overall | +| DEPLOYMENT.md | 8/10 | YES | Confusing exit code explanation | +| QUALITY-GATES.md | 7/10 | OPTIONAL | Could merge into README | +| MANUAL_TOOLS.md | 8.5/10 | YES | Good for maintainers | +| NPM_MIRROR.md | 6.5/10 | **REMOVE** | **Conflicts with other docs, appears outdated** | + +--- + +## Knowledge Gaps (Things Visitors Won't Understand) + +### Not Explained Anywhere +1. **What is an "AI Agent"?** - The entire site assumes you know this +2. **What is a "Tool" vs a "Package"?** - Used interchangeably, never defined +3. **Why semantic search matters** - Just says "semantic" without explaining benefit +4. **What frameworks are supported** - Mentioned in FAQ but not prominently +5. **The Package → Tool relationship** - Can one package have multiple tools? + +### Assumed Technical Knowledge +- Zod schemas (used throughout, never introduced) +- AI SDK tool format (referenced as "standard" but what standard?) +- esm.sh and Deno sandboxing (mentioned in How It Works) +- BM25 ranking algorithm (mentioned in docs) + +### Missing Use Cases +- "Use TPMJS when..." section doesn't exist +- No comparison to alternatives (why not just npm?) +- No "before/after" showing the problem solved + +--- + +## Human-Written Assessment + +### Reads Like Human: YES ✓ +- Developer stories use specific metrics ("500 lines to 3") +- Technical explanations show genuine understanding +- Problem section addresses real pain points +- No buzzword soup or meaningless marketing phrases + +### Minor AI-Sounding Phrases Found +| Location | Phrase | Issue | +|----------|--------|-------| +| NPM_MIRROR.md:7 | "automated NPM-integrated registry" | Marketing speak | +| NPM_MIRROR.md:27 | "✨ Listed automatically" | Emoji in technical doc | +| NPM_MIRROR.md:500 | "Built with ❤️" | Remove emoji | +| HOW_TO_PUBLISH:389 | "AI-friendly descriptions" | Vague - what makes it "AI-friendly"? | +| Vision Section | "gives agents superpowers" | Metaphor without substance | + +--- + +## Critical Inconsistencies Found + +### Category Lists Don't Match +**HOW_TO_PUBLISH_A_TOOL.md says:** +``` +text-analysis, code-generation, data-processing, +image-generation, audio-processing, search, integration, other +``` + +**NPM_MIRROR.md says:** +``` +web-scraping, data-processing, file-operations, communication, +database, api-integration, image-processing, text-analysis, +automation, ai-ml, security, monitoring +``` + +**These are completely different!** Which is correct? + +### Quality Score Formula Conflicts +- HOW_TO_PUBLISH: "Tier: Rich (1.0) > Basic (0.5) > Minimal (0.25)" +- MANUAL_TOOLS: "Rich tier tools get 4x quality score multiplier" +- NPM_MIRROR: Different formula entirely + +### Field Names Inconsistent +- `exportName` used in MANUAL_TOOLS but not in HOW_TO_PUBLISH +- Deprecated fields (`parameters`, `returns`) mentioned but unclear when deprecated + +--- + +## Hardcoded Values to Fix + +| File | Issue | Line | +|------|-------|------| +| FAQ, Privacy, Terms | `thomasalwyndavis@gmail.com` hardcoded | Multiple | +| Privacy, Terms | Date "December 14, 2025" hardcoded | Multiple | +| Changelog page | Package list hardcoded in code | ~95-110 | +| Developer Stories | Fictional company names (Support.ai, DocFlow) | homePageData.ts | + +--- + +## Launch Checklist + +### Must Fix Before Launch (Blocking) - ALL DONE ✓ + +- [x] **Rewrite hero section** to explain TPMJS in one sentence + - Current: "TOOL REGISTRY FOR AI AGENTS" + - Suggested: "TPMJS lets AI agents discover and use npm packages as tools at runtime. Publish once to npm, get discovered automatically." + +- [x] **Add "What is TPMJS?" section** to landing page + - Define: What is an AI agent? + - Define: What is a "tool" in this context? + - Explain: Why not just use npm directly? + - Show: 3-step "how it works" visual + +- [x] **Reconcile category lists** between docs (deleted NPM_MIRROR.md) + - Pick one canonical list + - Update all docs to match + - Add categories to types package + +- [x] **Delete or archive NPM_MIRROR.md** (deleted) + - Conflicts with HOW_TO_PUBLISH + - Appears to be old design doc, not current state + - Move to `/docs/internal/` if historical value + +- [x] **Fix hardcoded values** (emails → hello@tpmjs.com, dates → December 2024) + - Email addresses → environment variable + - Dates → dynamic or remove + - Package lists → generated from filesystem + +### Should Fix (High Priority) - MOSTLY DONE + +- [x] **Add "Use TPMJS when..." section** to landing page (covered in "What is TPMJS?" section) + - List concrete scenarios: "Building a chatbot that needs web access" + - "Agent that processes different file formats" + - "Tool that should be discoverable by other agents" + +- [x] **Explain Package vs Tool distinction** (covered in "What is TPMJS?" section) + - Add glossary or definitions section + - Clarify: 1 package can have N tools + +- [x] **Add framework compatibility section** (mentioned in hero and publish sections) + - Which AI frameworks work with TPMJS? + - Are there adapters needed? + - Show code for each framework + +- [ ] **Simplify developer stories code** + - Current code snippet unexplained: + ```js + const agent = new Agent({ tools: await tpmjs.search(...) }) + ``` + - Add: Where does `Agent` come from? What's happening here? + +- [x] **Add README context** (completely rewritten with clear explanation) + - What is TPMJS for? + - Link to tpmjs.com + - Explain discovery mechanism + +### Nice to Have (Post-Launch) + +- [ ] Add video walkthrough (30-60 seconds) +- [ ] Interactive playground link from homepage +- [ ] "Compare to alternatives" section +- [ ] Case studies with real company names +- [ ] Quick links sidebar for docs page +- [ ] Status badges for each quality gate + +--- + +## Recommended Hero Section Rewrite + +### Current +``` +TOOL REGISTRY FOR AI AGENTS +Discover, share, and integrate tools that give your agents superpowers +The registry for AI tools +``` + +### Suggested +``` +MAKE YOUR AI AGENT SMARTER +TPMJS connects your AI agent to 2,500+ npm packages at runtime. +No config files. No manual imports. Just describe what you need. + +"Find me a tool that can scrape websites" → Your agent gets web-scraper +"I need to process markdown" → Your agent gets markdown-formatter + +Publish your npm package → It's discoverable by every AI agent in 15 minutes. +``` + +This version: +- Explains what it DOES (connects agents to npm packages) +- Shows HOW it works (natural language → tool) +- States the VALUE (no config, automatic discovery) +- Gives concrete examples + +--- + +## Recommended "What is TPMJS?" Section + +Add after hero, before featured tools: + +```markdown +## What is TPMJS? + +**The Problem:** AI agents need tools (web scraping, file processing, API calls) +but developers must manually configure each one. As the ecosystem grows, +this becomes unmanageable. + +**The Solution:** TPMJS is a registry that automatically discovers npm packages +designed for AI agents. Agents can search for tools by description and load them +at runtime. + +**For Tool Builders:** Add `tpmjs-tool` keyword to your package.json. +Your tool appears on tpmjs.com within 15 minutes. + +**For Agent Developers:** Use semantic search to find tools: +```javascript +import { searchRegistry } from '@tpmjs/sdk'; +const tools = await searchRegistry('send emails and slack messages'); +// Returns: email-sender, slack-notifier, ... +``` + +**One registry. Thousands of tools. Zero configuration.** +``` + +--- + +## Final Assessment + +### Ready for Launch? +**Not yet.** The core product is solid but messaging fails first-time visitors. + +### Estimated Fixes +- Hero rewrite: 30 minutes +- "What is TPMJS?" section: 1 hour +- Category reconciliation: 1 hour +- Hardcoded values: 30 minutes +- README updates: 30 minutes +- NPM_MIRROR cleanup: 15 minutes + +**Total: ~4 hours of work** + +### After Fixes +The site will be launch-ready. The technical content is excellent - it just needs a better front door. + +--- + +## Appendix: Positive Highlights + +Things that are already great and should NOT change: + +1. **How It Works page** - Excellent technical depth, clear structure +2. **Publishing guide** - Best-in-class documentation, real examples +3. **Problem section** - Concrete pain points, relatable issues +4. **Spec page** - Clear field reference, good validation info +5. **SDK documentation** - Quick start is excellent +6. **Code examples throughout** - Practical, copy-pasteable +7. **Visual design** - Clean, professional, developer-focused +8. **Quality scoring explanation** - Transparent, well-documented diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..7d72b03 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2024-2025 TPMJS + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/NPM_MIRROR.md b/NPM_MIRROR.md deleted file mode 100644 index 61d05cb..0000000 --- a/NPM_MIRROR.md +++ /dev/null @@ -1,500 +0,0 @@ -# TPMJS NPM-Integrated Registry Architecture - -> **Automated tool discovery from NPM with zero-click submission** - -## Vision - -Transform TPMJS from a manual directory into an **automated NPM-integrated registry** where package authors simply publish to NPM with a `tpmjs` field in their `package.json` and their tools are discovered and listed within seconds—no manual submission, no forms, no waiting. - -## Quick Start for Package Authors - -```json -{ - "name": "my-awesome-tool", - "version": "1.0.0", - "keywords": ["tpmjs-tool"], - "tpmjs": { - "category": "web-scraping", - "description": "Extract product data from e-commerce websites with ease", - "example": "const data = await scraper.extract('https://shop.com')" - } -} -``` - -```bash -npm publish -# ✨ Listed automatically within 15 minutes (keyword) or seconds (changes feed) -``` - ---- - -## Architecture Overview - -``` -NPM Ecosystem - ↓ -Changes Feed + Keyword Search - ↓ -Package Validator (Zod) - ↓ -PostgreSQL Database - ↓ -Next.js API Routes - ↓ -TPMJS Web App -``` - -### Core Components - -1. **NPM Sync Service** (Node.js) - Monitors NPM registry for new packages -2. **PostgreSQL Database** - Stores validated tool metadata -3. **Next.js API** - Serves tool data with search/filtering -4. **Web Frontend** - Browse, search, and discover tools - ---- - -## Discovery Mechanism: Hybrid Approach - -### Method 1: Keyword Search (Official) -- Search NPM for packages with `tpmjs-tool` keyword -- Runs every 15 minutes via cron -- Packages marked as "Official" - -### Method 2: Changes Feed (Automatic) -- Monitors `replicate.npmjs.com/registry/_changes` in real-time -- Detects packages with `tpmjs` field instantly -- Packages marked as "Community" (unless they also have keyword) - -### Why Hybrid? -- **Keywords** = Clear opt-in, queryable, respects NPM conventions -- **Changes Feed** = Real-time, catches packages without keywords -- **Together** = Best discoverability with fallback - ---- - -## The "tpmjs" Field: Tiered Schema - -### Minimal Tier (Required) - -```json -{ - "tpmjs": { - "category": "web-scraping", - "description": "Extract structured data from websites using CSS selectors", - "example": "const data = await tool.scrape({ url: 'https://example.com', selector: '.price' })" - } -} -``` - -**Categories:** -- web-scraping -- data-processing -- file-operations -- communication -- database -- api-integration -- image-processing -- text-analysis -- automation -- ai-ml -- security -- monitoring - -### Rich Tier (Optional) - -Extend with any of these optional fields: - -```json -{ - "tpmjs": { - // ... Required fields ... - - "parameters": [ - { - "name": "url", - "type": "string", - "description": "Target URL to scrape", - "required": true - } - ], - "returns": { - "type": "object", - "description": "Extracted data matching the selector" - }, - "authentication": { - "required": false, - "type": "api-key", - "envVar": "SCRAPER_API_KEY", - "docsUrl": "https://docs.example.com/auth" - }, - "pricing": { - "model": "freemium", - "freeLimit": "100 requests/month", - "paidUrl": "https://example.com/pricing" - }, - "frameworks": ["vercel-ai", "langchain", "llamaindex"], - "links": { - "documentation": "https://docs.example.com", - "playground": "https://example.com/try", - "repository": "https://github.com/user/repo" - }, - "tags": ["web", "scraping", "html", "css"], - "status": "stable", - "aiAgent": { - "useCase": "Use when agent needs to extract data from websites", - "limitations": "Cannot handle JavaScript-heavy SPAs" - } - } -} -``` - ---- - -## Database Schema - -### Tools Table - -```sql -CREATE TABLE tools ( - -- NPM Metadata - npm_package_name VARCHAR(214) UNIQUE NOT NULL, - npm_version VARCHAR(50) NOT NULL, - npm_published_at TIMESTAMP NOT NULL, - npm_description TEXT, - npm_repository JSONB, - npm_homepage TEXT, - npm_license VARCHAR(50), - - -- TPMJS Metadata - category VARCHAR(50) NOT NULL, - description TEXT NOT NULL, - example TEXT NOT NULL, - parameters JSONB, - authentication JSONB, - pricing JSONB, - frameworks TEXT[], - links JSONB, - tags TEXT[], - status VARCHAR(20), - - -- Discovery - discovery_method VARCHAR(20) NOT NULL, -- 'keyword' | 'changes-feed' - is_official BOOLEAN DEFAULT false, - tier VARCHAR(20) NOT NULL, -- 'minimal' | 'rich' - - -- Metrics - npm_downloads_last_month INTEGER DEFAULT 0, - github_stars INTEGER DEFAULT 0, - quality_score DECIMAL(3,2), -- 0.00 to 1.00 - - -- Timestamps - created_at TIMESTAMP DEFAULT NOW(), - updated_at TIMESTAMP DEFAULT NOW() -); -``` - ---- - -## Sync Service Architecture - -### Workers - -**1. Changes Feed Worker** -- Connects to `replicate.npmjs.com/registry/_changes` -- Receives real-time change events -- Fetches package metadata for each change -- Checks for `tpmjs` field -- Validates and inserts to database - -**2. Keyword Search Worker** -- Runs every 15 minutes (cron) -- Searches `/-/v1/search?text=keywords:tpmjs-tool` -- Processes all results -- Marks as "Official" - -**3. Metrics Worker** (Optional Phase 4) -- Updates download counts from NPM API -- Fetches GitHub stars -- Calculates quality scores - -### Package Processing Pipeline - -``` -1. Fetch package metadata from NPM -2. Extract `tpmjs` field from latest version -3. Validate against Zod schema -4. If valid → Insert/Update database -5. If invalid → Log error -6. If no field → Skip -``` - ---- - -## API Routes - -### GET /api/tools -Search and list tools - -**Query Parameters:** -- `q` - Search query -- `category` - Filter by category -- `official` - Only official tools (true/false) -- `limit` - Results per page (default 20) -- `offset` - Pagination offset - -**Response:** -```json -{ - "tools": [...], - "pagination": { - "total": 150, - "limit": 20, - "offset": 0, - "hasMore": true - } -} -``` - -### GET /api/tools/[id] -Get tool details by ID - -### POST /api/tools/validate -Validate a `tpmjs` field before publishing - -**Request:** -```json -{ - "category": "web-scraping", - "description": "...", - "example": "..." -} -``` - -**Response:** -```json -{ - "valid": true, - "tier": "minimal", - "errors": [] -} -``` - -### GET /api/stats -Registry statistics - -```json -{ - "totalTools": 2847, - "officialTools": 150, - "categories": { - "web-scraping": 320, - "communication": 280, - ... - } -} -``` - ---- - -## Quality Scoring Algorithm - -Tools are scored 0.00 to 1.00 based on: - -- **Base validity** (0.3) - Has valid schema -- **Tier** (0.1-0.2) - Rich tier > Minimal tier -- **NPM downloads** (0.2) - Based on monthly downloads -- **GitHub stars** (0.15) - Repository popularity -- **Documentation** (0.1) - Has docs URL -- **Example quality** (0.05) - Example length > 100 chars - -Score is used for default sorting and quality indicators. - ---- - -## Implementation Phases - -### Phase 1: Foundation (Week 1-2) -- Set up PostgreSQL + Prisma -- Create Zod schemas in `@tpmjs/types` -- Build sync service structure -- Implement NPM API client - -### Phase 2: Discovery (Week 2-3) -- Implement changes feed worker -- Implement keyword search worker -- Deploy sync service (Railway/Fly.io) -- Test with real packages - -### Phase 3: API & Frontend (Week 3-4) -- Build Next.js API routes -- Update tool listing page -- Update tool detail pages -- Add validation endpoint - -### Phase 4: Polish (Week 4-5) -- Add metrics worker -- Create documentation -- Build CLI validator -- Launch to community - -### Phase 5: Enhancements (Post-Launch) -- Semantic search (embeddings) -- Usage analytics -- Tool recommendations -- GitHub Actions integration - ---- - -## Infrastructure Requirements - -### Sync Service -- **Platform:** Railway or Fly.io -- **Runtime:** Node.js 22+ -- **Resources:** 512MB RAM, 1 CPU -- **Cost:** ~$5-10/month - -### Database -- **Platform:** Neon Postgres (serverless) -- **Size:** Free tier (start), scale as needed -- **Backups:** Automatic with Neon -- **Cost:** Free tier available, ~$10-20/month for production - -### Web App -- **Platform:** Vercel (existing) -- **No changes required** - ---- - -## Monitoring & Health - -### Metrics to Track - -1. **Sync Health** - - Changes feed uptime - - Packages processed per hour - - Validation success rate - -2. **Database** - - Total tools - - Official vs community ratio - - Tier distribution - -3. **API** - - Request latency (p95 < 200ms) - - Search performance - - Error rates - -### Alerts - -- Sync service down > 5 minutes -- Database connection failures -- Validation error rate > 10% - ---- - -## Developer Experience - -### Validation Before Publishing - -```bash -# Using TPMJS CLI (to be built) -npx tpmjs validate - -# Or via API -curl -X POST https://tpmjs.com/api/tools/validate \ - -H "Content-Type: application/json" \ - -d '{"category":"web-scraping","description":"...","example":"..."}' -``` - -### Documentation Pages Needed - -1. **Getting Started** - Adding TPMJS support -2. **Schema Reference** - Complete field docs -3. **Best Practices** - Tips for quality tools -4. **Examples** - Sample configurations -5. **FAQ** - Common questions - ---- - -## Migration from Mock Data - -### Current State -- 12 mock tools in `toolData.ts` -- Client-side search -- Hard-coded categories - -### Migration Strategy - -1. **Publish Real Packages** - - Create NPM packages for mock tools - - Add `tpmjs` fields - - Publish with `tpmjs-tool` keyword - -2. **Update Frontend** - - Replace mock data with API calls - - Keep existing UI components - - Update types to match Prisma models - -3. **Gradual Rollout** - - Dual mode (mock + real) - - Real data primary, mock fallback - - Remove mock entirely - ---- - -## Success Metrics - -### Technical -- ✓ Discovery latency < 60 seconds -- ✓ API response time < 200ms p95 -- ✓ Support 10,000+ tools -- ✓ 99.9% uptime - -### User Experience -- ✓ 0-click submission (automatic) -- ✓ Instant validation feedback -- ✓ <100ms search speed -- ✓ 100% mobile features - -### Business -- Week 1: 10 official tools -- Month 1: 50 official tools -- Month 3: 200+ tools -- Month 6: 1000+ tools -- 50+ active package authors - ---- - -## Comparison to Vercel's Approach - -| Feature | Vercel AI SDK | TPMJS | -|---------|---------------|-------| -| **Submission** | Manual file edit + PR | Automatic via NPM | -| **Discovery** | None | Real-time changes feed | -| **Validation** | Manual review | Automated Zod schema | -| **Updates** | New PR required | Automatic on publish | -| **Search** | Static array | Full-text + categories | -| **Scale** | 6 tools | 1000+ tools ready | - ---- - -## Next Steps - -1. Review this architecture plan -2. Approve database schema and API design -3. Set up infrastructure (Railway + Postgres) -4. Start Phase 1: Foundation -5. Launch MVP in 4-5 weeks - ---- - -## References - -- [NPM Registry API Docs](https://github.com/npm/registry/blob/main/docs/REGISTRY-API.md) -- [NPM Changes Feed](https://github.com/npm/registry/blob/main/docs/REPLICATE-API.md) -- [Vercel AI Tools Registry](https://github.com/vercel/ai/blob/main/content/tools-registry/registry.ts) -- [TPMJS Architecture Plan](/.claude/plans/goofy-inventing-stearns.md) (Full details) - ---- - -**Built with ❤️ for the AI agent ecosystem** diff --git a/OPENAI_SCHEMA_ERROR.md b/OPENAI_SCHEMA_ERROR.md deleted file mode 100644 index f7ef9e9..0000000 --- a/OPENAI_SCHEMA_ERROR.md +++ /dev/null @@ -1,522 +0,0 @@ -# OpenAI Schema Validation Error - AI SDK v6 - -## ✅ RESOLVED - -**Solution:** Use `tool()` and `jsonSchema()` from AI SDK instead of Zod for tool definitions. - -## Error Message - -``` -Error [AI_APICallError]: Invalid schema for function 'helloWorld': schema must be a JSON Schema of 'type: "object"', got 'type: "None"'. -``` - -## Context - -Building a Next.js playground app to test AI SDK v6 tool execution with OpenAI's GPT-4o-mini model. The error occurs when OpenAI validates the tool schema sent in the API request. - -## Root Cause - -Zod 4.0.0 generates JSON Schema with `allOf` + `$ref` at the root level instead of a direct `type: "object"`. OpenAI's API requires a JSON Schema with `type: "object"` at the root, so it rejects Zod 4 schemas with `type: "None"` error. - -## Environment - -- **AI SDK Version**: `ai@6.0.0-beta.124` -- **OpenAI Provider**: `@ai-sdk/openai@3.0.0-beta.74` -- **OpenAI Library**: `openai@^6.9.1` -- **Zod Version**: `zod@^4.0.0` -- **Next.js Version**: `next@^16.0.4` -- **Node.js**: Latest -- **TypeScript**: Strict mode enabled - -## Tool Definition - -Located at: `packages/tools/hello/src/index.ts` - -```typescript -import { z } from 'zod'; - -/** - * Hello World Tool - * Returns a simple "Hello, World!" greeting - * - * This is a proper AI SDK v6 tool that can be used with streamText() - */ -export const helloWorldTool = { - description: 'Returns a simple "Hello, World!" greeting message', - parameters: z.object({ - // OpenAI requires at least one optional parameter, can't be completely empty - includeTimestamp: z.boolean().optional().describe('Whether to include a timestamp in the response'), - }), - execute: async ({ includeTimestamp = true }: { includeTimestamp?: boolean }) => { - const response: any = { - message: 'Hello, World!', - }; - - if (includeTimestamp) { - response.timestamp = new Date().toISOString(); - } - - return response; - }, -}; - -/** - * Hello Name Tool - * Returns a personalized greeting with the provided name - * - * This is a proper AI SDK v6 tool that can be used with streamText() - */ -export const helloNameTool = { - description: 'Returns a personalized greeting with the provided name', - parameters: z.object({ - name: z.string().describe('The name of the person to greet'), - }), - execute: async ({ name }: { name: string }) => { - return { - message: `Hello, ${name}!`, - timestamp: new Date().toISOString(), - }; - }, -}; -``` - -## Tool Loading - -Located at: `apps/playground/src/lib/tool-loader.ts` - -```typescript -// Static imports for tools (required for Next.js/webpack) -import { helloWorldTool, helloNameTool } from '@tpmjs/hello'; -import { scrapeTool, crawlTool, searchTool } from 'firecrawl-aisdk'; - -/** - * Load a specific TPMJS tool by package name - */ -export async function loadTpmjsTool(packageName: string): Promise { - try { - // Map package names to their tool functions - switch (packageName) { - case '@tpmjs/hello': - // Hello has multiple tools, return all of them - return { - helloWorld: helloWorldTool, - helloName: helloNameTool, - }; - - case 'firecrawl-aisdk': - // Firecrawl has multiple tools, return all of them - return { - scrapeTool, - crawlTool, - searchTool, - }; - - default: - throw new Error(`Unknown tool package: ${packageName}`); - } - } catch (error) { - if (error instanceof Error) { - throw new Error(`Failed to load tool from package ${packageName}: ${error.message}`); - } - throw new Error(`Failed to load tool from package ${packageName}: Unknown error`); - } -} - -/** - * Load all installed TPMJS tools - */ -export async function loadAllTools(): Promise> { - const installedTools = ['@tpmjs/hello', 'firecrawl-aisdk']; - - const tools: Record = {}; - - for (const packageName of installedTools) { - try { - const tool = await loadTpmjsTool(packageName); - - // If the tool returns an object with multiple tools (like firecrawl), spread them - if (tool && typeof tool === 'object' && !tool.description) { - Object.assign(tools, tool); - } else { - // Single tool - use a cleaned name (remove hyphens, camelCase) - const toolName = packageName.replace(/-([a-z])/g, (_match, letter) => letter.toUpperCase()).replace(/-/g, ''); - tools[toolName] = tool; - } - } catch (error) { - console.error(`Failed to load tool ${packageName}:`, error); - // Continue loading other tools even if one fails - } - } - - return tools; -} -``` - -## API Route - -Located at: `apps/playground/src/app/api/chat/route.ts` - -```typescript -import { loadAllTools } from '~/lib/tool-loader'; -import { openai } from '@ai-sdk/openai'; -import { streamText } from 'ai'; -import { NextRequest } from 'next/server'; - -export const runtime = 'nodejs'; -export const dynamic = 'force-dynamic'; -export const maxDuration = 60; - -export async function POST(request: NextRequest) { - try { - const body = await request.json(); - const { messages } = body; - - if (!messages || !Array.isArray(messages)) { - return new Response(JSON.stringify({ error: 'Invalid request: messages array required' }), { - status: 400, - headers: { 'Content-Type': 'application/json' }, - }); - } - - // Load all available tools - const tools = await loadAllTools(); - - console.log('Loaded tools:', Object.keys(tools)); - - // Create system message - const systemMessage = { - role: 'system' as const, - content: `You are a helpful AI assistant that can use TPMJS tools to help users. - -Available tools: -${Object.entries(tools) - .map(([name, tool]) => `- ${name}: ${tool.description}`) - .join('\n')} - -Call tools as needed to answer user questions. Execute tools directly.`, - }; - - // Stream the AI response with tools - const result = streamText({ - model: openai('gpt-4o-mini'), - messages: [systemMessage, ...messages], - tools, - maxSteps: 5, - }); - - return result.toTextStreamResponse(); - } catch (error) { - console.error('Chat API error:', error); - return new Response( - JSON.stringify({ - error: error instanceof Error ? error.message : 'Unknown error occurred', - }), - { - status: 500, - headers: { 'Content-Type': 'application/json' }, - } - ); - } -} -``` - -## Package Configuration - -Located at: `packages/tools/hello/package.json` - -```json -{ - "name": "@tpmjs/hello", - "version": "0.0.1", - "private": true, - "description": "Example TPMJS tools - Hello World and Hello Name", - "main": "dist/index.js", - "types": "dist/index.d.ts", - "scripts": { - "build": "tsc", - "dev": "tsc --watch", - "clean": "rm -rf dist", - "type-check": "tsc --noEmit" - }, - "keywords": [ - "tpmjs-tool", - "ai-sdk", - "hello", - "example" - ], - "tpmjs": { - "category": "text-analysis", - "description": "Simple greeting tools - Hello World and personalized Hello Name greetings" - }, - "dependencies": { - "ai": "6.0.0-beta.124", - "zod": "^4.0.0" - }, - "devDependencies": { - "@tpmjs/tsconfig": "workspace:*", - "typescript": "^5.9.3" - }, - "files": [ - "dist", - "README.md" - ] -} -``` - -## TypeScript Configuration - -Located at: `packages/tools/hello/tsconfig.json` - -```json -{ - "compilerOptions": { - "target": "ES2020", - "module": "commonjs", - "lib": ["ES2020"], - "outDir": "./dist", - "rootDir": "./src", - "declaration": true, - "declarationMap": true, - "sourceMap": true, - "strict": true, - "esModuleInterop": true, - "skipLibCheck": true, - "forceConsistentCasingInFileNames": true, - "moduleResolution": "node", - "resolveJsonModule": true - }, - "include": ["src/**/*"], - "exclude": ["node_modules", "dist"] -} -``` - -## Compiled Output - -Located at: `packages/tools/hello/dist/index.js` - -```javascript -"use strict"; -Object.defineProperty(exports, "__esModule", { value: true }); -exports.helloNameTool = exports.helloWorldTool = void 0; -const zod_1 = require("zod"); -/** - * Hello World Tool - * Returns a simple "Hello, World!" greeting - * - * This is a proper AI SDK v6 tool that can be used with streamText() - */ -exports.helloWorldTool = { - description: 'Returns a simple "Hello, World!" greeting message', - parameters: zod_1.z.object({ - // OpenAI requires at least one optional parameter, can't be completely empty - includeTimestamp: zod_1.z.boolean().optional().describe('Whether to include a timestamp in the response'), - }), - execute: async ({ includeTimestamp = true }) => { - const response = { - message: 'Hello, World!', - }; - if (includeTimestamp) { - response.timestamp = new Date().toISOString(); - } - return response; - }, -}; -/** - * Hello Name Tool - * Returns a personalized greeting with the provided name - * - * This is a proper AI SDK v6 tool that can be used with streamTime() - */ -exports.helloNameTool = { - description: 'Returns a personalized greeting with the provided name', - parameters: zod_1.z.object({ - name: zod_1.z.string().describe('The name of the person to greet'), - }), - execute: async ({ name }) => { - return { - message: `Hello, ${name}!`, - timestamp: new Date().toISOString(), - }; - }, -}; -``` - -## Full Error Response from OpenAI - -```json -{ - "error": { - "message": "Invalid schema for function 'helloWorld': schema must be a JSON Schema of 'type: \"object\"', got 'type: \"None\"'.", - "type": "invalid_request_error", - "param": "tools[0].parameters", - "code": "invalid_function_parameters" - } -} -``` - -API endpoint: `https://api.openai.com/v1/responses` -Status code: 400 - -## Problem Analysis - -1. **OpenAI expects JSON Schema format** - The `tools[0].parameters` field must be a valid JSON Schema object with `type: "object"` - -2. **AI SDK v6 should convert Zod to JSON Schema** - The AI SDK is supposed to automatically convert Zod schemas to JSON Schema when sending to OpenAI, but it's producing `type: "None"` instead - -3. **Potential causes**: - - Zod 4.0.0 compatibility issue with AI SDK v6 beta - - AI SDK not properly converting the Zod schema - - Issue with how the tool object is structured - - Problem with how tools are passed to `streamText()` - -4. **Already tried**: - - Added at least one parameter (even optional) to helloWorldTool - - Used proper Zod schema with `.describe()` for descriptions - - Followed AI SDK v6 tool definition format exactly - - Built the package successfully (dist folder exists) - -## AI SDK v6 Tool Format Reference - -According to AI SDK v6 documentation, a tool should be defined as: - -```typescript -{ - description: string, - parameters: ZodSchema, - execute: async (args) => Promise -} -``` - -This matches our implementation exactly. - -## Questions for ChatGPT - -1. Is there a known compatibility issue between AI SDK v6 Beta (6.0.0-beta.124) and Zod 4.0.0? - -2. Does the AI SDK v6 require a specific tool registration format when passing to `streamText()`? - -3. Should tools be wrapped in a different structure (e.g., using `tool()` helper function)? - -4. Is there a way to manually convert Zod schema to JSON Schema that OpenAI accepts? - -5. Are there any known issues with using workspace packages (`@tpmjs/hello`) in Next.js API routes with dynamic imports? - -6. Should we downgrade to Zod 3.x instead of Zod 4.0.0? - -7. Is there a debug mode to see what JSON Schema is being sent to OpenAI? - -## Additional Context - -- The `firecrawl-aisdk` package works correctly with the same setup -- Build process completes successfully with no TypeScript errors -- The tool is being loaded and passed to `streamText()` correctly -- Error only occurs when OpenAI validates the tool schema -- This is a monorepo using pnpm workspaces and Turborepo - -## Related Files - -- Tool definition: `packages/tools/hello/src/index.ts` -- Tool loader: `apps/playground/src/lib/tool-loader.ts` -- API route: `apps/playground/src/app/api/chat/route.ts` -- Package config: `packages/tools/hello/package.json` -- Compiled output: `packages/tools/hello/dist/index.js` - -## Expected Behavior - -Tools should be automatically converted from Zod schema to JSON Schema by AI SDK v6 and accepted by OpenAI's API. - -## Actual Behavior - -OpenAI rejects the tool schema with error: `got 'type: "None"'` instead of a valid JSON Schema object. - ---- - -## ✅ SOLUTION IMPLEMENTED - -### What We Changed - -Instead of using Zod schemas with `parameters`, we now use AI SDK's `tool()` helper with `jsonSchema()` for the input schema. This bypasses Zod's JSON Schema conversion entirely. - -### Before (Broken with Zod 4) - -```typescript -import { z } from 'zod'; - -export const helloWorldTool = { - description: 'Returns a simple "Hello, World!" greeting message', - parameters: z.object({ - includeTimestamp: z.boolean().optional().describe('Whether to include a timestamp'), - }), - execute: async ({ includeTimestamp = true }) => { - // ... - }, -}; -``` - -### After (Working with jsonSchema) - -```typescript -import { jsonSchema, tool } from 'ai'; - -type HelloWorldInput = { - includeTimestamp?: boolean; -}; - -export const helloWorldTool = tool({ - description: 'Returns a simple "Hello, World!" greeting message', - inputSchema: jsonSchema({ - type: 'object', - properties: { - includeTimestamp: { - type: 'boolean', - description: 'Whether to include a timestamp in the response', - }, - }, - additionalProperties: false, - }), - async execute({ includeTimestamp = true }) { - const response: any = { - message: 'Hello, World!', - }; - if (includeTimestamp) { - response.timestamp = new Date().toISOString(); - } - return response; - }, -}); -``` - -### Key Changes - -1. **Import from `ai`**: Added `jsonSchema` and `tool` imports -2. **Define TypeScript types**: Created `HelloWorldInput` type for type safety -3. **Use `tool()` wrapper**: Wraps the entire tool definition -4. **Use `jsonSchema()` for schema**: Provides explicit JSON Schema with `type: "object"` at root -5. **Removed Zod dependency**: No longer need `zod` in package.json - -### Benefits - -- ✅ Works with OpenAI's strict schema validation -- ✅ Explicit control over JSON Schema structure -- ✅ Full TypeScript type safety with generic types -- ✅ No dependency on Zod (one less package to maintain) -- ✅ Follows AI SDK v6 best practices -- ✅ Guaranteed `type: "object"` at root level - -### Updated Package Dependencies - -```json -{ - "dependencies": { - "ai": "6.0.0-beta.124" - } -} -``` - -Zod is no longer needed in tool packages that use `jsonSchema()`. - -### References - -- [AI SDK Core: tool](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool) -- [AI SDK Core: jsonSchema](https://ai-sdk.dev/docs/reference/ai-sdk-core/json-schema) -- [GitHub Issue: Zod 4 JSON Schema compatibility](https://github.com/vercel/ai/issues/10240) diff --git a/RAILWAY_DEPLOYMENT_NOTE.md b/RAILWAY_DEPLOYMENT_NOTE.md deleted file mode 100644 index 2f192ce..0000000 --- a/RAILWAY_DEPLOYMENT_NOTE.md +++ /dev/null @@ -1,98 +0,0 @@ -# Railway Executor - Deployment Status - -## Issue Discovered - -Node.js does not support HTTP(S) imports by default, even with `--experimental-network-imports` flag (that flag doesn't exist in current Node versions). - -## Solutions Considered - -1. **Custom ESM Loader** - Complex, requires Node.js 18.19+ with `--loader` flag -2. **fetch + eval** - Security concerns, doesn't handle ES modules properly -3. **Bundler approach** - Would defeat the purpose of dynamic imports -4. **Deno** - Supports HTTP imports natively, but different ecosystem - -## Recommended Solution - -Since the core issue is that we need truly dynamic runtime imports from HTTP URLs, and Node.js doesn't support this, we have **two viable paths**: - -### Option A: Use Deno on Railway (RECOMMENDED) - -Deno supports HTTP imports natively: - -```typescript -// server.ts (Deno) -import { serve } from "https://deno.land/std@0.208.0/http/server.ts"; - -const moduleCache = new Map(); - -async function loadTool(url: string, exportName: string) { - if (moduleCache.has(url)) { - return moduleCache.get(url); - } - - // Deno supports this natively! - const module = await import(url); - const tool = module[exportName]; - moduleCache.set(url, tool); - return tool; -} - -serve(async (req) => { - // ... handle requests -}, { port: 3002 }); -``` - -**Deploy to Railway:** -```bash -# In Railway dashboard: -# - Set Start Command: deno run --allow-net --allow-env server.ts -# - Or use railway.json with deno runtime -``` - -### Option B: Pre-build Bundle Approach - -Instead of truly dynamic imports, pre-fetch and cache tools: - -1. Playground searches for tools -2. Backend fetches tool code once and caches it -3. Use `vm2` or similar to execute in sandbox -4. Not truly "dynamic" but works with Node.js - -## Current Status - -The Railway executor service is **created** but **not deployed** because Node.js doesn't support the required HTTP imports. - -**Files created:** -- `apps/railway-executor/package.json` -- `apps/railway-executor/server.js` (incomplete - needs Deno or vm2 approach) -- `apps/railway-executor/README.md` - -## Next Steps - -**If using Deno (recommended):** -1. Rewrite server.js as server.ts for Deno -2. Deploy to Railway with Deno runtime -3. Test HTTP imports work -4. Update playground to use Railway URL - -**If sticking with Node.js:** -1. Install `vm2` package for sandboxed execution -2. Implement fetch + vm2 approach -3. Deploy to Railway -4. Accept limitations (less dynamic, more complex) - -## Alternative: Skip Railway, Use Different Architecture - -Since the original issue is Next.js bundler limitations, consider: - -**Web Workers in Browser** - Load tools client-side using native `import()` -- Pros: No server needed, truly dynamic -- Cons: Exposes API keys, security concerns - -**Serverless Functions with Pre-installed Tools** - Deploy each tool as separate function -- Pros: Works with Vercel/Next.js -- Cons: Not truly dynamic, requires redeployment for new tools - ---- - -**Recommendation**: Use Deno on Railway. It's designed for exactly this use case. diff --git a/RAILWAY_DYNAMIC_TOOL_LOADER.md b/RAILWAY_DYNAMIC_TOOL_LOADER.md deleted file mode 100644 index e77b66a..0000000 --- a/RAILWAY_DYNAMIC_TOOL_LOADER.md +++ /dev/null @@ -1,376 +0,0 @@ -# Railway Service - Dynamic Tool Loader Implementation - -## Overview - -This document describes the Railway service implementation needed to support dynamic tool loading from esm.sh in the TPMJS playground. - -## Why Railway Service? - -Next.js/Turbopack intercepts all `import()` calls and tries to resolve them through its module graph. HTTP URLs like `https://esm.sh/...` are not supported. - -**Solution**: Use a plain Node.js service on Railway that: -- Runs with `--experimental-network-imports` flag -- Can dynamically import from HTTP URLs (esm.sh) -- Executes tool functions and returns results -- Is already set up for existing ToolPlayground - -## New Endpoint Required - -### `POST /load-and-describe` - -**Purpose**: Dynamically import a tool package and return its AI SDK tool definition (description, schema) without executing it. - -**Request**: -```json -{ - "packageName": "firecrawl-aisdk", - "exportName": "webSearchTool", - "version": "0.7.2", - "importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2" -} -``` - -**Response**: -```json -{ - "success": true, - "tool": { - "exportName": "webSearchTool", - "description": "Search the web using Firecrawl", - "inputSchema": { - "type": "object", - "properties": { - "query": { "type": "string", "description": "Search query" } - }, - "required": ["query"] - } - } -} -``` - -**Implementation** (pseudo-code for Railway service): - -```javascript -// server.js (Railway service) -import express from 'express'; - -const app = express(); -app.use(express.json()); - -// Cache for imported modules -const moduleCache = new Map(); - -app.post('/load-and-describe', async (req, res) => { - const { packageName, exportName, version, importUrl } = req.body; - - const cacheKey = `${packageName}::${exportName}`; - - try { - let toolModule; - - // Check cache first - if (moduleCache.has(cacheKey)) { - console.log(`✅ Cache hit: ${cacheKey}`); - toolModule = moduleCache.get(cacheKey); - } else { - // Dynamic import from esm.sh - const url = importUrl || `https://esm.sh/${packageName}@${version}`; - console.log(`📦 Importing: ${url}`); - - const module = await import(url); - toolModule = module[exportName]; - - if (!toolModule) { - return res.status(404).json({ - success: false, - error: `Export "${exportName}" not found in module` - }); - } - - // Validate it's an AI SDK tool - if (!toolModule.description || !toolModule.execute) { - return res.status(400).json({ - success: false, - error: `Invalid AI SDK tool structure` - }); - } - - // Cache it - moduleCache.set(cacheKey, toolModule); - } - - // Extract tool definition (description + schema) - // AI SDK v6 tools have: description, inputSchema, execute - res.json({ - success: true, - tool: { - exportName, - description: toolModule.description, - inputSchema: toolModule.inputSchema || toolModule.parameters?.shape || {}, - } - }); - } catch (error) { - console.error('Failed to load tool:', error); - res.status(500).json({ - success: false, - error: error.message - }); - } -}); - -// Start server -const PORT = process.env.PORT || 3000; -app.listen(PORT, () => { - console.log(`Railway tool loader running on port ${PORT}`); -}); -``` - -**Railway Deployment**: -```bash -# Start command in Railway settings: -node --experimental-network-imports server.js - -# Or in package.json: -{ - "scripts": { - "start": "node --experimental-network-imports server.js" - } -} -``` - -## Modified Endpoint: `POST /execute-tool` - -**Purpose**: Execute a dynamically loaded tool with parameters. - -**Request**: -```json -{ - "packageName": "firecrawl-aisdk", - "exportName": "webSearchTool", - "version": "0.7.2", - "importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2", - "params": { - "query": "latest AI news" - } -} -``` - -**Response**: -```json -{ - "success": true, - "output": { - "results": [...] - }, - "executionTimeMs": 1234 -} -``` - -**Implementation** (pseudo-code): - -```javascript -app.post('/execute-tool', async (req, res) => { - const { packageName, exportName, version, importUrl, params } = req.body; - - const cacheKey = `${packageName}::${exportName}`; - const startTime = Date.now(); - - try { - let toolModule; - - // Check cache or import - if (moduleCache.has(cacheKey)) { - toolModule = moduleCache.get(cacheKey); - } else { - const url = importUrl || `https://esm.sh/${packageName}@${version}`; - const module = await import(url); - toolModule = module[exportName]; - - if (!toolModule || !toolModule.execute) { - return res.status(404).json({ - success: false, - error: 'Tool not found or invalid' - }); - } - - moduleCache.set(cacheKey, toolModule); - } - - // Execute the tool - const result = await toolModule.execute(params); - - res.json({ - success: true, - output: result, - executionTimeMs: Date.now() - startTime - }); - } catch (error) { - res.status(500).json({ - success: false, - error: error.message, - executionTimeMs: Date.now() - startTime - }); - } -}); -``` - -## Integration with Playground - -### 1. Update `dynamic-tool-loader.ts` - -Replace local dynamic imports with Railway service calls: - -```typescript -// apps/playground/src/lib/dynamic-tool-loader.ts - -const RAILWAY_SERVICE_URL = process.env.RAILWAY_SERVICE_URL || 'http://localhost:3001'; - -export async function loadToolDynamically( - packageName: string, - exportName: string, - version: string, - importUrl?: string -): Promise { - const cacheKey = getCacheKey(packageName, exportName); - - // Check local cache first - if (moduleCache.has(cacheKey)) { - console.log(`✅ Cache hit: ${cacheKey}`); - return moduleCache.get(cacheKey); - } - - try { - console.log(`📦 Loading from Railway: ${packageName}/${exportName}`); - - // Call Railway service to load and describe tool - const response = await fetch(`${RAILWAY_SERVICE_URL}/load-and-describe`, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - packageName, - exportName, - version, - importUrl, - }), - }); - - if (!response.ok) { - console.error(`❌ Railway service error: ${response.status}`); - return null; - } - - const data = await response.json(); - - if (!data.success) { - console.error(`❌ Failed to load tool: ${data.error}`); - return null; - } - - // Create a tool wrapper that executes remotely - const tool = { - description: data.tool.description, - inputSchema: data.tool.inputSchema, - execute: async (params: any) => { - console.log(`🚀 Executing ${packageName}/${exportName} remotely`); - - const execResponse = await fetch(`${RAILWAY_SERVICE_URL}/execute-tool`, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - packageName, - exportName, - version, - importUrl, - params, - }), - }); - - const result = await execResponse.json(); - - if (!result.success) { - throw new Error(result.error || 'Tool execution failed'); - } - - return result.output; - }, - }; - - // Cache the wrapper - moduleCache.set(cacheKey, tool); - console.log(`✅ Loaded and cached: ${cacheKey}`); - - return tool; - } catch (error) { - console.error(`❌ Failed to load ${packageName}#${exportName}:`, error); - return null; - } -} -``` - -### 2. Environment Variables - -Add to `.env.local`: -```bash -RAILWAY_SERVICE_URL=https://your-railway-service.up.railway.app -``` - -Or for local testing with Railway running locally: -```bash -RAILWAY_SERVICE_URL=http://localhost:3001 -``` - -## Testing Locally - -### Terminal 1: Run Railway service locally -```bash -cd railway-service -node --experimental-network-imports server.js -``` - -### Terminal 2: Run playground -```bash -cd tpmjs -pnpm dev --filter=@tpmjs/playground -``` - -### Test the flow: -```bash -# Test Railway service directly -curl -X POST http://localhost:3001/load-and-describe \ - -H "Content-Type: application/json" \ - -d '{ - "packageName": "firecrawl-aisdk", - "exportName": "webSearchTool", - "version": "0.7.2" - }' - -# Then test via playground UI -# Navigate to http://localhost:3000/playground -# Ask: "search the web for latest AI news" -``` - -## Deployment Checklist - -- [ ] Create Railway service with Node.js -- [ ] Add `--experimental-network-imports` flag to start command -- [ ] Deploy `/load-and-describe` endpoint -- [ ] Deploy `/execute-tool` endpoint (or modify existing `/execute`) -- [ ] Set `RAILWAY_SERVICE_URL` in Vercel environment variables -- [ ] Test with real tools from TPMJS registry -- [ ] Monitor Railway logs for import errors - -## Benefits - -1. ✅ **Works around Next.js limitations** - Imports happen in plain Node -2. ✅ **Reuses existing Railway infrastructure** - No new service needed -3. ✅ **Caching on both sides** - Local cache + Railway cache -4. ✅ **Security** - Tools execute in Railway sandbox, not Next.js -5. ✅ **Scalability** - Railway handles the heavy lifting - -## Next Steps - -1. Implement Railway service endpoints -2. Update `dynamic-tool-loader.ts` to use Railway -3. Test locally -4. Deploy to Railway + Vercel -5. Celebrate dynamic tool loading! 🎉 diff --git a/README.md b/README.md index 484cd9f..6b2454d 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,59 @@ -# TPMJS Monorepo +# TPMJS -[![CI](https://github.com/YOUR_ORG/tpmjs/actions/workflows/ci.yml/badge.svg)](https://github.com/YOUR_ORG/tpmjs/actions/workflows/ci.yml) +[![CI](https://github.com/tpmjs/tpmjs/actions/workflows/ci.yml/badge.svg)](https://github.com/tpmjs/tpmjs/actions/workflows/ci.yml) -Tool Package Manager for AI Agents - A Turborepo monorepo with strict TypeScript, Next.js 16, and best practices. +**TPMJS is a registry that lets AI agents discover and use npm packages as tools at runtime.** -## Structure +Instead of manually importing and configuring tools, agents search by description and load what they need on-demand. Publish your npm package with the `tpmjs-tool` keyword—it appears on [tpmjs.com](https://tpmjs.com) within 15 minutes. + +## Why TPMJS? + +- **No config files** - Agents discover tools by describing what they need +- **Always up-to-date** - Tools load from npm at runtime, no manual updates +- **Works with any agent** - Compatible with Vercel AI SDK, LangChain, OpenAI, Claude, etc. +- **Publish once** - Add one keyword to package.json, publish to npm, done + +## Quick Start + +**For AI agent developers:** +```bash +npm install @tpmjs/sdk +``` + +```typescript +import { searchRegistry, executeRegistry } from '@tpmjs/sdk'; + +// Find tools by description +const tools = await searchRegistry({ query: 'parse PDF documents' }); + +// Execute a tool +const result = await executeRegistry({ + toolId: 'pdf-parser/extractText', + input: { url: 'https://example.com/doc.pdf' } +}); +``` + +**For tool publishers:** +```bash +npx @tpmjs/create-basic-tools +``` + +Or add manually to your package.json: +```json +{ + "keywords": ["tpmjs-tool"], + "tpmjs": { + "category": "text-analysis", + "description": "What your tool does" + } +} +``` + +See [HOW_TO_PUBLISH_A_TOOL.md](./HOW_TO_PUBLISH_A_TOOL.md) for the full guide. + +--- + +## Monorepo Structure ``` apps/ diff --git a/STREAMING_EMPTY_RESPONSE.md b/STREAMING_EMPTY_RESPONSE.md deleted file mode 100644 index 98383d7..0000000 --- a/STREAMING_EMPTY_RESPONSE.md +++ /dev/null @@ -1,363 +0,0 @@ -# AI SDK v6 Streaming Empty Response Issue - -## Problem - -Using AI SDK v6 Beta with OpenAI and `streamText()`, the API route returns a 200 OK response, but the streamed response body is **completely empty** when tools are involved. - -- **Normal chat** (without tool calls): Works fine, streams text back -- **Tool calls** (when user asks "say hello world"): Returns empty response body, no error messages - -## Environment - -- **AI SDK Version**: `ai@6.0.0-beta.124` -- **OpenAI Provider**: `@ai-sdk/openai@3.0.0-beta.74` -- **OpenAI Library**: `openai@^6.9.1` -- **Next.js Version**: `next@^16.0.4` (App Router) -- **Runtime**: Node.js (`runtime = 'nodejs'`) -- **Model**: `gpt-4o-mini` - -## API Route Implementation - -Located at: `apps/playground/src/app/api/chat/route.ts` - -```typescript -import { createOpenAI } from '@ai-sdk/openai'; -import { streamText, type CoreMessage, tool, jsonSchema } from 'ai'; -import { type NextRequest } from 'next/server'; -import { z } from 'zod'; -import { env } from '~/env'; - -export const runtime = 'nodejs'; -export const dynamic = 'force-dynamic'; -export const maxDuration = 60; - -// Initialize OpenAI provider -const openai = createOpenAI({ - apiKey: env.OPENAI_API_KEY, -}); - -// Request schema -const RequestSchema = z.object({ - messages: z.array( - z.object({ - role: z.enum(['user', 'assistant', 'system']), - content: z.string(), - }) - ), -}); - -// Simple inline test tool to verify streaming works -const testHelloTool = tool({ - description: 'Returns a simple hello world greeting', - inputSchema: jsonSchema<{ includeTimestamp?: boolean }>({ - type: 'object', - properties: { - includeTimestamp: { - type: 'boolean', - description: 'Whether to include a timestamp', - }, - }, - additionalProperties: false, - }), - async execute({ includeTimestamp = true }) { - const response: any = { message: 'Hello, World!' }; - if (includeTimestamp) { - response.timestamp = new Date().toISOString(); - } - return response; - }, -}); - -/** - * POST /api/chat - * Chat with AI agent that can execute TPMJS tools - */ -export async function POST(request: NextRequest) { - try { - const body = await request.json(); - const { messages } = RequestSchema.parse(body); - - // Use simple inline tool for testing - const tools = { - testHello: testHelloTool, - }; - - // Create system prompt listing available tools - const toolsList = Object.keys(tools) - .map((name) => `- ${name}: ${tools[name]?.description}`) - .join('\n'); - - const systemMessage: CoreMessage = { - role: 'system', - content: `You are a helpful AI assistant that can use TPMJS tools to help users. - -Available tools: -${toolsList} - -Call tools as needed to answer user questions. When a user asks to say hello world or for a greeting, use the testHello tool.`, - }; - - // Stream the response - const result = streamText({ - model: openai('gpt-4o-mini'), - messages: [systemMessage, ...messages], - tools, - }); - - // Return the stream as SSE - return result.toTextStreamResponse(); - } catch (error) { - console.error('Chat API error:', error); - - if (error instanceof z.ZodError) { - return new Response( - JSON.stringify({ - success: false, - error: 'Invalid request format', - details: error.issues, - }), - { - status: 400, - headers: { 'Content-Type': 'application/json' }, - } - ); - } - - return new Response( - JSON.stringify({ - success: false, - error: error instanceof Error ? error.message : 'Unknown error', - }), - { - status: 500, - headers: { 'Content-Type': 'application/json' }, - } - ); - } -} -``` - -## Client-Side Hook - -Located at: `apps/playground/src/hooks/useChat.ts` - -```typescript -'use client'; - -import { useCallback, useState } from 'react'; - -export interface ChatMessage { - id: string; - role: 'user' | 'assistant' | 'system'; - content: string; - timestamp: Date; -} - -export function useChat() { - const [messages, setMessages] = useState([]); - const [isLoading, setIsLoading] = useState(false); - const [error, setError] = useState(null); - - const sendMessage = useCallback(async (content: string) => { - if (!content.trim()) return; - - // Add user message immediately - const userMessage: ChatMessage = { - id: crypto.randomUUID(), - role: 'user', - content, - timestamp: new Date(), - }; - - setMessages((prev) => [...prev, userMessage]); - setIsLoading(true); - setError(null); - - try { - // Create assistant message placeholder - const assistantMessageId = crypto.randomUUID(); - const assistantMessage: ChatMessage = { - id: assistantMessageId, - role: 'assistant', - content: '', - timestamp: new Date(), - }; - - setMessages((prev) => [...prev, assistantMessage]); - - // Send request to API - const response = await fetch('/api/chat', { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - }, - body: JSON.stringify({ - messages: [...messages, userMessage].map((m) => ({ - role: m.role, - content: m.content, - })), - }), - }); - - if (!response.ok) { - throw new Error(`HTTP error! status: ${response.status}`); - } - - // Read the streaming response - const reader = response.body?.getReader(); - const decoder = new TextDecoder(); - - if (!reader) { - throw new Error('Response body is null'); - } - - let accumulatedContent = ''; - - while (true) { - const { done, value } = await reader.read(); - - if (done) { - break; - } - - // Decode the chunk - const chunk = decoder.decode(value, { stream: true }); - accumulatedContent += chunk; - - // Update the assistant message with accumulated content - setMessages((prev) => - prev.map((m) => - m.id === assistantMessageId - ? { ...m, content: accumulatedContent } - : m - ) - ); - } - } catch (err) { - console.error('Error sending message:', err); - setError(err instanceof Error ? err.message : 'Failed to send message'); - } finally { - setIsLoading(false); - } - }, [messages]); - - const clearChat = useCallback(() => { - setMessages([]); - setError(null); - }, []); - - return { - messages, - isLoading, - error, - sendMessage, - clearChat, - }; -} -``` - -## Observed Behavior - -### Working Case (Normal Chat) -- User types: "hi" -- API response: 200 OK -- Response body: Streams text chunks successfully -- UI shows: "Hi! How can I help you today?" - -### Broken Case (Tool Call) -- User types: "say hello world" -- API response: 200 OK ✅ -- Response body: **EMPTY** ❌ (no chunks, no data, nothing) -- UI shows: Empty message bubble -- Console: No errors logged - -## HTTP Response Details - -``` -Request Method: POST -Status Code: 200 OK -URL: http://localhost:3001/api/chat -Content-Type: text/plain; charset=utf-8 -Transfer-Encoding: chunked -``` - -The response headers look correct for a streaming response, but the body is completely empty. - -## What We've Tried - -1. ✅ Fixed OpenAI schema validation error (was `type: "None"`, now uses proper JSON Schema) -2. ✅ Using `tool()` and `jsonSchema()` from AI SDK -3. ✅ Simplified to a single inline test tool -4. ✅ Tool executes without errors (no schema validation issues) -5. ✅ Normal chat works fine (proves streaming infrastructure is correct) - -## Questions - -1. **Is `toTextStreamResponse()` the correct method for streaming with tools in AI SDK v6?** - - Should we use a different method like `toDataStreamResponse()` for tool calls? - -2. **Are we constructing the messages array correctly?** - - We're sending `{ role: 'user' | 'assistant' | 'system', content: string }[]` - - Do we need to include tool call messages or tool result messages? - -3. **Does AI SDK v6 require a specific message format for tool calls?** - - Should we be including `toolInvocations` or `tool_calls` in the message history? - - Are we missing required fields in the `CoreMessage` type? - -4. **Is the client-side streaming reader correct?** - - We're reading chunks with `response.body.getReader()` - - Should we be parsing SSE events differently for tool calls? - -5. **Does `streamText()` with tools require `maxSteps` parameter?** - - Do we need to set `maxSteps: 5` to allow multi-step reasoning? - -6. **Are we handling the conversation history correctly?** - - We're sending all previous messages on each request - - Should we be including assistant messages with tool call results? - -## AI SDK v6 Documentation References - -We're following these patterns from the official docs: - -- [streamText() API](https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text) -- [tool() API](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool) -- [Tool Calling Guide](https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling) - -But we might be missing something specific about: -- How to handle streaming when tools are executed -- What response format tool calls produce -- How to parse the stream when tools are involved - -## Suspected Issue - -**The message format might be wrong.** We're sending: - -```typescript -const systemMessage: CoreMessage = { - role: 'system', - content: `You are a helpful AI assistant...`, -}; - -const result = streamText({ - model: openai('gpt-4o-mini'), - messages: [systemMessage, ...messages], - tools, -}); -``` - -But `CoreMessage` might need additional fields when tools are involved, or we might need to handle tool call results differently in the conversation history. - -## What We Need - -1. Correct message format for `streamText()` with tools -2. How to properly stream responses that include tool calls -3. Whether we need different client-side parsing for tool call streams -4. Example of a working Next.js API route using AI SDK v6 with `streamText()` and tools - -## Repo Context - -- Monorepo using Turborepo + pnpm workspaces -- Next.js 16 App Router with Turbopack -- TypeScript strict mode -- All UI components from internal `@tpmjs/ui` package -- Tools are imported from workspace package `@tpmjs/hello` diff --git a/USECHAT_INPUT_UNDEFINED.md b/USECHAT_INPUT_UNDEFINED.md deleted file mode 100644 index 861026c..0000000 --- a/USECHAT_INPUT_UNDEFINED.md +++ /dev/null @@ -1,400 +0,0 @@ -# useChat Hook Returns Undefined Input Property - -## Problem - -Using `@ai-sdk/react`'s `useChat` hook, the `input` property is returning `undefined`, causing the application to crash when trying to call `.trim()` on it. - -## Error - -``` -TypeError: Cannot read properties of undefined (reading 'trim') - -at ChatInput (src/components/chat/ChatInput.tsx:41:48) -``` - -**Code that fails:** -```typescript -