engineering/tpmjs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1.9k

chore: prepare for Hacker News launch

Browse source 
- Add MIT LICENSE file
- Fix YOUR_ORG placeholders in README and DEPLOYMENT docs
- Fix Node.js version mismatch in release workflow (21 → 22)
- Delete 17 internal debug/development docs
- Rewrite hero section for clarity (explain what TPMJS is in seconds)
- Add "What is TPMJS?" section to landing page
- Fix hardcoded emails to hello@tpmjs.com
- Fix hardcoded dates to December 2024
- Add package metadata (author, license, repository) to all published packages
- Clean up AI-sounding language throughout
- Add comprehensive LAUNCH_REVIEW.md with checklist

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Ajax Davis 2025-12-28 11:46:59 +10:00
parent 81efba0de5
commit 8b532ffa79
40 changed files with 569 additions and 39423 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.github/workflows/release.yml vendored
View file

@ -23,7 +23,7 @@ jobs:
- uses: actions/setup-node@v4
with:
node-version: 21
node-version: 22
cache: 'pnpm'
registry-url: 'https://registry.npmjs.org'

275
2025-BEST-PRACTICES.md
View file

@ -1,275 +0,0 @@
# 2025 Best Practices for TPMJS Monorepo
This document outlines recommendations to make TPMJS a cutting-edge 2025 monorepo optimized for both human and agentic development (Claude Code, Cursor, etc.).
## High-Impact Additions
### 1. Agent-First Documentation
```
packages/docs/
├── architecture-decisions/ # ADRs in markdown
├── patterns/ # Common patterns with examples
├── schemas/ # JSON schemas for all data structures
└── examples/ # Working code examples per feature
```
**Why:** Claude Code and other agents work better with:
- Explicit decision documentation (ADRs)
- Pattern libraries showing "the right way"
- Machine-readable schemas
- Real working examples to reference
### 2. Automated Testing Pyramid
```bash
# Add to package.json scripts
"test:unit": "vitest" # ✅ Already have this
"test:integration": "vitest -c vitest.integration.config.ts" # Add
"test:e2e": "playwright test" # Add
"test:visual": "playwright test --grep @visual" # Add
"test:contracts": "pactum" # Add for API testing
```
**Packages to add:**
- `@playwright/test` - E2E testing
- `@playwright/experimental-ct-react` - Component testing
- `pactum` or `msw` integration tests (you have mocks setup)
- `chromatic` or `percy` - Visual regression
### 3. Type Coverage & Quality Gates
```json
// Add to root package.json
{
"scripts": {
"type-check": "tsc --noEmit",
"type-coverage": "type-coverage --at-least 95",
"find-deadcode": "knip",
"check-architecture": "depcruiser --validate"
}
}
```
**Add packages:**
- `type-coverage` - Ensure no implicit `any`
- `knip` - Find unused files/exports/dependencies
- `dependency-cruiser` - Enforce architecture rules
- `@total-typescript/ts-reset` - Better built-in types
### 4. Development Containers
```json
// .devcontainer/devcontainer.json
{
"name": "TPMJS Dev",
"dockerComposeFile": "docker-compose.yml",
"service": "dev",
"features": {
"ghcr.io/devcontainers/features/node:1": {},
"ghcr.io/devcontainers-contrib/features/pnpm:2": {}
},
"customizations": {
"vscode": {
"extensions": [
"biomejs.biome",
"bradlc.vscode-tailwindcss",
"lokalise.i18n-ally"
]
}
}
}
```
**Why:** Agents like Claude Code work better when environment is reproducible. This also helps human developers.
### 5. Code Generation & Scaffolding
```typescript
// packages/cli/ - Internal dev tool
import { scaffold } from '@tpmjs/cli';
// Commands:
pnpm gen:component ButtonGroup
pnpm gen:package @tpmjs/new-package
pnpm gen:app marketing-site
```
**Create:**
- `plop` or `hygen` templates
- Component scaffolding (with tests, stories, exports)
- Package scaffolding (with tsconfig, package.json, exports)
- Consistent file structure generation
**Why:** Agents can use these commands to create new code following your exact patterns.
### 6. Enhanced Strict Mode TypeScript
```json
// packages/tsconfig/base.json - Add these
{
"compilerOptions": {
"exactOptionalPropertyTypes": true,
"noUncheckedIndexedAccess": true,
"noPropertyAccessFromIndexSignature": true,
"allowUnusedLabels": false,
"allowUnreachableCode": false,
"noImplicitOverride": true
}
}
```
### 7. Bundle Analysis & Performance
```json
{
"scripts": {
"analyze": "turbo run build --filter=@tpmjs/web -- --analyze",
"lighthouse": "lhci autorun",
"bundle-size": "size-limit"
}
}
```
**Add:**
- `@next/bundle-analyzer`
- `@lhci/cli` - Lighthouse CI
- `size-limit` - Bundle size tracking in CI
### 8. Smart Dependency Management
```json
// .github/renovate.json
{
"extends": ["config:base"],
"packageRules": [
{
"matchPackagePatterns": ["*"],
"matchUpdateTypes": ["minor", "patch"],
"groupName": "all non-major dependencies",
"groupSlug": "all-minor-patch"
}
]
}
```
**Use:** Renovate or Dependabot with auto-merge for passing tests
### 9. API Documentation Generation
```bash
pnpm add -D -w typedoc typedoc-plugin-markdown
```
Auto-generate API docs from TSDoc comments that both humans and agents can read.
### 10. Schema-First Development
```typescript
// packages/schemas/ - Central schema definitions
export * from './tool-schema';
export * from './registry-api-schema';
export * from './event-schema';
// Use Zod for runtime + type generation
// Agents can read schemas to understand contracts
```
## Monorepo-Specific Improvements
### 11. Better Local Development
```typescript
// turbo.json
{
"pipeline": {
"dev": {
"cache": false,
"persistent": true,
"dependsOn": ["^build"]
},
"build": {
"dependsOn": ["^build"],
"outputs": ["dist/**", ".next/**"]
}
}
}
```
### 12. Workspace Protocols & Constraints
```yaml
# .pnpm-workspace.yaml
packages:
- 'apps/*'
- 'packages/*'
# Add constraints
pnpm-workspace-constraints:
dependencies:
'@tpmjs/ui': 'workspace:*'
'@tpmjs/utils': 'workspace:*'
```
## Recommended Final Structure
```
.
├── .devcontainer/ # Dev containers config
├── .github/
│ ├── workflows/ # CI/CD
│ └── renovate.json # Dependency automation
├── apps/
│ └── web/
├── packages/
│ ├── cli/ # ⭐ NEW: Dev tooling
│ ├── schemas/ # ⭐ NEW: Central schemas
│ └── ...existing
├── docs/
│ ├── adr/ # ⭐ NEW: Architecture decisions
│ ├── patterns/ # ⭐ NEW: Code patterns
│ └── examples/ # ⭐ NEW: Working examples
├── scripts/
│ ├── scaffold.ts # ⭐ NEW: Code generation
│ └── validate-deps.ts # ⭐ NEW: Architecture validation
├── playwright.config.ts # ⭐ NEW: E2E testing
├── .lighthouserc.json # ⭐ NEW: Performance
└── knip.json # ⭐ NEW: Dead code detection
```
## Priority Order for Implementation
### Phase 1 (Foundation)
1. **Knip + type-coverage** - Catch issues early
2. **Code generation scripts** - Ensure consistency
3. **ADR documentation structure** - Decision tracking
### Phase 2 (Quality)
4. **E2E testing with Playwright** - Full user flow coverage
5. **Bundle analysis + performance budgets** - Keep app fast
6. **Stricter TypeScript settings** - Catch more bugs at compile time
### Phase 3 (DX)
7. **Dev containers** - Reproducible environments
8. **API documentation generation** - Auto-generated from code
9. **Renovate automation** - Keep dependencies fresh
## Benefits for Agent-Driven Development
1. **Explicit Patterns** - Agents can reference documented patterns instead of guessing
2. **Code Generation** - Consistent scaffolding commands agents can use
3. **Machine-Readable Schemas** - JSON schemas help agents understand data structures
4. **Quality Gates** - Automated checks catch agent mistakes early
5. **Working Examples** - Agents can copy-paste-adapt proven patterns
6. **Architecture Enforcement** - Dependency rules prevent agents from creating invalid imports
## Next Steps
Start with the highest ROI items:
1. Install Knip to find dead code
2. Set up code generation for components/packages
3. Create docs/patterns/ with common examples
4. Add stricter TypeScript compiler options
5. Set up Playwright for E2E testing
These changes will make the codebase more maintainable and significantly improve the experience of working with AI coding agents.

932
API_ROUTES_TIMEOUT_INVESTIGATION.md
View file

@ -1,932 +0,0 @@
# API Routes Timeout Issue - Complete Investigation Report
## Problem Statement
API routes deployed to Vercel are timing out with no response. The Next.js application pages work perfectly, but all API endpoints at `/api/*` return timeouts or "Redirecting..." messages.
**Affected URLs:**
- `https://tpmjs.com/api/health` - Returns "Redirecting..."
- `https://tpmjs.com/api/tools` - Returns "Redirecting..."
- `https://tpmjs-1chh44d1u-tpmjs.vercel.app/api/health` - Timeouts (exit code 28)
- `https://tpmjs-1chh44d1u-tpmjs.vercel.app/api/tools` - Timeouts (exit code 28)
**Working:**
- All page routes work correctly (e.g., `/`, `/tool/[slug]`)
- UI navigation and client-side routing function normally
- Local development API routes work perfectly
## Environment Details
### Project Structure
- **Monorepo:** Turborepo setup with pnpm workspaces
- **Framework:** Next.js 16.0.4 (App Router)
- **Node Version:** 24.x (on Vercel)
- **Deployment Platform:** Vercel
- **Custom Domains:** tpmjs.com, www.tpmjs.com
### Repository Structure
```
tpmjs/
├── apps/
│ └── web/ # Next.js 16 App Router application
│ ├── src/
│ │ └── app/
│ │ ├── api/
│ │ │ ├── health/route.ts
│ │ │ ├── stats/route.ts
│ │ │ ├── tools/
│ │ │ │ ├── route.ts
│ │ │ │ ├── [id]/route.ts
│ │ │ │ ├── [slug]/route.ts
│ │ │ │ └── validate/route.ts
│ │ │ └── sync/
│ │ │ ├── changes/route.ts
│ │ │ ├── keyword/route.ts
│ │ │ └── metrics/route.ts
│ │ ├── page.tsx
│ │ └── tool/[slug]/page.tsx
│ ├── next.config.ts
│ └── vercel.json
├── packages/
│ ├── db/ # Prisma client
│ ├── types/ # Shared TypeScript types
│ ├── utils/ # Utility functions
│ ├── env/ # Environment validation
│ └── ui/ # React component library
├── vercel.json # Root Vercel configuration
└── turbo.json
```
## API Route Examples
### `/apps/web/src/app/api/health/route.ts`
```typescript
import { NextResponse } from 'next/server';
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const maxDuration = 60;
/**
* GET /api/health
* Simple health check endpoint that doesn't touch the database
*/
export async function GET() {
return NextResponse.json({
status: 'ok',
timestamp: new Date().toISOString(),
env: {
hasDatabase: !!process.env.DATABASE_URL,
nodeEnv: process.env.NODE_ENV,
},
});
}
```
### `/apps/web/src/app/api/tools/route.ts`
```typescript
import { NextResponse } from 'next/server';
import { prisma } from '@tpmjs/db/client';
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export async function GET(request: Request) {
// ... query string parsing
const [tools, totalCount] = await Promise.all([
prisma.tool.findMany({
where,
orderBy: [
{ qualityScore: 'desc' },
{ npmDownloadsLastMonth: 'desc' },
{ createdAt: 'desc' },
],
take: limit,
skip: offset,
}),
prisma.tool.count({ where }),
]);
return NextResponse.json({
data: tools,
pagination: {
page,
limit,
total: totalCount,
totalPages: Math.ceil(totalCount / limit),
},
});
}
```
## Configuration Files
### `/apps/web/next.config.ts` (Current)
```typescript
import type { NextConfig } from 'next';
const nextConfig: NextConfig = {
transpilePackages: ['@tpmjs/ui', '@tpmjs/utils', '@tpmjs/db', '@tpmjs/types', '@tpmjs/env'],
reactStrictMode: true,
};
export default nextConfig;
```
### `/apps/web/vercel.json` (Current)
```json
{
"$schema": "https://openapi.vercel.sh/vercel.json",
"buildCommand": "cd ../.. && pnpm --filter=@tpmjs/web build",
"installCommand": "pnpm install"
}
```
### `/vercel.json` (Root)
```json
{
"$schema": "https://openapi.vercel.sh/vercel.json",
"git": {
"deploymentEnabled": {
"main": true
}
},
"github": {
"silent": false,
"autoJobCancelation": true
},
"crons": [
{
"path": "/api/sync/changes",
"schedule": "*/2 * * * *"
},
{
"path": "/api/sync/keyword",
"schedule": "*/15 * * * *"
},
{
"path": "/api/sync/metrics",
"schedule": "0 * * * *"
}
]
}
```
## Local Build Verification
### Local Build Output Structure
```bash
$ ls -R /Users/ajaxdavis/repos/tpmjs/tpmjs/apps/web/.next/server/app/api/
health/
stats/
sync/
tools/
/apps/web/.next/server/app/api/health:
route
route.js
route.js.map
route.js.nft.json
route_client-reference-manifest.js
/apps/web/.next/server/app/api/stats:
route
route.js
route.js.map
route.js.nft.json
route_client-reference-manifest.js
/apps/web/.next/server/app/api/tools:
[id]/
[slug]/
validate/
route
route.js
route.js.map
route.js.nft.json
route_client-reference-manifest.js
```
### Routes Manifest Confirmation
```bash
$ cat /apps/web/.next/routes-manifest.json | jq '.staticRoutes[] | select(.page | contains("api"))'
{
"page": "/api/health",
"regex": "^/api/health(?:/)?$",
"routeKeys": {},
"namedRegex": "^/api/health(?:/)?$"
}
{
"page": "/api/stats",
"regex": "^/api/stats(?:/)?$",
"routeKeys": {},
"namedRegex": "^/api/stats(?:/)?$"
}
{
"page": "/api/sync/changes",
"regex": "^/api/sync/changes(?:/)?$",
"routeKeys": {},
"namedRegex": "^/api/sync/changes(?:/)?$"
}
{
"page": "/api/sync/keyword",
"regex": "^/api/sync/keyword(?:/)?$",
"routeKeys": {},
"namedRegex": "^/api/sync/keyword(?:/)?$"
}
{
"page": "/api/sync/metrics",
"regex": "^/api/sync/metrics(?:/)?$",
"routeKeys": {},
"namedRegex": "^/api/sync/metrics(?:/)?$"
}
```
### Node File Trace (NFT) Verification
```bash
$ cat /apps/web/.next/server/app/api/health/route.js.nft.json
{
"version": 1,
"files": [
"../../../../../../../node_modules/.pnpm/next@16.0.4_@babel+core@7.28.5_react-dom@19.2.0_react@19.2.0__react@19.2.0/node_modules/next/dist/client/components/app-router-headers.js",
"../../../../../../../node_modules/.pnpm/next@16.0.4_@babel+core@7.28.5_react-dom@19.2.0_react@19.2.0__react@19.2.0/node_modules/next/dist/compiled/@opentelemetry/api/index.js",
// ... many more dependencies
]
}
```
**Conclusion:** API routes build correctly locally with all dependencies properly traced.
## Vercel Deployment Analysis
### Deployment Inspection Output
```bash
$ vercel inspect https://tpmjs-1chh44d1u-tpmjs.vercel.app
General
id dpl_2FHyiTWtBZzvT8EcohwYEZmdb8rb
name tpmjs-web
target production
status ● Ready
url https://tpmjs-1chh44d1u-tpmjs.vercel.app
created Fri Nov 28 2025 20:44:03 GMT+1000
Aliases
╶ https://www.tpmjs.com
╶ https://tpmjs-web.vercel.app
╶ https://tpmjs-web-tpmjs.vercel.app
╶ https://tpmjs-web-git-main-tpmjs.vercel.app
╶ https://tpmjs.com
Builds
┌ . [0ms]
├── λ tool/[slug] (562.92KB) [iad1]
├── λ tool/[slug].rsc (562.92KB) [iad1]
├── λ _global-error (642.48KB) [iad1]
├── λ _global-error.rsc (642.48KB) [iad1]
├── λ _global-error.segments/__PAGE__.segment.rsc (642.48KB) [iad1]
└── 56 output items hidden
```
**CRITICAL FINDING:** No API routes are listed in the build output. Only pages (`tool/[slug]`, `_global-error`, etc.) appear as serverless functions (`λ`).
Expected API routes that should appear:
- `λ api/health`
- `λ api/tools`
- `λ api/tools/[id]`
- `λ api/tools/[slug]`
- `λ api/sync/changes`
- etc.
### Testing Results
```bash
# Direct Vercel URL - Timeouts
$ curl -s -m 10 https://tpmjs-1chh44d1u-tpmjs.vercel.app/api/health
# Exit code 28 (timeout)
# Custom Domain - Returns "Redirecting..."
$ curl -s -m 10 https://tpmjs.com/api/health
Redirecting...
# Custom Domain - Returns "Redirecting..."
$ curl -s -m 10 https://tpmjs.com/api/tools
Redirecting...
# Check redirect headers
$ curl -I https://tpmjs.com
HTTP/2 307
cache-control: public, max-age=0, must-revalidate
content-type: text/plain
date: Fri, 28 Nov 2025 10:38:12 GMT
location: https://www.tpmjs.com/
server: Vercel
```
## Vercel Configuration Details
### User-Confirmed Settings
- **Root Directory:** `apps/web` (set in Vercel dashboard)
- **DATABASE_URL:** Configured in Vercel environment variables (Production)
- **Framework Preset:** (Unknown - needs verification)
- **Build Output Directory:** (Unknown - using default `.next`)
### Project List
```bash
$ vercel project ls | grep -i tpmjs
tpmjs -- 16h 24.x
v0-tool-registry-page https://tpmjs.com 3d 22.x
```
**NOTE:** Two projects exist:
1. `tpmjs` - Current project (Node 24.x)
2. `v0-tool-registry-page` - Also has tpmjs.com domain (Node 22.x)
This could indicate a domain routing conflict or outdated project.
## Investigation Timeline & Attempts
### Attempt 1: Remove Root-Level Redirects
**Hypothesis:** The redirect in `/vercel.json` was intercepting API requests.
**Original `/vercel.json`:**
```json
{
"redirects": [
{
"source": "/:path*",
"has": [
{
"type": "host",
"value": "www.tpmjs.com"
}
],
"destination": "https://tpmjs.com/:path*",
"permanent": true
}
]
}
```
**Action:** Removed the `redirects` array from root `/vercel.json`.
**Result:** ❌ API routes still timeout. Redirect rule was not the root cause.
**Commit:** `9be3af7 fix(routing): move www redirect from vercel.json to Next.js config`
### Attempt 2: Move Redirects to Next.js Config
**Hypothesis:** Next.js should handle redirects after routing.
**Action:** Added `async redirects()` to `apps/web/next.config.ts`:
```typescript
async redirects() {
return [
{
source: '/:path*',
has: [
{
type: 'host',
value: 'www.tpmjs.com',
},
],
destination: 'https://tpmjs.com/:path*',
permanent: true,
},
];
}
```
**Result:** ❌ API routes returned "Redirecting..." instead of executing. Next.js `async redirects()` applies to ALL routes including API routes.
**Commit:** `9be3af7 fix(routing): move www redirect from vercel.json to Next.js config`
### Attempt 3: Exclude API Routes from Redirect
**Hypothesis:** Use regex to exclude `/api/*` from redirects.
**Action:** Modified redirect pattern:
```typescript
async redirects() {
return [
{
source: '/((?!api).*)', // Negative lookahead to exclude /api/*
has: [
{
type: 'host',
value: 'www.tpmjs.com',
},
],
destination: 'https://tpmjs.com/$1',
permanent: true,
},
];
}
```
**Result:** ❌ API routes back to timing out (not redirecting anymore, but still not working).
**Commit:** `c42bfb6 fix(redirects): exclude API routes from www redirect`
### Attempt 4: Remove All Redirects
**Hypothesis:** Eliminate redirect loop causing ERR_TOO_MANY_REDIRECTS.
**Action:** Removed `async redirects()` entirely from `next.config.ts`.
**Result:** ✅ Redirect loop fixed. ❌ API routes still timeout.
**Commit:** `b93cd42 fix: remove redirects to resolve redirect loop`
### Attempt 5: Add Vercel Functions Configuration
**Hypothesis:** Vercel needs explicit configuration to detect API routes.
**Action:** Added to `apps/web/vercel.json`:
```json
{
"functions": {
"app/api/**/*.ts": {
"maxDuration": 60
}
}
}
```
**Result:** ❌ No change. API routes still timeout.
**Commit:** `748ca4d fix(api): configure Vercel functions for API routes with maxDuration`
### Attempt 6: Add maxDuration to Route Files
**Hypothesis:** Export configuration directly in route handlers.
**Action:** Added to `apps/web/src/app/api/health/route.ts`:
```typescript
export const maxDuration = 60;
```
**Result:** ❌ No change. API routes still timeout.
**Commit:** `8281f8f fix(build): disable Turbopack for Vercel deployment`
### Attempt 7: Disable Turbopack
**Hypothesis:** Turbopack (Next.js 16 default) has compatibility issues with Vercel.
**Action:** Added `--webpack` flag to build command:
```json
{
"buildCommand": "cd ../.. && turbo build --filter=@tpmjs/web -- --webpack"
}
```
**Result:** ❌ Build failed completely. Invalid flag syntax.
**Commit:** `8281f8f fix(build): disable Turbopack for Vercel deployment`
### Attempt 8: Simplify Build Command
**Hypothesis:** Use direct pnpm build instead of Turbo wrapper.
**Action:** Changed to:
```json
{
"buildCommand": "cd ../.. && pnpm --filter=@tpmjs/web build"
}
```
**Result:** ⏳ Pending deployment test.
**Commit:** `065196d fix(build): simplify Vercel build command`
## Root Cause Analysis
### What We Know FOR SURE
1. ✅ **API routes build correctly locally**
- All 9 API routes compile to `.next/server/app/api/`
- NFT (Node File Trace) files are generated with proper dependencies
- Routes manifest includes all API routes
2. ✅ **Next.js configuration is correct**
- `export const runtime = 'nodejs'` set correctly
- `export const dynamic = 'force-dynamic'` set correctly
- `transpilePackages` includes all workspace packages
3. ✅ **Pages deploy and work perfectly**
- `/tool/[slug]` renders correctly
- Homepage loads
- Client-side navigation works
4. ❌ **API routes are NOT deployed as serverless functions**
- `vercel inspect` shows NO API routes in build output
- Only pages appear as `λ` (lambda) functions
- This is the PRIMARY issue
5. ❌ **Direct Vercel URLs timeout**
- Not just a custom domain issue
- Affects `*.vercel.app` URLs
- Exit code 28 (timeout) - no response at all
6. ❌ **Custom domain shows "Redirecting..."**
- Even with all redirects removed from config
- Suggests a redirect at Vercel platform level OR DNS level
- Could be from the `v0-tool-registry-page` project conflict
### Possible Root Causes
#### Theory 1: Vercel Project Misconfiguration
**Likelihood:** HIGH
**Evidence:**
- Two projects with same domain (`tpmjs` and `v0-tool-registry-page`)
- Framework Preset might not be set to "Next.js"
- Root Directory is `apps/web` but Vercel might not be detecting Next.js properly
**What to Check:**
1. Vercel Dashboard → Project Settings → General
- Framework Preset: Should be "Next.js"
- Root Directory: Should be "apps/web"
- Build Command: Should match vercel.json
- Output Directory: Should be blank (default `.next`)
2. Vercel Dashboard → Domains
- Check if both projects have tpmjs.com
- Remove domain from `v0-tool-registry-page` if present
3. Vercel Dashboard → Deployments → Build Logs
- Search for "API" or "route"
- Look for errors about missing functions
- Check if Next.js is detected correctly
#### Theory 2: Monorepo Detection Issue
**Likelihood:** MEDIUM
**Evidence:**
- Build command uses `cd ../.. && pnpm --filter=@tpmjs/web build`
- Vercel might not be correctly detecting workspace structure
- `transpilePackages` includes workspace packages
**What to Check:**
1. Build logs for workspace resolution errors
2. Check if `node_modules` is being created in correct location
3. Verify pnpm workspace configuration
**Potential Fix:**
Try setting `installCommand` to:
```json
{
"installCommand": "pnpm install --shamefully-hoist"
}
```
#### Theory 3: Next.js 16 + Vercel Incompatibility
**Likelihood:** MEDIUM
**Evidence:**
- Next.js 16 released recently (November 2024)
- Turbopack is default (might have Vercel issues)
- App Router API routes behave differently than Pages Router
**What to Check:**
1. Vercel build logs for Next.js version detection
2. Any warnings about incompatible features
3. Check Vercel's Next.js 16 support status
**Potential Fix:**
Downgrade to Next.js 15.x temporarily to test:
```json
{
"dependencies": {
"next": "^15.0.0"
}
}
```
#### Theory 4: Environment Variable Issue
**Likelihood:** LOW
**Evidence:**
- DATABASE_URL is configured
- Pages work (they might not need env vars)
- API routes use Prisma (requires DATABASE_URL)
**What to Check:**
1. Vercel Dashboard → Settings → Environment Variables
- Verify DATABASE_URL is set for Production
- Verify it's not blocked or empty
- Check if other vars are needed
2. Build logs for Prisma generation errors
**Potential Fix:**
None - user confirmed DATABASE_URL is set.
#### Theory 5: Build Output Issue
**Likelihood:** MEDIUM-HIGH
**Evidence:**
- `vercel inspect` doesn't show API routes
- Only pages are listed as functions
- Build completes successfully (38-41 seconds)
**What to Check:**
1. Build logs: Does Next.js report building API routes?
- Look for "Route (app)" or "λ" indicators for API routes
- Compare to local build output
2. Check if Vercel is using correct build output structure
- App Router uses `.next/server/app/`
- Pages Router uses `.next/server/pages/`
**Potential Fix:**
Try forcing Vercel to recognize the build:
```json
{
"builds": [
{
"src": "package.json",
"use": "@vercel/next"
}
]
}
```
(Note: `builds` is legacy, modern Next.js should auto-detect)
## Recommended Next Steps
### Immediate Actions (High Priority)
1. **Check Vercel Project Settings**
- Go to Vercel Dashboard → tpmjs-web project
- Verify Framework Preset is "Next.js"
- Verify Root Directory is `apps/web`
- Screenshot settings for reference
2. **Review Build Logs**
- Go to latest deployment
- Download complete build logs
- Search for:
- "Route (app)" - should show API routes
- "λ" - should show API functions
- "api" - any mentions
- Errors or warnings
3. **Check Domain Configuration**
- Verify only ONE project has tpmjs.com domain
- Remove domain from `v0-tool-registry-page` project if present
- Check DNS settings aren't redirecting
4. **Test Simple API Route**
- Create minimal API route:
```typescript
// apps/web/src/app/api/test/route.ts
export async function GET() {
return new Response('Hello from API', { status: 200 });
}
```
- Deploy and test
- If this doesn't work, confirms platform issue
### Investigation Actions (Medium Priority)
5. **Compare Working vs Non-Working**
- Find a deployment where pages DO work
- Compare build output between page routes and API routes
- Look for differences in how they're compiled
6. **Test Vercel CLI Deploy**
- Deploy directly via CLI: `vercel --prod`
- Check if behavior differs from Git-based deploy
- Might reveal configuration issues
7. **Check Vercel Function Logs**
- Even though functions aren't in build output, try:
- `vercel logs <deployment-url> --since 1h`
- Look for any API route invocations or errors
8. **Review Turbo Configuration**
```bash
# Check turbo.json for Next.js build config
cat /turbo.json
# Verify build runs correctly locally
pnpm --filter=@tpmjs/web build
```
### Alternative Approaches (If Above Fails)
9. **Create New Vercel Project**
- Import from Git fresh
- Use identical settings
- Test if fresh project works
10. **Contact Vercel Support**
- This may be a platform bug with Next.js 16
- Provide this document as context
- Ask specifically why API routes aren't in build output
11. **Temporary Workaround**
- Deploy API routes separately (different service)
- Use Vercel proxy to route `/api/*` to separate deployment
- Not ideal but unblocks development
## Environment Variables Needed
```bash
# Required for API routes
DATABASE_URL="postgresql://..."
# Optional (check if needed)
NODE_ENV="production"
NEXT_PUBLIC_* # Any public env vars
```
## Build Commands Reference
### Local Development
```bash
# Install dependencies
pnpm install
# Generate Prisma client
pnpm --filter=@tpmjs/db db:generate
# Run development server
pnpm --filter=@tpmjs/web dev
# Build for production
pnpm --filter=@tpmjs/web build
# Test build locally
pnpm --filter=@tpmjs/web start
```
### Vercel Configuration
**Current:**
```json
{
"buildCommand": "cd ../.. && pnpm --filter=@tpmjs/web build",
"installCommand": "pnpm install"
}
```
**Alternative to try:**
```json
{
"buildCommand": "cd ../.. && turbo build --filter=@tpmjs/web",
"installCommand": "pnpm install",
"framework": "nextjs"
}
```
## Key Files to Review
1. `/apps/web/next.config.ts` - Next.js configuration
2. `/apps/web/vercel.json` - Vercel app-level config
3. `/vercel.json` - Vercel root config
4. `/turbo.json` - Turborepo configuration
5. `/apps/web/.next/routes-manifest.json` - Route definitions
6. `/apps/web/.next/build-manifest.json` - Build output
7. Vercel build logs (from dashboard)
## Questions for Vercel Support
If escalating to Vercel support, ask:
1. Why are API routes not appearing in the build output (`vercel inspect`) when pages are deploying correctly?
2. Is there a known issue with Next.js 16 App Router API routes in Turborepo monorepos?
3. What's the correct way to configure `vercel.json` for a Next.js 16 app in a monorepo with custom build commands?
4. Could having two projects (`tpmjs` and `v0-tool-registry-page`) with the same domain cause routing issues?
5. Are there any specific requirements for deploying Next.js 16 API routes that differ from Next.js 15?
## Related Documentation
- [Next.js 16 Upgrade Guide](https://nextjs.org/docs/app/guides/upgrading/version-16)
- [Vercel Next.js Deployment](https://vercel.com/docs/frameworks/nextjs)
- [Vercel Functions Configuration](https://vercel.com/docs/functions/configuring-functions)
- [Turborepo with Vercel](https://vercel.com/docs/monorepos/turborepo)
- [Next.js App Router API Routes](https://nextjs.org/docs/app/api-reference/file-conventions/route)
## Recent Commits Related to This Issue
```
065196d fix(build): simplify Vercel build command
8281f8f fix(build): disable Turbopack for Vercel deployment
748ca4d fix(api): configure Vercel functions for API routes with maxDuration
b93cd42 fix: remove redirects to resolve redirect loop
c42bfb6 fix(redirects): exclude API routes from www redirect
9be3af7 fix(routing): move www redirect from vercel.json to Next.js config
a92dfff fix(build): add workspace packages to Next.js transpilePackages
cc6c824 fix(vercel): configure Turborepo monorepo build for apps/web
```
## ✅ CONCLUSION - ROOT CAUSE IDENTIFIED
### The Real Problem
**Vercel is NOT detecting this project as a Next.js application.**
When Vercel doesn't detect Next.js, it:
- Uses `@vercel/static-builder` instead of `@vercel/next`
- Treats the deployment as a static site
- Deploys pages (static HTML) successfully
- **Completely drops all App Router API routes**
- Never generates serverless functions for `/api/*` routes
This explains EVERY symptom:
- ✅ Pages work (they're static files)
- ❌ API routes timeout (they were never deployed)
- ❌ No `λ api/*` in build output (functions don't exist)
- ❌ Direct Vercel URLs timeout (not a DNS issue)
- ❌ "Redirecting..." on custom domain (wrong project owns the domain)
### Why Vercel Doesn't Detect Next.js
**1. Wrong Root Directory**
- Vercel project likely has Root Directory set to `.` or empty
- Should be exactly: `apps/web`
- A single character difference breaks Next.js detection
**2. Wrong Framework Preset**
- When Vercel can't auto-detect Next.js (due to monorepo + wrong root)
- It defaults to Framework Preset = "Other"
- "Other" uses static builder, not Next.js builder
**3. Domain Conflict**
- Two projects exist: `tpmjs` and `v0-tool-registry-page`
- Both have `tpmjs.com` domain assigned
- Requests randomly route to wrong project
- "Redirecting..." comes from the old project, not your code
**4. Custom Build Commands**
- Custom build commands can bypass Vercel's auto-detection
- Should let Vercel auto-detect and use default commands
### Evidence
**Local build works:**
```bash
$ ls apps/web/.next/server/app/api/
health/ stats/ sync/ tools/
# All API routes compile correctly
```
**Vercel deployment missing API routes:**
```bash
$ vercel inspect https://tpmjs-1chh44d1u-tpmjs.vercel.app
Builds
├── λ tool/[slug] # Pages present
├── λ _global-error # Pages present
└── 56 output items hidden
# NO api/health, NO api/tools, NO api/* at all
```
This is **not a Next.js bug, not a Turborepo bug, not a code issue**.
This is **pure Vercel project misconfiguration**.
### The Fix (4 Steps, 5 Minutes)
See `VERCEL_FIX_CHECKLIST.md` for detailed instructions.
**Quick version:**
1. Set Root Directory to `apps/web` (exactly)
2. Set Framework Preset to "Next.js"
3. Remove `tpmjs.com` domain from `v0-tool-registry-page` project
4. Clear custom build commands (let Vercel auto-detect)
**No code changes needed. Just fix Vercel dashboard settings.**
### After the Fix
Next deployment will show:
```bash
$ vercel inspect <new-deployment>
Builds
├── λ api/health
├── λ api/tools
├── λ api/sync/changes
├── λ api/tools/[id]
├── λ api/tools/[slug]
├── λ tool/[slug]
...
```
And API routes will work instantly:
```bash
$ curl https://tpmjs.com/api/health
{"status":"ok","timestamp":"..."}
```
---
## Summary for ChatGPT
The core issue is that **Vercel is not detecting the project as Next.js** due to incorrect Root Directory, Framework Preset set to "Other", domain conflicts between two projects, and custom build commands bypassing auto-detection.
When Vercel doesn't detect Next.js, it uses a static site builder which deploys pages but **completely drops all App Router API routes** from the build. This is why pages work but all `/api/*` requests timeout - the API route serverless functions were never created.
**The fix is purely configuration** - no code changes needed. Set Root Directory to `apps/web`, Framework Preset to "Next.js", remove the domain from the old project, and clear custom build commands. See `VERCEL_FIX_CHECKLIST.md` for step-by-step instructions.

306
DENO_NODE_PACKAGE_ISSUE.md
View file

@ -1,306 +0,0 @@
# Running `ai-sdk-tool-code-execution` in Deno - Compatibility Issue
## Problem Summary
We need to run the npm package `ai-sdk-tool-code-execution` in a Deno runtime environment on Railway. The package requires Node.js built-ins (`node:sqlite`, `undici`) that don't exist in Deno, and we're looking for a solution to make it work.
## Environment
- **Runtime:** Deno 1.39.0 on Railway
- **Package:** `ai-sdk-tool-code-execution@0.0.2`
- **Import Method:** Dynamic imports via esm.sh CDN
- **Use Case:** Remote code execution for AI SDK tools
## What We're Trying to Do
We have a Deno server that dynamically imports npm packages at runtime to provide AI SDK tools. The workflow is:
1. User requests a tool (e.g., `executeCode`)
2. Deno server fetches the package from esm.sh or npm
3. Server loads the tool's schema and execution function
4. Server executes the tool with user-provided parameters
## The Package We Need
**Package:** `ai-sdk-tool-code-execution`
**Version:** `0.0.2`
**Description:** Execute Python code in a sandboxed environment using Vercel Sandbox
**npm URL:** https://www.npmjs.com/package/ai-sdk-tool-code-execution
**CDN URLs:**
- esm.sh: `https://esm.sh/ai-sdk-tool-code-execution@0.0.2`
- jsdelivr: `https://cdn.jsdelivr.net/npm/ai-sdk-tool-code-execution@0.0.2/+esm`
**Dependencies (from package.json):**
```json
{
"dependencies": {
"ai": "^4.0.18",
"better-sqlite3": "^11.8.1",
"undici": "^7.16.0"
}
}
```
**Key Issue:** The package depends on:
- `better-sqlite3` → which requires `node:sqlite` (Node.js built-in)
- `undici` → HTTP client that uses Node.js internals
## What We've Tried
### Attempt 1: Deno npm: Specifier (Node.js Compatibility Mode)
**Code:**
```typescript
const npmUrl = `npm:ai-sdk-tool-code-execution@0.0.2`;
const module = await import(npmUrl);
```
**Error:**
```
Loading unprepared module: npm:ai-sdk-tool-code-execution@0.0.2
```
**Why it failed:** Deno's npm compatibility requires the package to be "prepared" (downloaded/cached) before import. Dynamic imports of unprepared npm packages fail.
### Attempt 2: esm.sh with Node.js Target
**Code:**
```typescript
const esmUrl = `https://esm.sh/ai-sdk-tool-code-execution@0.0.2?target=esnext`;
const module = await import(esmUrl);
```
**Error:**
```
Module not found "https://esm.sh/node:sqlite?target=esnext"
at https://esm.sh/undici@^7.16.0?target=esnext:25:8
```
**Why it failed:** The package code imports `node:sqlite` which esm.sh tries to load from `https://esm.sh/node:sqlite?target=esnext`, but `node:sqlite` is a Node.js built-in, not an npm package.
### Attempt 3: Multi-Strategy with Fallback
**Code:**
```typescript
let module;
let importError;
// Strategy 1: npm: specifier
try {
const npmUrl = `npm:${packageName}@${version}`;
module = await import(npmUrl);
} catch (error) {
importError = error;
// Strategy 2: esm.sh with esnext target
try {
const esmUrl = `https://esm.sh/${packageName}@${version}?target=esnext`;
module = await import(esmUrl);
} catch (esmError) {
return { success: false, error: esmError.message };
}
}
```
**Result:** Both strategies fail with the same errors as above.
## Current Deno Configuration
**`deno.json`:**
```json
{
"compilerOptions": {
"allowJs": true,
"lib": ["deno.window"],
"strict": true
},
"nodeModulesDir": true,
"unstable": ["byonm"],
"imports": {
"zod-to-json-schema": "https://esm.sh/zod-to-json-schema@3.25.0"
}
}
```
**Key Settings:**
- `nodeModulesDir: true` - Creates `node_modules` directory for npm packages
- `unstable: ["byonm"]` - Enables "Bring Your Own Node Modules" mode
## Full Error Details
### npm: Strategy Error
```json
{
"success": false,
"error": "Failed to import package: ...",
"details": {
"npmError": "Loading unprepared module: npm:ai-sdk-tool-code-execution@0.0.2, imported from: file:///app/server.ts"
}
}
```
### esm.sh Strategy Error
```json
{
"success": false,
"error": "Failed to import package: Module not found \"https://esm.sh/node:sqlite?target=esnext\"",
"details": {
"esmError": "Module not found \"https://esm.sh/node:sqlite?target=esnext\".\n at https://esm.sh/undici@^7.16.0?target=esnext:25:8"
}
}
```
## Technical Deep Dive
### Why This Package Needs Node.js
1. **better-sqlite3** - Native Node.js addon for SQLite
- Uses `node:sqlite` built-in
- Compiled C++ bindings
- Not available in Deno without Node compatibility layer
2. **undici** - Modern HTTP client for Node.js
- Uses Node.js streams and buffer APIs
- Optimized for Node.js internals
- May work in Deno with polyfills, but blocked by sqlite dependency
### Deno's Node.js Compatibility
Deno supports many Node.js built-ins via `node:*` imports:
- `node:fs`, `node:path`, `node:http`, `node:crypto`, etc.
**BUT** it does NOT support:
- `node:sqlite` (not a standard Node.js built-in)
- Native addons (`.node` files)
- Some advanced internal APIs
### The Import Flow
1. **Deno tries to import** `npm:ai-sdk-tool-code-execution@0.0.2`
2. **Package resolves to** esm.sh or npm registry
3. **Package imports** `better-sqlite3`
4. **better-sqlite3 imports** `node:sqlite`
5. **FAILURE:** `node:sqlite` doesn't exist in Deno or esm.sh
## Questions for ChatGPT
1. **Can Deno's npm compatibility layer handle `better-sqlite3` or `node:sqlite`?**
- Is there a Deno-compatible SQLite library we could alias?
- Can we use import maps to redirect `node:sqlite` to a Deno polyfill?
2. **Can we "prepare" the npm module in Deno before dynamic import?**
- Is there a way to pre-cache npm packages in Deno?
- Can we use `deno vendor` or similar to prepare the package?
3. **Can esm.sh or other CDNs provide Node.js built-in polyfills?**
- Does esm.sh have a mode that bundles Node.js built-ins?
- Are there CDN parameters we're missing?
4. **Could we use Deno's `--node-modules-dir` flag differently?**
- Should we install the package via npm/pnpm first?
- Can we point Deno to pre-installed node_modules?
5. **Is there a way to patch/bundle the package to remove Node.js dependencies?**
- Could we create a Deno-compatible fork?
- Are there tools to transpile Node.js packages to Deno?
6. **Alternative: Different code execution package?**
- Are there Deno-native code execution tools?
- Could we use WebAssembly or browser-based sandboxing?
## What Would Success Look Like
**Ideal outcome:**
```typescript
// This should work in Deno:
const module = await import('npm:ai-sdk-tool-code-execution@0.0.2');
const { executeCode } = module;
// And this should execute:
const result = await executeCode.execute({
code: 'print(fibonacci(10))',
language: 'python'
});
```
**Acceptable outcome:**
```typescript
// Some preparation step, then:
const module = await import('https://esm.sh/ai-sdk-tool-code-execution@0.0.2');
// Works without errors
```
## Repository Context
**Project:** TPMJS - Tool Package Manager for AI SDK
**Server:** `apps/railway-executor/server.ts`
**Config:** `apps/railway-executor/deno.json`
**Deployment:** Railway with Deno runtime
**Server Code (Simplified):**
```typescript
async function loadAndDescribe(req: Request): Promise<Response> {
const { packageName, exportName, version, importUrl } = await req.json();
// Try npm: specifier first
try {
const npmUrl = `npm:${packageName}@${version}`;
const module = await import(npmUrl);
const tool = module[exportName];
return Response.json({ success: true, tool });
} catch (error) {
// Try esm.sh fallback
const esmUrl = `https://esm.sh/${packageName}@${version}?target=esnext`;
const module = await import(esmUrl);
const tool = module[exportName];
return Response.json({ success: true, tool });
}
}
Deno.serve({ port: 3001 }, handler);
```
## Live Error Logs
**Request:**
```bash
curl -X POST https://endearing-commitment-production.up.railway.app/load-and-describe \
-H "Content-Type: application/json" \
-d '{
"packageName": "ai-sdk-tool-code-execution",
"exportName": "executeCode",
"version": "0.0.2",
"importUrl": "https://esm.sh/ai-sdk-tool-code-execution@0.0.2"
}'
```
**Response:**
```json
{
"success": false,
"error": "Failed to import package: Module not found \"https://esm.sh/node:sqlite?target=esnext\"",
"details": {
"npmError": "Loading unprepared module: npm:ai-sdk-tool-code-execution@0.0.2",
"esmError": "Module not found \"https://esm.sh/node:sqlite?target=esnext\""
}
}
```
## Additional Context
- We successfully load other packages (e.g., `@tpmjs/hello`, `zod-to-json-schema`)
- Only packages with Node.js built-in dependencies fail
- Switching to Node.js would work, but we prefer Deno's security model
- This is for a production tool registry serving AI SDK tools to users
## Related Resources
- **Deno npm compatibility:** https://deno.com/manual/node/npm_specifiers
- **Deno Node built-ins:** https://deno.com/manual/node/node_specifiers
- **esm.sh documentation:** https://esm.sh/
- **Package source:** https://www.npmjs.com/package/ai-sdk-tool-code-execution
- **Deno SQLite libraries:** https://deno.land/x/sqlite@v3.8
---
**Question for ChatGPT:** Is there any way to make `ai-sdk-tool-code-execution` work in Deno, given these constraints? If not, what's the closest alternative that would work in Deno's runtime?

2
DEPLOYMENT.md
View file

@ -167,7 +167,7 @@ Not needed for Option 1 (Deployment Protection).
Add to README.md to show CI status:
```markdown
[![CI](https://github.com/YOUR_ORG/YOUR_REPO/actions/workflows/ci.yml/badge.svg)](https://github.com/YOUR_ORG/YOUR_REPO/actions/workflows/ci.yml)
[![CI](https://github.com/tpmjs/tpmjs/actions/workflows/ci.yml/badge.svg)](https://github.com/tpmjs/tpmjs/actions/workflows/ci.yml)
```
## Summary

864
DYNAMIC_IMPORT_ISSUE.md
View file

@ -1,864 +0,0 @@
# Dynamic Import Issue: Cannot Import ESM Modules from CDN in Next.js Server-Side API Route
## Executive Summary
We're building a dynamic tool loading system where AI agents can discover and load tools at runtime from npm packages via esm.sh CDN. The system successfully searches and finds relevant tools, but fails when trying to dynamically import them using `import()` in a Next.js App Router API route.
**Error**: `Error: Cannot find module 'unknown'` with code `MODULE_NOT_FOUND`
**Critical Question**: How can we dynamically import ESM modules from external URLs (like esm.sh) in Next.js 16 App Router API routes running in Node.js runtime?
---
## System Architecture
### High-Level Flow
```
1. User sends message → "use firecrawl to search for ajax davis"
2. Chat API extracts query → "use firecrawl to search for ajax davis"
3. Pre-flight search → Calls searchTpmjsToolsTool.execute({ query, limit: 5 })
4. Search API returns → Top 5 matching tools from database (BM25-like scoring)
5. Dynamic loading → Tries to import tools from esm.sh URLs ❌ FAILS HERE
6. Agent uses tools → Would pass loaded tools to AI model
```
### Tech Stack
- **Framework**: Next.js 16.0.4
- **Build Tool**: Turbopack (default in Next.js 15+)
- **Runtime**: Node.js (not edge)
- **Package Manager**: pnpm (monorepo with workspaces)
- **Deployment Target**: Vercel (eventually, currently local dev)
- **AI SDK**: Vercel AI SDK v6.0.0-beta.124
- **Model**: OpenAI GPT-4o-mini via `streamText()`
### Monorepo Structure
```
tpmjs/
├── apps/
│ ├── playground/ # Next.js app with chat interface
│ │ └── src/
│ │ ├── app/api/chat/route.ts # Where dynamic import fails
│ │ └── lib/dynamic-tool-loader.ts
│ └── web/ # Tool registry website
│ └── src/app/api/tools/search/route.ts
└── packages/
└── tools/
├── hello/ # Static tool (works fine)
└── search-registry/ # Meta-tool for searching registry
```
---
## Detailed Code Implementation
### File 1: `apps/playground/src/lib/dynamic-tool-loader.ts`
**Purpose**: Load tools dynamically from esm.sh CDN
```typescript
// Cache for imported tool modules (process-level)
const moduleCache = new Map<string, any>();
// Cache for per-conversation active tools
const conversationTools = new Map<string, Set<string>>();
/**
* Generate cache key for a tool
*/
function getCacheKey(packageName: string, exportName: string): string {
return `${packageName}::${exportName}`;
}
/**
* Validate that an import is a valid AI SDK tool
*/
function isValidTool(value: any): boolean {
return (
value &&
typeof value === 'object' &&
typeof value.description === 'string' &&
typeof value.execute === 'function'
);
}
/**
* Dynamically import a tool from ESM CDN
*
* THIS IS WHERE IT FAILS ❌
*/
export async function loadToolDynamically(
packageName: string,
exportName: string,
version: string,
importUrl?: string
): Promise<any | null> {
const cacheKey = getCacheKey(packageName, exportName);
// Check cache first
if (moduleCache.has(cacheKey)) {
console.log(`✅ Cache hit: ${cacheKey}`);
return moduleCache.get(cacheKey);
}
// Build import URL
const url = importUrl || `https://esm.sh/${packageName}@${version}`;
try {
console.log(`📦 Importing: ${url}`);
// Example: https://esm.sh/firecrawl-aisdk@0.7.2
// Dynamic import with @vite-ignore to bypass bundler
const module = await import(/* @vite-ignore */ url);
console.log(`🔍 Module imported successfully`);
console.log(`🔍 Module type: ${typeof module}`);
console.log(`🔍 Module keys: ${Object.keys(module).join(', ')}`);
console.log(`🔍 Looking for export: "${exportName}"`);
console.log(`🔍 Export exists: ${exportName in module}`);
console.log(`🔍 Export type: ${typeof module[exportName]}`);
// Get the specific export
const tool = module[exportName];
if (!tool) {
console.error(`❌ Export "${exportName}" not found in module. Available exports:`, Object.keys(module));
return null;
}
console.log(`🔍 Tool structure:`, {
hasDescription: 'description' in tool,
hasExecute: 'execute' in tool,
hasInputSchema: 'inputSchema' in tool,
keys: Object.keys(tool),
});
if (!isValidTool(tool)) {
console.error(`❌ Invalid tool structure: ${exportName} from ${packageName}`);
console.error(` Tool:`, tool);
return null;
}
// Cache successful import
moduleCache.set(cacheKey, tool);
console.log(`✅ Loaded: ${cacheKey}`);
return tool;
} catch (error) {
console.error(`❌ Failed to load ${packageName}#${exportName}:`, error);
console.error(` URL: ${url}`);
console.error(` Stack:`, error instanceof Error ? error.stack : 'No stack trace');
return null;
}
}
/**
* Load multiple tools in parallel
*/
export async function loadToolsBatch(
toolMetadata: Array<{
packageName: string;
exportName: string;
version: string;
importUrl?: string;
}>
): Promise<Record<string, any>> {
const promises = toolMetadata.map((meta) =>
loadToolDynamically(
meta.packageName,
meta.exportName,
meta.version,
meta.importUrl
).then((tool) => ({
key: getCacheKey(meta.packageName, meta.exportName),
tool,
}))
);
const results = await Promise.all(promises);
const tools: Record<string, any> = {};
for (const { key, tool } of results) {
if (tool) {
tools[key] = tool;
}
}
return tools;
}
```
### File 2: `apps/playground/src/app/api/chat/route.ts`
**Purpose**: Main chat API that orchestrates tool discovery and loading
```typescript
import { createOpenAI } from '@ai-sdk/openai';
import { type UIMessage, convertToModelMessages, stepCountIs, streamText } from 'ai';
import type { NextRequest } from 'next/server';
import { NextResponse } from 'next/server';
import { env } from '~/env';
import { loadAllTools, sanitizeToolName } from '~/lib/tool-loader';
import { searchTpmjsToolsTool } from '@tpmjs/search-registry';
import {
loadToolsBatch,
addConversationTools,
} from '~/lib/dynamic-tool-loader';
export const runtime = 'nodejs'; // ⚠️ Important: We're using Node.js runtime, not edge
export const dynamic = 'force-dynamic';
export const maxDuration = 60;
// Initialize OpenAI provider
const openai = createOpenAI({
apiKey: env.OPENAI_API_KEY,
});
// Add conversation state tracking (in-memory for MVP)
const conversationStates = new Map<string, { loadedTools: Record<string, any> }>();
/**
* POST /api/chat
* Chat with AI agent that can execute TPMJS tools
*/
export async function POST(request: NextRequest) {
try {
const body = await request.json();
console.log('📥 Request body:', JSON.stringify(body, null, 2));
const messages: UIMessage[] = body.messages || [];
const conversationId: string = body.conversationId || 'default';
console.log(`🔑 Conversation ID: ${conversationId}`);
// Get or create conversation state
if (!conversationStates.has(conversationId)) {
console.log('✨ Creating new conversation state');
conversationStates.set(conversationId, { loadedTools: {} });
}
const state = conversationStates.get(conversationId)!;
console.log(`📊 Current loaded tools in conversation: ${Object.keys(state.loadedTools).length}`);
// 1. Load static tools + search tool
const staticTools = await loadAllTools();
console.log(`🔧 Loaded ${Object.keys(staticTools).length} static tools`);
staticTools.searchTpmjsTools = searchTpmjsToolsTool;
console.log('✅ Added searchTpmjsTools to static tools');
// 2. Extract user query from last message for tool search
const lastMessage = messages[messages.length - 1];
let userQuery = '';
if (lastMessage?.role === 'user') {
const parts = (lastMessage as any).parts || [];
for (const part of parts) {
if (part.type === 'text') {
userQuery = part.text;
break;
}
}
}
console.log(`💬 User query: "${userQuery}"`);
// 3. Automatically search for relevant tools based on the user's message
if (userQuery && userQuery.trim().length > 0) {
console.log('🔎 Searching for relevant tools...');
try {
const searchResult = await searchTpmjsToolsTool.execute({
query: userQuery,
limit: 5, // Get top 5 relevant tools
}, {} as any);
console.log(`📦 Found ${searchResult.matchCount} matching tools`);
if (searchResult.tools && searchResult.tools.length > 0) {
console.log(`🔧 Tools found:`, searchResult.tools.map((t: any) => `${t.packageName}/${t.exportName}`));
// Dynamically load tools from esm.sh
console.log(`📥 Loading ${searchResult.tools.length} tools dynamically...`);
const toolsToLoad = searchResult.tools.map((meta: any) => ({
packageName: meta.packageName,
exportName: meta.exportName,
version: meta.version,
importUrl: meta.importUrl,
}));
try {
// ❌ THIS IS WHERE IT FAILS
const loadedTools = await loadToolsBatch(toolsToLoad);
console.log(`✅ Successfully loaded ${Object.keys(loadedTools).length} tools`);
// Add sanitized tools to conversation state
for (const [key, tool] of Object.entries(loadedTools)) {
const [pkg, exp] = key.split('::');
const sanitizedKey = sanitizeToolName(`${pkg}-${exp}`);
state.loadedTools[sanitizedKey] = tool;
console.log(`✅ Added to conversation: ${sanitizedKey}`);
}
// Track for this conversation
addConversationTools(conversationId, Object.keys(state.loadedTools));
} catch (error) {
console.error('❌ Error loading tools:', error);
}
} else {
console.log('ℹ️ No matching tools found for this query');
}
} catch (error) {
console.error('❌ Error searching for tools:', error);
}
}
// 4. Merge with conversation's dynamically loaded tools
const allTools: Record<string, any> = { ...staticTools, ...state.loadedTools };
// 5. Build system prompt with available tools
const toolsList = Object.keys(allTools)
.map((name) => {
const tool = allTools[name] as { description?: string } | undefined;
return `- ${name}: ${tool?.description || 'No description'}`;
})
.join('\n');
const system = `You are a helpful AI assistant that can use TPMJS tools to help users.
Available tools:
${toolsList}
When you use a tool, you MUST always follow up with a natural language answer to the user summarizing the result.`;
// 6. Stream response with all available tools
const result = streamText({
model: openai('gpt-4o-mini'),
system,
messages: convertToModelMessages(messages),
tools: allTools,
stopWhen: stepCountIs(5),
});
return result.toUIMessageStreamResponse();
} catch (error) {
console.error('Chat API error:', error);
return new Response(
JSON.stringify({
success: false,
error: error instanceof Error ? error.message : 'Unknown error',
}),
{
status: 500,
headers: { 'Content-Type': 'application/json' },
}
);
}
}
```
### File 3: Example Tool Metadata (from search API)
When we search for "firecrawl", the search API returns:
```json
{
"success": true,
"query": "firecrawl ajax davis",
"results": {
"total": 29,
"returned": 5,
"tools": [
{
"id": "cm4abc123",
"exportName": "searchTool",
"description": "Search the web using Firecrawl's search API",
"qualityScore": 0.85,
"package": {
"npmPackageName": "firecrawl-aisdk",
"npmVersion": "0.7.2",
"category": "web-scraping",
"frameworks": ["vercel-ai"],
"env": "server"
},
"importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2",
"cdnUrl": "https://cdn.jsdelivr.net/npm/firecrawl-aisdk@0.7.2/+esm"
}
]
}
}
```
So we're trying to:
```typescript
const module = await import('https://esm.sh/firecrawl-aisdk@0.7.2');
const tool = module.searchTool; // Get the exported tool
```
---
## The Error
### Console Output
```
📥 Loading 5 tools dynamically...
📦 Importing: https://esm.sh/firecrawl-aisdk@0.7.2
❌ Failed to load firecrawl-aisdk#searchTool: Error: Cannot find module 'unknown'
at <unknown> (.next/dev/server/chunks/[root-of-the-server]__746deca2._.js:357:23)
at loadToolDynamically (.next/dev/server/chunks/[root-of-the-server]__746deca2._.js:360:11)
at <unknown> (src/lib/dynamic-tool-loader.ts:108:5)
at Array.map (<anonymous>)
at loadToolsBatch (src/lib/dynamic-tool-loader.ts:107:33)
at POST (src/app/api/chat/route.ts:104:53)
{
code: 'MODULE_NOT_FOUND'
}
URL: https://esm.sh/firecrawl-aisdk@0.7.2
Stack: Error: Cannot find module 'unknown'
at /Users/ajaxdavis/repos/tpmjs/tpmjs/apps/playground/.next/dev/server/chunks/[root-of-the-server]__746deca2._.js:357:23
at loadToolDynamically (/Users/ajaxdavis/repos/tpmjs/tpmjs/apps/playground/.next/dev/server/chunks/[root-of-the-server]__746deca2._.js:360:11)
```
### Key Observations
1. **Error happens immediately** - Never gets to our debug logs after `await import()`
2. **Error is MODULE_NOT_FOUND** - Treating URL as a module path
3. **Error says "unknown"** - Not even using the actual module name
4. **Code is in .next/dev/server/chunks/** - Next.js/Turbopack transformed our code
5. **Same error for all packages** - firecrawl-aisdk, @exalabs/ai-sdk, etc.
---
## Verification: The URL Works
### Manual Test 1: Browser
```
Visit: https://esm.sh/firecrawl-aisdk@0.7.2
```
Returns valid ESM module:
```javascript
/* esm.sh - firecrawl-aisdk@0.7.2 */
import * as __1$ from "/v135/@ai-sdk/provider-utils@2.0.8/...";
// ... rest of module code
export { searchTool, scrapeTool, crawlTool };
```
### Manual Test 2: Plain Node.js Script
Create `test-import.mjs`:
```javascript
const module = await import('https://esm.sh/firecrawl-aisdk@0.7.2');
console.log('Module:', module);
console.log('Exports:', Object.keys(module));
```
Run: `node test-import.mjs`
**Expected**: Would work in plain Node.js with `--experimental-network-imports` flag
**In Next.js**: Can't even get this far
---
## What We've Tried
### Attempt 1: `/* @vite-ignore */` Comment
```typescript
const module = await import(/* @vite-ignore */ url);
```
**Result**: Still fails with MODULE_NOT_FOUND
### Attempt 2: `/* webpackIgnore: true */` Comment
```typescript
const module = await import(/* webpackIgnore: true */ url);
```
**Result**: Still fails with MODULE_NOT_FOUND
### Attempt 3: Force Dynamic Runtime
```typescript
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
```
**Result**: Still fails (we're already using this)
### Attempt 4: Verify esm.sh Works
- Tested URLs in browser: ✅ Works
- All packages return valid ESM: ✅ Valid
- esm.sh is accessible: ✅ Reachable
### Attempt 5: Check Static Imports
```typescript
import { helloWorldTool } from '@tpmjs/hello';
```
**Result**: Works perfectly (but bundled at build time)
---
## Configuration Files
### `apps/playground/next.config.ts`
```typescript
import type { NextConfig } from 'next';
const config: NextConfig = {
reactStrictMode: true,
transpilePackages: ['@tpmjs/ui'],
experimental: {
turbo: {
// Using Turbopack (Next.js 15+ default)
},
},
};
export default config;
```
### `apps/playground/package.json` (relevant parts)
```json
{
"name": "@tpmjs/playground",
"version": "0.0.0",
"private": true,
"scripts": {
"dev": "next dev --port 3001",
"build": "next build",
"start": "next start"
},
"dependencies": {
"@ai-sdk/openai": "^1.0.15",
"@tpmjs/hello": "workspace:*",
"@tpmjs/search-registry": "workspace:*",
"ai": "6.0.0-beta.124",
"next": "16.0.4",
"react": "19.0.0"
}
}
```
### `turbo.json` (monorepo config)
```json
{
"$schema": "https://turbo.build/schema.json",
"tasks": {
"dev": {
"cache": false,
"persistent": true
},
"build": {
"dependsOn": ["^build"],
"outputs": [".next/**", "dist/**"]
}
}
}
```
---
## Why This Matters
### The Bigger Picture
We're building a **self-referential tool discovery system**:
1. **Tool Registry** (tpmjs.com) - Indexes all TPMJS-compatible tools from npm
2. **Search Tool** - AI SDK tool that searches the registry
3. **Dynamic Loader** - Loads found tools at runtime
4. **AI Agent** - Uses dynamically loaded tools
This creates infinite extensibility:
- No need to bundle all possible tools
- Tools can be published to npm independently
- System discovers and loads tools as needed
- Bundle size stays small
### Use Case Example
```
User: "Search Wikipedia for quantum computing"
↓
System searches registry: Finds "wikipedia-aisdk" tool
↓
System loads tool: import('https://esm.sh/wikipedia-aisdk@1.0.0')
↓
AI uses tool: wikipediaSearchTool.execute({ query: "quantum computing" })
↓
User gets answer with Wikipedia citations
```
---
## Possible Root Causes
### Hypothesis 1: Turbopack Doesn't Support Dynamic Import URLs
- Turbopack intercepts all `import()` calls
- Transforms them to module resolution
- Doesn't handle external URLs
### Hypothesis 2: Next.js Security Restriction
- Next.js blocks dynamic imports from external URLs for security
- Prevents arbitrary code execution
- No way to whitelist esm.sh
### Hypothesis 3: Dev Mode Only Issue
- Turbopack dev mode has more restrictions
- Production webpack build might work
- But we need dev mode to work too
### Hypothesis 4: Node.js Runtime Limitation in Next.js
- Next.js Node.js runtime is sandboxed
- Dynamic imports are intercepted before reaching Node.js
- Plain Node.js would work with --experimental-network-imports
---
## Alternative Approaches We're Considering
### Option A: Fetch + VM Module
```typescript
import { SourceTextModule } from 'vm';
const response = await fetch(url);
const code = await response.text();
const module = new SourceTextModule(code);
await module.link(() => {});
await module.evaluate();
const exports = module.namespace;
```
**Pros**: Bypasses import() entirely
**Cons**: Complex, security concerns, might not work in Next.js
### Option B: Separate Microservice
```typescript
// New service: tool-loader-service (Express or Fastify)
POST /load-tool
Body: { packageName, exportName, version }
Response: { tool: <serialized tool object> }
```
**Pros**: Full control, definitely works
**Cons**: Extra infrastructure, latency, complexity
### Option C: Switch to Edge Runtime
```typescript
export const runtime = 'edge'; // Instead of 'nodejs'
```
**Pros**: Edge might have different import behavior
**Cons**: Edge has limitations (no Node.js APIs), might still not work
### Option D: Pre-bundle Common Tools
```typescript
// Generate static imports for top 100 tools
import { tool1 } from 'package1';
import { tool2 } from 'package2';
// ... etc
```
**Pros**: Definitely works
**Cons**: Defeats the purpose, huge bundle size
### Option E: Use unpkg or jsdelivr with Different Strategy
```typescript
// Fetch raw code, eval in isolated context
const response = await fetch(`https://unpkg.com/${pkg}@${ver}/dist/index.mjs`);
const code = await response.text();
const exports = evalInContext(code);
```
**Pros**: More control
**Cons**: Same security/execution issues
---
## Specific Questions for ChatGPT
### Question 1: Is This Possible?
**Can Next.js 16 App Router API routes (Node.js runtime) dynamically import ESM modules from external URLs using `import()`?**
If yes:
- What configuration is needed?
- Are there security allowlists?
- Does it work in both dev and production?
If no:
- Why not?
- What's the recommended alternative?
- Is this a Turbopack limitation or Next.js design?
### Question 2: Turbopack Behavior
**Does Turbopack intercept all `import()` calls, even with magic comments?**
We've tried:
- `/* @vite-ignore */`
- `/* webpackIgnore: true */`
None work. Is there a Turbopack-specific comment or config?
### Question 3: Edge vs Node Runtime
**Would switching to edge runtime change import behavior?**
```typescript
export const runtime = 'edge'; // vs 'nodejs'
```
Does edge runtime allow dynamic imports from URLs?
### Question 4: Best Practice
**What's the recommended way to implement dynamic tool loading in Next.js?**
Given constraints:
- Need to load arbitrary npm packages at runtime
- Packages are ESM modules from CDN
- Can't pre-bundle all possibilities
- Need to work in production on Vercel
### Question 5: Security Model
**Is Next.js intentionally blocking this for security?**
- Is there a whitelist for allowed CDNs?
- Can we configure allowed import sources?
- Is this related to CSP or other security headers?
---
## Environment Details
### Versions
```json
{
"next": "16.0.4",
"react": "19.0.0",
"turbo": "2.6.1",
"pnpm": "9.15.0",
"node": "v20.11.0",
"ai": "6.0.0-beta.124"
}
```
### Operating System
- **OS**: macOS (Darwin 23.5.0)
- **Architecture**: arm64 (Apple Silicon)
### Development Commands
```bash
# Start dev server
pnpm dev --filter=@tpmjs/playground
# Output
▲ Next.js 16.0.4 (Turbopack)
- Local: http://localhost:3001
- Network: http://192.168.0.25:3001
✓ Ready in 2.5s
```
### Build Output Structure
```
apps/playground/.next/
├── dev/
│ └── server/
│ └── chunks/
│ └── [root-of-the-server]__746deca2._.js # ← Error originates here
```
---
## Success Criteria
### What We Need Working
```typescript
// In Next.js API route (Node.js runtime)
const url = 'https://esm.sh/firecrawl-aisdk@0.7.2';
const module = await import(url);
const tool = module.searchTool;
console.log(tool.description); // "Search the web using Firecrawl's search API"
console.log(typeof tool.execute); // "function"
// Tool is ready to use with AI SDK
const result = await tool.execute({ query: "test" }, context);
```
### Acceptable Outcomes
1. ✅ **Best**: Dynamic `import()` works with configuration change
2. ✅ **Good**: Alternative approach that doesn't require microservice
3. ✅ **Acceptable**: Workaround that works in production even if dev is tricky
4. ❌ **Unacceptable**: "You can't do this in Next.js" without alternative
---
## Additional Context
### Why Not Just Bundle Everything?
Currently have ~30 tools in registry, growing to 100s or 1000s:
- Bundle size would be massive (10+ MB)
- Most tools won't be used in most conversations
- Tools are published independently by community
- Want instant availability of new tools without redeploying
### Why esm.sh Specifically?
- ✅ Converts any npm package to ESM
- ✅ Handles dependencies automatically
- ✅ Fast CDN with caching
- ✅ No build step required
- ✅ Version pinning built-in
But we're flexible - if jsdelivr, unpkg, or another approach works better, we'll use it.
### Static Imports Work Fine
This works perfectly (but defeats the purpose):
```typescript
import { searchTool } from 'firecrawl-aisdk';
```
The tools themselves are fine. We just can't load them dynamically.
---
## What We're Hoping For
### Ideal Answer Format
1. **Root cause**: Why it's failing
2. **Solution**: How to fix it (with code example)
3. **Configuration**: Any Next.js config needed
4. **Limitations**: What won't work / tradeoffs
5. **Alternatives**: If dynamic import truly impossible
### We're Happy to Try
- Different CDN (unpkg, jsdelivr, etc.)
- Different import strategy (fetch + eval, vm module, etc.)
- Different runtime (edge if it works)
- Different Next.js version (if specific version supports this)
- Webpack instead of Turbopack (if webpack handles this better)
We just need a path forward that enables runtime tool loading in a production Next.js app on Vercel.
---
## Files to Reference
All code is in this monorepo:
- `apps/playground/src/lib/dynamic-tool-loader.ts` - Import logic
- `apps/playground/src/app/api/chat/route.ts` - API route
- `apps/playground/next.config.ts` - Next.js config
- `DYNAMIC_IMPORT_ISSUE.md` - This document
---
## Thank You
This is a critical blocker for our dynamic tool loading system. Any insights, workarounds, or alternative approaches would be immensely helpful!

988
DYNAMIC_TOOL_LOADING_PRD.md
View file

@ -1,988 +0,0 @@
# Dynamic Tool Loading System - Product Requirements Document
## Executive Summary
Build a self-referential tool discovery system where AI agents can search the TPMJS registry, find relevant tools, and dynamically import them during conversation. This creates a "meta-tool" that makes the entire TPMJS ecosystem available to any agent at runtime.
**Core Innovation:** An AI agent can discover and load tools on-demand by searching the registry, rather than having all tools pre-loaded. This enables infinite tool extensibility without bundle size concerns.
---
## Problem Statement
### Current Limitations
1. **Static Tool Loading**: Playground requires all tools to be hardcoded in `tool-loader.ts`
2. **Bundle Size**: Loading many tools increases bundle size and initialization time
3. **Discovery Gap**: Agents can't discover new tools that match their current task
4. **Manual Updates**: Adding tools requires code changes and redeployment
### User Pain Points
- Users want agents to access the full TPMJS registry without manual configuration
- Developers want to publish tools that are immediately available to all agents
- Agents need context-aware tool selection based on the conversation
---
## Solution Overview
### The Meta-Tool: `searchTpmjsTools`
A TPMJS tool that searches the TPMJS registry and returns tool metadata needed for dynamic import.
**Flow:**
```
User: "Search Wikipedia for quantum computing"
↓
Agent: Calls searchTpmjsTools("wikipedia search")
↓
API: Returns tools matching "wikipedia" (BM25 search)
↓
Playground: Dynamically imports matching tools
↓
Agent: Now has Wikipedia tools available, uses them
```
### Key Components
1. **`@tpmjs/search-registry`** - NPM package exporting `searchTpmjsToolsTool`
2. **`/api/tools/search`** - New API endpoint with BM25 full-text search
3. **Playground Dynamic Loader** - Runtime tool import system
4. **Tool Import Strategy** - ESM CDN imports or bundled approach
---
## Technical Architecture
### Component 1: Search Tool Package
**Package:** `packages/tools/search-registry/`
```typescript
// packages/tools/search-registry/src/index.ts
import { tool } from 'ai';
import { z } from 'zod';
export const searchTpmjsToolsTool = tool({
description: 'Search the TPMJS tool registry to find AI SDK tools. Use this when you need a tool that isn\'t currently available. Returns tool metadata including package names and descriptions.',
parameters: z.object({
query: z.string().describe('Search query (e.g., "weather", "database", "wikipedia")'),
category: z.enum([
'text-analysis',
'code-generation',
'data-processing',
'image-generation',
'audio-processing',
'search',
'integration',
'other'
]).optional().describe('Filter by tool category'),
limit: z.number().min(1).max(20).default(10).describe('Max number of tools to return'),
}),
execute: async ({ query, category, limit }) => {
// Call TPMJS search API
const params = new URLSearchParams({
q: query,
limit: String(limit),
...(category && { category }),
});
const response = await fetch(
`https://tpmjs.com/api/tools/search?${params}`
);
if (!response.ok) {
throw new Error(`Search failed: ${response.statusText}`);
}
const data = await response.json();
// Return structured tool metadata
return {
query,
matchCount: data.tools.length,
tools: data.tools.map((tool: any) => ({
packageName: tool.package.npmPackageName,
exportName: tool.exportName,
description: tool.description,
category: tool.package.category,
qualityScore: tool.qualityScore,
frameworks: tool.package.frameworks,
env: tool.package.env,
})),
};
},
});
```
**Package Metadata:**
```json
{
"name": "@tpmjs/search-registry",
"version": "0.1.0",
"description": "AI SDK tool for searching the TPMJS tool registry",
"keywords": ["tpmjs-tool", "ai", "search"],
"tpmjs": {
"category": "search",
"frameworks": ["vercel-ai"],
"tools": [
{
"exportName": "searchTpmjsToolsTool",
"description": "Search the TPMJS tool registry to find AI SDK tools by keyword, category, or description. Returns tool metadata for dynamic loading.",
"parameters": [
{
"name": "query",
"type": "string",
"description": "Search query (keywords, tool names, descriptions)",
"required": true
},
{
"name": "category",
"type": "string",
"description": "Filter by category (text-analysis, search, etc.)",
"required": false
},
{
"name": "limit",
"type": "number",
"description": "Maximum number of results (1-20, default 10)",
"required": false
}
],
"returns": {
"type": "object",
"description": "Search results with tool metadata for dynamic import"
},
"aiAgent": {
"useCase": "Use this tool when you need a tool that isn't currently available. For example, if asked to search Wikipedia but you don't have a Wikipedia tool, search for 'wikipedia' to find and load it.",
"examples": [
"Search for 'weather' tools when asked about weather",
"Search for 'database' tools when working with data",
"Search for 'code' tools when generating code"
],
"limitations": "Returns metadata only - the playground handles actual tool loading"
}
}
]
}
}
```
---
### Component 2: BM25 Search API Endpoint
**File:** `apps/web/src/app/api/tools/search/route.ts`
**Requirements:**
1. **Full-Text Search with BM25**
- Search across: tool description, package name, npm description, npm keywords
- BM25 scoring for relevance ranking
- Category filtering
- Quality score boosting (rich tier tools rank higher)
2. **Search Implementation Options**
**Option A: PostgreSQL Full-Text Search**
```sql
-- Add tsvector column to tools table
ALTER TABLE tools ADD COLUMN search_vector tsvector;
-- Create GIN index for fast full-text search
CREATE INDEX tools_search_idx ON tools USING GIN(search_vector);
-- Update search vector on insert/update
CREATE TRIGGER tools_search_update
BEFORE INSERT OR UPDATE ON tools
FOR EACH ROW EXECUTE FUNCTION
tsvector_update_trigger(search_vector, 'pg_catalog.english',
description);
```
**Option B: JavaScript BM25 Library**
```typescript
import { BM25 } from 'bm25';
// Load all tools into memory (cached)
const tools = await prisma.tool.findMany({
include: { package: true },
});
// Build BM25 index
const documents = tools.map(tool => ({
id: tool.id,
text: `${tool.description} ${tool.package.npmPackageName} ${tool.package.npmDescription} ${tool.package.npmKeywords.join(' ')}`,
}));
const bm25 = new BM25(documents);
const results = bm25.search(query);
```
**Option C: Hybrid Approach**
- Use PostgreSQL `LIKE` for exact matches (fastest)
- Fall back to BM25 for fuzzy/semantic search
- Cache search results in Redis
3. **API Response Format**
```typescript
// GET /api/tools/search?q=weather&category=integration&limit=10
{
"success": true,
"query": "weather",
"filters": {
"category": "integration"
},
"results": {
"total": 23,
"returned": 10,
"tools": [
{
"id": "clx...",
"exportName": "getWeatherTool",
"description": "Get current weather data for any location using OpenWeatherMap API",
"qualityScore": 0.85,
"package": {
"npmPackageName": "@tpmjs/weather",
"npmVersion": "1.2.0",
"category": "integration",
"frameworks": ["vercel-ai"],
"env": [
{
"name": "OPENWEATHER_API_KEY",
"description": "OpenWeatherMap API key",
"required": true
}
],
"npmRepository": {
"type": "git",
"url": "https://github.com/user/weather-tool"
},
"isOfficial": false
},
// Include everything needed for dynamic import
"importUrl": "https://esm.sh/@tpmjs/weather@1.2.0",
"cdnUrl": "https://cdn.jsdelivr.net/npm/@tpmjs/weather@1.2.0/+esm"
}
// ... more tools
]
}
}
```
---
### Component 3: Dynamic Tool Loader (Playground)
**File:** `apps/playground/src/lib/dynamic-tool-loader.ts`
**Requirements:**
1. **Runtime ESM Import**
```typescript
async function loadToolDynamically(
packageName: string,
exportName: string,
version: string
) {
// Option 1: ESM CDN (esm.sh, unpkg, jsdelivr)
const cdnUrl = `https://esm.sh/${packageName}@${version}`;
try {
const module = await import(/* @vite-ignore */ cdnUrl);
const tool = module[exportName];
if (!isValidTool(tool)) {
throw new Error(`Invalid tool: ${exportName}`);
}
return tool;
} catch (error) {
console.error(`Failed to load ${packageName}:`, error);
return null;
}
}
```
2. **Tool Caching Strategy**
```typescript
// Cache loaded tools to avoid redundant imports
const toolCache = new Map<string, any>();
function getCacheKey(packageName: string, exportName: string): string {
return `${packageName}::${exportName}`;
}
async function loadToolWithCache(
packageName: string,
exportName: string,
version: string
) {
const key = getCacheKey(packageName, exportName);
if (toolCache.has(key)) {
return toolCache.get(key);
}
const tool = await loadToolDynamically(packageName, exportName, version);
if (tool) {
toolCache.set(key, tool);
}
return tool;
}
```
3. **Tool Registry Integration**
```typescript
// Merge static tools + dynamically loaded tools
async function getAllAvailableTools(
staticTools: Record<string, any>,
searchResults: SearchResult[]
): Promise<Record<string, any>> {
const allTools = { ...staticTools };
// Load tools from search results
for (const result of searchResults) {
const tool = await loadToolWithCache(
result.package.npmPackageName,
result.exportName,
result.package.npmVersion
);
if (tool) {
const key = sanitizeToolName(
`${result.package.npmPackageName}-${result.exportName}`
);
allTools[key] = tool;
}
}
return allTools;
}
```
---
### Component 4: Playground Chat Integration
**File:** `apps/playground/src/app/api/chat/route.ts`
**Flow:**
1. **Initial Tool Set**
- Load static tools (hardcoded in tool-loader)
- Always include `searchTpmjsToolsTool` in initial set
2. **Agent Invokes Search**
- Agent calls `searchTpmjsToolsTool` with query
- Search API returns matching tool metadata
- Response includes tool metadata
3. **Dynamic Loading Trigger**
- Detect when agent successfully calls `searchTpmjsToolsTool`
- Extract tool metadata from response
- Load tools dynamically before next agent turn
4. **Tool Availability Update**
- Merge dynamically loaded tools into available tool set
- Agent can now use newly loaded tools in subsequent turns
**Implementation:**
```typescript
// apps/playground/src/app/api/chat/route.ts
export async function POST(req: Request) {
const { messages } = await req.json();
// 1. Load static tools + search tool
let availableTools = await loadAllTools(); // static
availableTools['searchTpmjsTools'] = searchTpmjsToolsTool; // meta-tool
// 2. Create streamText with current tools
const result = streamText({
model: openai('gpt-4'),
messages,
tools: availableTools,
maxSteps: 10, // Allow multiple tool call rounds
onStepFinish: async (step) => {
// 3. Check if agent called searchTpmjsToolsTool
for (const toolCall of step.toolCalls) {
if (toolCall.toolName === 'searchTpmjsTools') {
const searchResults = toolCall.result?.tools || [];
// 4. Dynamically load tools from search results
console.log(`Loading ${searchResults.length} tools dynamically...`);
for (const toolMeta of searchResults) {
const tool = await loadToolWithCache(
toolMeta.packageName,
toolMeta.exportName,
'latest' // or toolMeta.version
);
if (tool) {
const key = sanitizeToolName(
`${toolMeta.packageName}-${toolMeta.exportName}`
);
availableTools[key] = tool;
console.log(`✅ Loaded: ${key}`);
}
}
// 5. Update tool registry for subsequent steps
// Note: This requires AI SDK to support dynamic tool updates
// May need to restart the streamText with updated tools
}
}
},
});
return result.toDataStreamResponse();
}
```
---
## Technical Challenges & Solutions
### Challenge 1: AI SDK Doesn't Support Dynamic Tool Updates Mid-Stream
**Problem:** Vercel AI SDK's `streamText` sets tools at initialization. Can't add tools after streaming starts.
**Solutions:**
**Option A: Multi-Turn Pattern**
```typescript
// Turn 1: Agent searches for tools
// Turn 2: Agent uses loaded tools
// Detect search tool call, return early
if (hasSearchToolCall) {
return new Response(JSON.stringify({
type: 'tools_loaded',
tools: searchResults,
message: 'Tools loaded. Please continue your request.',
}));
}
```
**Option B: Pre-Flight Search (Recommended)**
```typescript
// Before calling streamText, analyze user message
const needsTools = await analyzeMessageForToolNeeds(userMessage);
if (needsTools.length > 0) {
// Pre-load tools based on intent
const searchResults = await searchTools(needsTools);
const dynamicTools = await loadToolsFromResults(searchResults);
availableTools = { ...staticTools, ...dynamicTools };
}
// Now call streamText with full tool set
const result = streamText({
model,
messages,
tools: availableTools,
});
```
**Option C: Agent-Driven Two-Phase**
```typescript
// Phase 1: Planning
const planResult = await generateText({
model,
messages: [
{ role: 'system', content: 'Analyze this request and determine what tools are needed. Call searchTpmjsTools if needed.' },
...messages,
],
tools: { searchTpmjsTools },
});
// Phase 2: Execution with loaded tools
const executionResult = await streamText({
model,
messages,
tools: { ...staticTools, ...loadedTools },
});
```
---
### Challenge 2: ESM Dynamic Import in Browser vs Node.js
**Problem:** Dynamic `import()` works differently in browser vs server environments.
**Solutions:**
**Server-Side (Recommended):**
```typescript
// Use Node.js dynamic import
// Works with esm.sh CDN
const tool = await import(`https://esm.sh/${pkg}@${version}`);
```
**Client-Side (Avoid):**
```typescript
// Browser import() has CORS and CSP restrictions
// Would require:
// 1. CDN supports CORS
// 2. CSP allows script-src from CDN
// 3. Tools are browser-compatible (no Node.js APIs)
```
**Hybrid Approach:**
```typescript
// Load tools server-side, serialize to client
// Client displays available tools
// Server executes tool calls
```
---
### Challenge 3: Tool Dependencies & Environment Variables
**Problem:** Dynamically loaded tools may require:
- Environment variables (API keys)
- npm dependencies not in bundle
- Node.js-specific APIs
**Solutions:**
**Option A: Require Pre-Configuration**
```typescript
// Before loading, check if tool requirements are met
async function canLoadTool(toolMeta: ToolMetadata): Promise<boolean> {
// Check required env vars
for (const env of toolMeta.package.env || []) {
if (env.required && !process.env[env.name]) {
console.warn(`Missing required env: ${env.name}`);
return false;
}
}
return true;
}
```
**Option B: Graceful Degradation**
```typescript
// Load tool, catch errors, inform agent
try {
const tool = await loadTool(packageName, exportName);
return tool;
} catch (error) {
return createStubTool(packageName, exportName, error);
}
function createStubTool(pkg: string, exp: string, error: Error) {
return tool({
description: `[UNAVAILABLE] ${exp} from ${pkg}: ${error.message}`,
parameters: z.object({}),
execute: async () => {
throw new Error(`Cannot execute ${exp}: ${error.message}`);
},
});
}
```
**Option C: Proxy Through Server**
```typescript
// All tools execute server-side where env vars exist
// Client just displays tool calls, server handles execution
```
---
### Challenge 4: Security & Sandboxing
**Problem:** Dynamically importing arbitrary npm packages is a security risk.
**Solutions:**
**Option A: Allowlist Only**
```typescript
// Only load tools from TPMJS registry (already vetted)
const allowedPackages = await prisma.package.findMany({
select: { npmPackageName: true }
});
if (!allowedPackages.includes(packageName)) {
throw new Error('Package not in TPMJS registry');
}
```
**Option B: Version Pinning**
```typescript
// Only load specific versions from registry
// Don't use 'latest' to avoid supply chain attacks
const version = toolMeta.package.npmVersion; // e.g., "1.2.0"
const url = `https://esm.sh/${pkg}@${version}`;
```
**Option C: VM Sandbox (Advanced)**
```typescript
// Execute tools in isolated VM context
import { VM } from 'vm2';
const vm = new VM({
timeout: 5000,
sandbox: {
fetch: safeFetch, // Wrapped fetch with rate limits
console: safeConsole,
},
});
const tool = vm.run(toolCode);
```
---
### Challenge 5: Performance & Bundle Size
**Problem:** Loading many tools dynamically could be slow.
**Solutions:**
**Option A: Lazy Loading**
```typescript
// Only load tools when agent decides to use them
// Not when they're discovered
```
**Option B: Parallel Loading**
```typescript
// Load multiple tools concurrently
const toolPromises = searchResults.map(result =>
loadToolWithCache(result.package.npmPackageName, result.exportName, result.package.npmVersion)
);
const tools = await Promise.all(toolPromises);
```
**Option C: CDN Caching**
```typescript
// Use CDN with aggressive caching
// esm.sh has built-in caching
const url = `https://esm.sh/${pkg}@${version}?target=es2022&bundle`;
```
---
## Implementation Plan
### Phase 1: MVP (Week 1-2)
**Goal:** Prove dynamic loading works with simple prototype
1. **Create `@tpmjs/search-registry` package**
- Implement `searchTpmjsToolsTool`
- Publish to npm
- Add to manual-tools registry
2. **Build `/api/tools/search` endpoint**
- Start with simple PostgreSQL `LIKE` search
- Return tool metadata with package info
- Test with curl
3. **Implement basic dynamic loader**
- Use esm.sh CDN for imports
- Load tools server-side only
- Cache in memory
4. **Playground integration - Two-Turn Pattern**
- User asks question
- Agent calls `searchTpmjsToolsTool`
- Backend loads tools
- Agent uses tools in next turn
**Success Criteria:**
- Agent can search registry
- Agent can use dynamically loaded tools
- End-to-end flow works for 1-2 example tools
---
### Phase 2: BM25 Search (Week 3)
**Goal:** Improve search relevance with BM25
1. **Research BM25 implementation options**
- Test PostgreSQL full-text search
- Test JavaScript BM25 libraries
- Benchmark performance
2. **Implement chosen approach**
- Add search vector column if using PostgreSQL
- Create search index
- Update search endpoint
3. **Test search quality**
- Create test queries
- Measure precision/recall
- Compare to baseline `LIKE` search
**Success Criteria:**
- BM25 search returns more relevant results than LIKE
- Search latency < 100ms for 95th percentile
- Agent can find tools for diverse queries
---
### Phase 3: Production Hardening (Week 4)
**Goal:** Make system production-ready
1. **Error Handling**
- Handle import failures gracefully
- Validate tool schemas
- Return helpful error messages to agent
2. **Security**
- Implement package allowlist
- Pin versions from registry
- Add rate limiting to search API
3. **Performance**
- Implement Redis caching for search results
- Add CDN caching headers
- Optimize tool loading parallelism
4. **Monitoring**
- Log all dynamic tool loads
- Track search queries and results
- Monitor import success/failure rates
**Success Criteria:**
- System handles errors without crashing
- Security review passes
- Latency and reliability SLOs met
---
### Phase 4: Advanced Features (Week 5+)
**Goal:** Enhance UX and capabilities
1. **Pre-flight Search**
- Analyze user message for intent
- Proactively load tools before agent call
- Reduce total turns needed
2. **Tool Recommendations**
- "You might also need..." suggestions
- Based on tool co-occurrence data
- Help agent discover related tools
3. **Client-Side Tool Display**
- Show which tools are available
- Indicate dynamically loaded tools
- Allow user to manually load tools
4. **Tool Versioning**
- Support multiple versions of same tool
- Let agent choose version
- Handle breaking changes gracefully
---
## Success Metrics
### Technical Metrics
1. **Search Quality**
- Precision@10 > 0.8 (80% of top 10 results are relevant)
- Mean Reciprocal Rank (MRR) > 0.7
- Search latency p95 < 100ms
2. **Tool Loading**
- Import success rate > 95%
- Tool load time p95 < 2 seconds
- Cache hit rate > 70% after warmup
3. **End-to-End Performance**
- Total conversation latency < 5 seconds (including tool search + load + execution)
- Agent uses correct tools > 90% of time
### User Metrics
1. **Adoption**
- % of playground sessions using dynamic tools > 30%
- Number of unique tools loaded dynamically per week > 50
2. **Tool Coverage**
- % of user queries satisfied with available tools > 80%
- Tool search leading to successful task completion > 70%
---
## Open Questions
### 1. CDN Choice for ESM Imports
**Options:**
- **esm.sh** - Purpose-built for ESM imports, fast, reliable
- **unpkg** - Popular, simple, but slower
- **jsdelivr** - Fast CDN, good for production
- **Custom bundler** - Pre-bundle tools, serve from our CDN
**Recommendation:** Start with esm.sh for MVP, evaluate custom bundler for production.
---
### 2. When to Load Tools?
**Options:**
- **On-demand**: Load when agent calls search tool (current plan)
- **Pre-flight**: Analyze user message, load proactively
- **Lazy**: Load when agent tries to use tool (not when discovered)
- **Eager**: Load all tools from search results immediately
**Recommendation:** Start with on-demand (Phase 1), add pre-flight in Phase 4.
---
### 3. How to Handle Environment Variables?
**Problem:** Dynamically loaded tools may need API keys (e.g., OpenWeather API).
**Options:**
- **User provides**: UI for users to enter API keys (like playground settings)
- **Server-managed**: Admin pre-configures keys in .env
- **Graceful fail**: Load tool, but execution fails if env missing
- **Hybrid**: Some tools work without keys (free tier), others require keys
**Recommendation:** Start with graceful fail (Phase 1), add user-provided keys (Phase 4).
---
### 4. Should Tools Load Client-Side or Server-Side?
**Client-Side Pros:**
- Reduces server load
- Faster for subsequent uses
- Better for browser-compatible tools
**Client-Side Cons:**
- Requires CORS-enabled CDN
- CSP restrictions
- Many tools need Node.js APIs
- Exposing API keys in browser is insecure
**Server-Side Pros:**
- Access to Node.js APIs
- Secure environment variable access
- No CORS issues
- Easier to implement
**Server-Side Cons:**
- Requires server memory for caching
- Increases server load
- Cold starts for new tools
**Recommendation:** Server-side for MVP (Phase 1), evaluate client-side for browser-compatible tools (Phase 4+).
---
### 5. How to Handle Tool Dependencies?
**Problem:** Some tools depend on other npm packages (e.g., `axios`, `cheerio`).
**Options:**
- **Bundled**: CDN bundles dependencies (esm.sh does this)
- **Peer deps**: Require dependencies in playground package.json
- **Dynamic install**: npm install on-the-fly (slow, risky)
- **Pre-vetted**: Only allow tools with no/minimal dependencies
**Recommendation:** Use esm.sh bundling (Phase 1), bundle size limits if issues arise.
---
## Risk Assessment
### High Risk
1. **Security Vulnerability**
- **Risk**: Malicious package in registry executes code
- **Mitigation**: Allowlist registry packages, version pinning, VM sandboxing
- **Owner**: Security team
2. **Performance Degradation**
- **Risk**: Loading many tools causes timeout/slow response
- **Mitigation**: Parallel loading, caching, lazy loading, timeouts
- **Owner**: Backend team
### Medium Risk
3. **Import Failures**
- **Risk**: CDN down, package incompatible, missing dependencies
- **Mitigation**: Fallback CDNs, error handling, stub tools
- **Owner**: Frontend team
4. **AI SDK Limitations**
- **Risk**: Can't dynamically update tools mid-stream
- **Mitigation**: Two-turn pattern, pre-flight search
- **Owner**: AI team
### Low Risk
5. **Search Quality**
- **Risk**: BM25 doesn't return relevant tools
- **Mitigation**: A/B test search algorithms, collect feedback
- **Owner**: Search team
---
## Future Enhancements
### 1. Tool Composition
- Agent can combine multiple tools
- Example: `searchTool` + `summarizeTool` = search and summarize
### 2. Tool Learning
- Track which tools are used together
- Recommend tool combinations
- "Users who used X also used Y"
### 3. Custom Tool Registry
- Users can add private tools
- Organization-specific tool registry
- Access control and permissions
### 4. Tool Marketplace
- Developers promote their tools
- Usage analytics and ratings
- Paid/premium tools
### 5. Agent Templates
- Pre-configured agents with tool sets
- "Research Agent" has search + summarize tools
- "Code Agent" has code generation tools
---
## Conclusion
This dynamic tool loading system represents a paradigm shift in how AI agents discover and use tools. By making the TPMJS registry itself searchable, we enable infinite extensibility without the limitations of static bundling.
**Key Innovation:** Self-referential tool discovery - a tool that searches for tools.
**Next Steps:**
1. Review this PRD with team
2. Validate technical feasibility with ChatGPT/Claude
3. Spike on BM25 search implementation
4. Spike on dynamic ESM import
5. Begin Phase 1 implementation
**Success Looks Like:**
- User: "Search Wikipedia for quantum computing"
- Agent: *searches registry, finds Wikipedia tool, loads it, uses it*
- User: Gets Wikipedia results without any manual tool configuration
This is a novel approach that could define how AI agents discover and use tools. Let's build it. 🚀

86
ENV_VAR_TRANSPORT_ISSUE.md
View file

@ -1,86 +0,0 @@
# Environment Variables Not Sent to API - Frontend Transport Issue
## Problem
Environment variables saved in localStorage are NOT being sent to `/api/chat` endpoint.
**Evidence from logs:**
```
📥 Request body: {
"conversationId": "7xur5hf1GDSOQMgFYF-l7",
"env": {}, // ❌ EMPTY - should have FIRECRAWL_API_KEY
...
}
```
## Root Cause
The issue is in `apps/playground/src/hooks/useChat.ts`:
```typescript
export function useChat() {
const [conversationId] = useState(() => nanoid());
const envVars = useEnvVars(); // ❌ Empty on first render (useEffect loads async)
const envObject = envVars.reduce(
(acc, { key, value }) => {
acc[key] = value;
return acc;
},
{} as Record<string, string>
);
const chat = useAISDKChat({
transport: new DefaultChatTransport({ // ❌ Created ONCE with empty envObject
api: '/api/chat',
body: {
conversationId,
env: envObject, // ❌ This is {} on first render, never updates
},
}),
});
return { ...chat, conversationId };
}
```
**Why it fails:**
1. `useEnvVars()` loads from localStorage inside a `useEffect` (async)
2. On first render, `envVars = []`, so `envObject = {}`
3. `DefaultChatTransport` is created with `body: { env: {} }`
4. Even when `envVars` updates later, the transport is already created and doesn't re-create
## Attempted Solutions That Don't Work
❌ **Just updating state** - Transport is created once and cached
❌ **Using useEffect** - Transport is already created before effect runs
## What We Need
The `body` field in `DefaultChatTransport` needs to be **dynamic** and read the latest env vars on each request, not just once during component mount.
## Questions for ChatGPT
1. **How do we make `DefaultChatTransport` body dynamic?** Can we pass a function instead of an object?
2. **Does AI SDK have a way to update transport body between messages?** The env vars might change while the chat is open.
3. **Should we use a custom transport instead?** Can we implement our own transport that reads env vars fresh on each request?
4. **Alternative: Can we manually add env to each message?** Is there a way to inject extra data per-request instead of per-transport?
## Current Code Files
- `apps/playground/src/hooks/useChat.ts` - The broken hook
- `apps/playground/src/components/sidebar/SettingsSidebar.tsx` - Where env vars are stored (works fine)
- `apps/playground/src/app/api/chat/route.ts` - Server expects `body.env` but receives `{}`
## What We Know Works
✅ Saving env vars to localStorage - working
✅ Reading env vars from localStorage - working
✅ Server accepting and using env vars - working
❌ **Sending env vars from client to server - BROKEN**
The ONLY broken part is the transport not sending the latest env object.

782
IMPLEMENTATION_CHECKLIST.md
View file

@ -1,782 +0,0 @@
# TPMJS NPM Registry - Implementation Checklist
> **Reference:** See [NPM_MIRROR.md](./NPM_MIRROR.md) for complete architecture details
**Stack Decision:** Vercel + Neon + Vercel Cron (polling-based sync)
---
## 🎯 Implementation Strategy
### Architecture Simplification
**Original Plan (NPM_MIRROR.md):**
- Separate Node.js sync service with persistent changes feed connection
- Self-hosted PostgreSQL
- More complex deployment
**Revised Plan (This Checklist):**
- All-in-one Next.js app on Vercel
- Neon Postgres (serverless)
- Vercel Cron for sync jobs (polling-based, no persistent connections)
- Simpler, faster to ship
### Why This Approach?
✅ **Simpler Infrastructure**
- One deployment (Vercel)
- Managed database (Neon)
- Built-in cron (Vercel Cron)
✅ **Lower Cost**
- No separate sync service hosting
- Neon free tier generous
- Vercel free/hobby tier sufficient for MVP
✅ **Same Functionality**
- Poll NPM changes feed every 1-2 minutes (effectively real-time)
- All discovery features from NPM_MIRROR.md maintained
- Quality scoring, validation, etc. all work the same
---
## 📋 Phase 1: Foundation (Week 1)
**Goal:** Set up database, types, and NPM client
### 1.1 Database Setup
- [ ] **Create Neon project**
- Go to https://neon.tech/
- Create new project
- Save connection string
- [ ] **Create `packages/db` package**
```bash
mkdir -p packages/db
cd packages/db
pnpm init
pnpm add prisma @prisma/client
pnpm add -D typescript @types/node
```
- [ ] **Initialize Prisma**
```bash
npx prisma init
```
- [ ] **Create Prisma schema**
- Copy schema from NPM_MIRROR.md Database Design section
- File: `packages/db/prisma/schema.prisma`
- Include all three models: `Tool`, `SyncCheckpoint`, `SyncLog`
- [ ] **Add database URL to `.env`**
```env
DATABASE_URL="postgresql://..."
```
- [ ] **Run first migration**
```bash
npx prisma migrate dev --name init
npx prisma generate
```
- [ ] **Create Prisma client singleton**
- File: `packages/db/src/client.ts`
```typescript
import { PrismaClient } from '@prisma/client';
const globalForPrisma = globalThis as unknown as {
prisma: PrismaClient | undefined;
};
export const prisma = globalForPrisma.prisma ?? new PrismaClient();
if (process.env.NODE_ENV !== 'production') {
globalForPrisma.prisma = prisma;
}
```
- [ ] **Export from package**
- File: `packages/db/src/index.ts`
```typescript
export { prisma } from './client';
export * from '@prisma/client';
```
- [ ] **Update package.json**
```json
{
"name": "@tpmjs/db",
"main": "./src/index.ts",
"types": "./src/index.ts"
}
```
- [ ] **Seed initial sync state**
```sql
INSERT INTO sync_checkpoints (source, checkpoint)
VALUES
('changes-feed', '{"sequence": 0}'::jsonb),
('keyword-search', '{"lastRun": null}'::jsonb),
('metrics', '{"lastRun": null}'::jsonb)
ON CONFLICT (source) DO NOTHING;
```
**Verification:**
```bash
cd packages/db
npx prisma studio # Should open DB browser with empty tables
```
---
### 1.2 Types Package
**Reference:** See NPM_MIRROR.md "The 'tpmjs' Field Schema" section
- [ ] **Update `packages/types/src/tool.ts`**
- Add `TpmjsMinimalSchema` with Zod
- Add `TpmjsRichSchema` extending minimal
- Export both schemas and inferred types
- [ ] **Create validation helper**
- File: `packages/types/src/validator.ts`
```typescript
export function validateTpmjsField(tpmjs: unknown): {
valid: boolean;
tier: 'minimal' | 'rich' | null;
data?: unknown;
errors?: ZodError[];
}
```
- [ ] **Update exports**
- File: `packages/types/src/index.ts`
- Export all schemas and validators
**Verification:**
```typescript
import { validateTpmjsField } from '@tpmjs/types';
const result = validateTpmjsField({
category: 'web-scraping',
description: 'Test description that is long enough',
example: 'const x = await tool.test()'
});
console.log(result); // Should be { valid: true, tier: 'minimal', ... }
```
---
### 1.3 NPM Client Package
**Reference:** See NPM_MIRROR.md "NPM Integration Strategy" section
- [ ] **Create `packages/npm-client`**
```bash
mkdir -p packages/npm-client/src
cd packages/npm-client
pnpm init
pnpm add zod
pnpm add -D typescript @types/node
```
- [ ] **Implement changes feed client**
- File: `packages/npm-client/src/changes.ts`
```typescript
export async function fetchChanges(since: string, limit = 100): Promise<{
results: Array<{ id: string; seq: string }>;
lastSeq: string;
}>
```
- Use endpoint: `https://replicate.npmjs.com/registry/_changes`
- Poll-based (no EventSource needed)
- [ ] **Implement keyword search**
- File: `packages/npm-client/src/search.ts`
```typescript
export async function searchByKeyword(
keyword: string,
size = 250,
from = 0
): Promise<Array<{ name: string }>>
```
- Use endpoint: `/-/v1/search?text=keywords:${keyword}`
- [ ] **Implement package metadata fetcher**
- File: `packages/npm-client/src/package.ts`
```typescript
export async function fetchPackageMetadata(packageName: string): Promise<{
name: string;
'dist-tags': { latest: string };
versions: Record<string, {
version: string;
description?: string;
tpmjs?: unknown;
// ... other fields
}>;
time: Record<string, string>;
} | null>
```
- Use endpoint: `https://registry.npmjs.org/${packageName}`
- [ ] **Implement download stats**
- File: `packages/npm-client/src/stats.ts`
```typescript
export async function fetchDownloadStats(
packageName: string
): Promise<number>
```
- Use endpoint: `https://api.npmjs.org/downloads/point/last-month/${packageName}`
- [ ] **Implement GitHub stats** (optional Phase 4)
- File: `packages/npm-client/src/github.ts`
```typescript
export async function fetchGithubStars(
repoUrl: string
): Promise<number>
```
- [ ] **Add rate limiting helper**
- File: `packages/npm-client/src/rate-limiter.ts`
- Simple delay between requests
- Exponential backoff on 429
- [ ] **Export all functions**
- File: `packages/npm-client/src/index.ts`
**Verification:**
```typescript
import { fetchPackageMetadata } from '@tpmjs/npm-client';
const pkg = await fetchPackageMetadata('express');
console.log(pkg?.name); // Should print 'express'
```
---
## 📋 Phase 2: Core API Routes (Week 2)
**Goal:** Build public API for searching/listing tools
### 2.1 Tool Search/List API
**Reference:** See NPM_MIRROR.md "API Routes" section
- [ ] **Create `apps/web/src/app/api/tools/route.ts`**
- Implement `GET` handler
- Query params: `q`, `category`, `official`, `limit`, `offset`
- Use Prisma to query `tools` table
- Return paginated results with metadata
- [ ] **Add full-text search**
- Use Postgres `ts_vector` for search
- Or simple `ILIKE` for MVP
- Search across: `npmPackageName`, `description`, `tags`
- [ ] **Add filtering**
- By `category`
- By `isOfficial`
- By `tier` (optional)
- [ ] **Add sorting**
- Default: `qualityScore DESC`, `npmDownloadsLastMonth DESC`
- Optional: `createdAt DESC`, `npmPackageName ASC`
**Verification:**
```bash
curl "http://localhost:3001/api/tools?q=web&limit=5"
# Should return JSON with tools array and pagination
```
---
### 2.2 Tool Detail API
- [ ] **Create `apps/web/src/app/api/tools/[id]/route.ts`**
- Implement `GET` handler
- Accept ID or package name
- Return full tool details
**Verification:**
```bash
curl "http://localhost:3001/api/tools/1"
# Should return single tool object
```
---
### 2.3 Validation API
- [ ] **Create `apps/web/src/app/api/tools/validate/route.ts`**
- Implement `POST` handler
- Accept JSON body with `tpmjs` field
- Use `@tpmjs/types` validator
- Return validation result with errors
**Verification:**
```bash
curl -X POST http://localhost:3001/api/tools/validate \
-H "Content-Type: application/json" \
-d '{"category":"web-scraping","description":"Test tool for validation","example":"const x = await tool.test()"}'
# Should return { valid: true, tier: "minimal" }
```
---
### 2.4 Stats API
- [ ] **Create `apps/web/src/app/api/stats/route.ts`**
- Implement `GET` handler
- Aggregate counts by category
- Total tools, official tools, etc.
**Verification:**
```bash
curl "http://localhost:3001/api/stats"
# Should return { totalTools: 0, officialTools: 0, categories: {} }
```
---
## 📋 Phase 3: Sync Workers (Week 2-3)
**Goal:** Implement automatic NPM package discovery
**Reference:** See NPM_MIRROR.md "NPM Integration Strategy" section
### 3.1 Changes Feed Sync
- [ ] **Create `apps/web/src/app/api/sync/changes/route.ts`**
- [ ] **Implement POST handler**
```typescript
export async function POST(request: Request) {
// 1. Verify CRON_SECRET header
// 2. Get last sequence from sync_checkpoints
// 3. Fetch changes from NPM (limit 100-500)
// 4. For each change:
// - Fetch package metadata
// - Check for tpmjs field
// - Validate with @tpmjs/types
// - Upsert to tools table
// - Log to sync_logs
// 5. Update checkpoint with new sequence
// 6. Return summary (processed, skipped, errors)
}
```
- [ ] **Add secret protection**
```typescript
const secret = request.headers.get('x-cron-secret');
if (secret !== process.env.CRON_SECRET) {
return new Response('Unauthorized', { status: 401 });
}
```
- [ ] **Add timeout protection**
- Limit processing to 50 packages per run
- Or 50 seconds max execution time
- Save checkpoint frequently
- [ ] **Add error handling**
- Try/catch around each package
- Log errors to `sync_logs`
- Continue processing other packages
**Verification:**
```bash
curl -X POST http://localhost:3001/api/sync/changes \
-H "x-cron-secret: your-secret"
# Should process changes and return summary
```
---
### 3.2 Keyword Search Sync
- [ ] **Create `apps/web/src/app/api/sync/keyword/route.ts`**
- [ ] **Implement POST handler**
```typescript
export async function POST(request: Request) {
// 1. Verify CRON_SECRET header
// 2. Search NPM for keyword 'tpmjs-tool'
// 3. For each result:
// - Fetch package metadata
// - Validate tpmjs field
// - Upsert with isOfficial=true
// - Log to sync_logs
// 4. Update checkpoint
// 5. Return summary
}
```
- [ ] **Handle pagination**
- NPM allows `size` up to 250
- May need multiple requests for all results
**Verification:**
```bash
curl -X POST http://localhost:3001/api/sync/keyword \
-H "x-cron-secret: your-secret"
# Should search and process keyword packages
```
---
### 3.3 Metrics Sync (Phase 4)
- [ ] **Create `apps/web/src/app/api/sync/metrics/route.ts`**
- [ ] **Implement POST handler**
```typescript
export async function POST(request: Request) {
// 1. Verify CRON_SECRET
// 2. Select tools to update (recent, popular, or sample)
// 3. For each tool:
// - Fetch NPM download stats
// - Fetch GitHub stars (if repo exists)
// - Calculate quality score
// - Update tools table
// 4. Update checkpoint
// 5. Return summary
}
```
**Verification:**
```bash
curl -X POST http://localhost:3001/api/sync/metrics \
-H "x-cron-secret: your-secret"
# Should update metrics for tools
```
---
### 3.4 Vercel Cron Configuration
- [ ] **Add to `vercel.json`**
```json
{
"crons": [
{
"path": "/api/sync/changes",
"schedule": "*/2 * * * *"
},
{
"path": "/api/sync/keyword",
"schedule": "*/15 * * * *"
},
{
"path": "/api/sync/metrics",
"schedule": "0 * * * *"
}
]
}
```
- [ ] **Set up environment variables in Vercel**
- `DATABASE_URL` - Neon connection string
- `CRON_SECRET` - Generate random secret
- `NPM_REGISTRY_URL` - https://registry.npmjs.org
- `NPM_CHANGES_URL` - https://replicate.npmjs.com/registry
---
## 📋 Phase 4: Frontend Integration (Week 3)
**Goal:** Replace mock data with real API calls
### 4.1 Update Tool Listing Page
- [ ] **Update `apps/web/src/app/tools/page.tsx`**
- Remove mock data import
- Fetch from `/api/tools`
- Add loading state
- Add error handling
- [ ] **Add search functionality**
- Search input component
- Debounced API calls
- Update URL with search params
- [ ] **Add category filter**
- Category dropdown/pills
- Filter API calls by category
- [ ] **Add pagination**
- Next/previous buttons
- Or infinite scroll
**Verification:**
- Visit http://localhost:3001/tools
- Should show real tools from database
- Search should work
- Filters should work
---
### 4.2 Update Tool Detail Page
- [ ] **Update `apps/web/src/app/tools/[id]/page.tsx`**
- Fetch from `/api/tools/[id]`
- Display all tool metadata
- Show rich tier fields if available
- [ ] **Add install instructions**
- npm install command
- Usage example from `tpmjs.example`
- [ ] **Add links**
- NPM package page
- GitHub repository
- Documentation
- Playground (if available)
**Verification:**
- Visit http://localhost:3001/tools/some-package
- Should show full tool details
---
### 4.3 Update Homepage
**Reference:** See NPM_MIRROR.md for stats display
- [ ] **Update stats in hero section**
- Fetch from `/api/stats`
- Show real tool count
- Show category breakdown
- [ ] **Update live metrics**
- Real download counts
- Real tool counts
- Update frequently (client-side polling or static)
**Verification:**
- Visit http://localhost:3001
- Stats should be real, not mock
---
## 📋 Phase 5: Testing & Polish (Week 4)
### 5.1 Create Test Packages
- [ ] **Publish 3-5 real NPM packages with `tpmjs` field**
- At least one with minimal tier
- At least one with rich tier
- Use `tpmjs-tool` keyword for official listing
- [ ] **Verify automatic discovery**
- Wait for next sync run
- Check they appear in database
- Check they appear on website
---
### 5.2 Documentation
- [ ] **Create docs section**
- `apps/web/src/app/docs/page.tsx`
- Getting started guide
- Schema reference
- Examples
- [ ] **Add validation playground**
- `apps/web/src/app/docs/validate/page.tsx`
- Form to test `tpmjs` field
- Real-time validation feedback
- Uses `/api/tools/validate`
---
### 5.3 CLI Tool (Optional)
- [ ] **Create `packages/cli`**
- Command: `tpmjs validate`
- Reads local `package.json`
- Validates `tpmjs` field
- Calls `/api/tools/validate`
---
### 5.4 Monitoring
- [ ] **Add health endpoint**
- `apps/web/src/app/api/health/route.ts`
- Check database connectivity
- Check sync status (last run times)
- [ ] **Set up uptime monitoring**
- Use UptimeRobot or Better Stack
- Monitor `/api/health`
- Alert if down or sync stale
- [ ] **Add error tracking**
- Set up Sentry for Next.js
- Track API errors
- Track sync errors
**Verification:**
```bash
curl http://localhost:3001/api/health
# Should return { status: "ok", db: "ok", sync: { ... } }
```
---
## 📋 Phase 6: Launch (Week 5)
### 6.1 Pre-Launch Checklist
- [ ] **Database**
- ✓ Prisma schema deployed
- ✓ Indexes created
- ✓ Backups enabled in Neon
- [ ] **Environment Variables**
- ✓ All secrets in Vercel
- ✓ `CRON_SECRET` set
- ✓ `DATABASE_URL` set
- [ ] **API Routes**
- ✓ All endpoints working
- ✓ Rate limiting added (optional)
- ✓ Error handling complete
- [ ] **Sync Workers**
- ✓ Changes feed running every 2 min
- ✓ Keyword search running every 15 min
- ✓ Checkpoints updating correctly
- [ ] **Frontend**
- ✓ All pages loading real data
- ✓ Search working
- ✓ Mobile responsive
- [ ] **Monitoring**
- ✓ Health check endpoint live
- ✓ Uptime monitoring active
- ✓ Error tracking active
---
### 6.2 Launch Steps
- [ ] **Deploy to production**
```bash
git push origin main
# Vercel auto-deploys
```
- [ ] **Verify deployment**
- Check all pages load
- Check API endpoints work
- Check cron jobs run
- [ ] **Publish announcement**
- Tweet/post about TPMJS
- Explain how to add `tpmjs` field
- Share validation endpoint
- [ ] **Monitor for 24 hours**
- Watch error logs
- Check sync is working
- Fix any issues
---
## 📋 Phase 7: Post-Launch (Ongoing)
### Enhancements
- [ ] **Semantic search**
- Add embeddings to tools table
- Use OpenAI/Cohere for semantic search
- [ ] **Usage analytics**
- Track tool views
- Track search queries
- Popular tools widget
- [ ] **Tool recommendations**
- "Similar tools" section
- "You might also like"
- [ ] **GitHub Actions**
- Validate `tpmjs` field in CI
- Auto-comment validation results
- [ ] **NPM webhooks**
- Listen for package updates
- Immediate sync (instead of polling)
---
## 🎯 Success Criteria
Check these metrics after launch:
### Week 1
- [ ] 10+ official tools listed
- [ ] All sync jobs running successfully
- [ ] Zero API errors
### Month 1
- [ ] 50+ official tools
- [ ] 5+ package authors using TPMJS
- [ ] <200ms API response time (p95)
### Month 3
- [ ] 200+ tools
- [ ] 20+ package authors
- [ ] Community contributions
### Month 6
- [ ] 1000+ tools
- [ ] 50+ active package authors
- [ ] Established as go-to AI tool registry
---
## 🔄 Ongoing Maintenance
Weekly:
- [ ] Check sync logs for errors
- [ ] Review new tools for quality
- [ ] Update documentation
Monthly:
- [ ] Database optimization (indexes, vacuum)
- [ ] Review and adjust quality scoring
- [ ] Update NPM_MIRROR.md with learnings
---
## 📚 Key Documents
**Read frequently during implementation:**
1. **NPM_MIRROR.md** - Complete architecture reference
- Database schema
- API specifications
- Validation rules
- Quality scoring
- All examples
2. **This checklist** - Implementation order and verification steps
3. **Plan file** - `.claude/plans/goofy-inventing-stearns.md` - Detailed planning notes
---
## 🚀 Ready to Build
This checklist is your complete implementation guide. Work through it phase by phase, checking off items as you go.
**Start with Phase 1, Step 1.1** and work sequentially. Each step has verification instructions to ensure it's working before moving on.
Good luck! 🎉

220
IMPLEMENTATION_STATUS.md
View file

@ -1,220 +0,0 @@
# Dynamic Tool Loading - Implementation Status
## ✅ Completed
### 1. Search Tool Package (`@tpmjs/search-registry`)
- ✅ Created package with AI SDK v6 JSON Schema format
- ✅ Connects to search API endpoint
- ✅ Returns tool metadata (packageName, exportName, version, importUrl)
- ✅ Fixed schema format (was using Zod, now uses jsonSchema)
- ✅ Location: `packages/tools/search-registry/`
### 2. Search API Endpoint (`/api/tools/search`)
- ✅ Implemented simple text-based search (BM25 had dependency issues)
- ✅ Searches by keywords in description, package name, keywords
- ✅ Returns tools with import URLs for esm.sh
- ✅ Location: `apps/web/src/app/api/tools/search/route.ts`
### 3. Pre-flight Tool Loading in Playground
- ✅ Automatic search on every user message
- ✅ Extracts user query from last message
- ✅ Calls searchTpmjsTools automatically
- ✅ Attempts to load discovered tools dynamically
- ✅ Location: `apps/playground/src/app/api/chat/route.ts`
### 4. Dynamic Tool Loader (Railway Service Approach)
- ✅ Updated to call Railway service instead of local imports
- ✅ Calls `/load-and-describe` endpoint to get tool schema
- ✅ Wraps tool with remote execution via `/execute-tool` endpoint
- ✅ Caches tool wrappers locally
- ✅ Location: `apps/playground/src/lib/dynamic-tool-loader.ts`
### 5. Documentation
- ✅ DYNAMIC_IMPORT_ISSUE.md - Comprehensive problem analysis
- ✅ RAILWAY_DYNAMIC_TOOL_LOADER.md - Railway implementation guide
- ✅ This file - Implementation status
## 🚧 Pending (Railway Service Implementation)
### Railway Service Endpoints Needed
You need to add these two endpoints to your existing Railway service:
#### 1. `POST /load-and-describe`
**Purpose**: Load a tool from esm.sh and return its schema
**Request**:
```json
{
"packageName": "firecrawl-aisdk",
"exportName": "webSearchTool",
"version": "0.7.2",
"importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2"
}
```
**Response**:
```json
{
"success": true,
"tool": {
"exportName": "webSearchTool",
"description": "Search the web using Firecrawl",
"inputSchema": {
"type": "object",
"properties": {
"query": { "type": "string" }
}
}
}
}
```
**Implementation Reference**: See `RAILWAY_DYNAMIC_TOOL_LOADER.md` for full code
#### 2. `POST /execute-tool`
**Purpose**: Execute a dynamically loaded tool with parameters
**Request**:
```json
{
"packageName": "firecrawl-aisdk",
"exportName": "webSearchTool",
"version": "0.7.2",
"importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2",
"params": {
"query": "latest AI news"
}
}
```
**Response**:
```json
{
"success": true,
"output": { "results": [...] },
"executionTimeMs": 1234
}
```
**Implementation Reference**: See `RAILWAY_DYNAMIC_TOOL_LOADER.md` for full code
### Deployment Requirements
1. **Railway Service**:
- Must run with `--experimental-network-imports` flag
- Add to start command: `node --experimental-network-imports server.js`
2. **Environment Variables** (Vercel):
```bash
RAILWAY_SERVICE_URL=https://your-railway-service.up.railway.app
# or reuse existing:
SANDBOX_EXECUTOR_URL=https://your-railway-service.up.railway.app
```
3. **Local Testing** (Railway service on port 3001):
```bash
RAILWAY_SERVICE_URL=http://localhost:3001
```
## 🎯 Testing Checklist
Once Railway endpoints are deployed:
- [ ] Test `/load-and-describe` endpoint directly with curl
- [ ] Test `/execute-tool` endpoint directly with curl
- [ ] Test full flow in playground:
- [ ] Ask: "search the web for latest AI news"
- [ ] Verify pre-flight search finds tools
- [ ] Verify tools load via Railway
- [ ] Verify tool execution works
- [ ] Check console logs for debugging info
## 📊 Current Flow
```
User: "search the web for latest AI news"
↓
Chat API extracts query
↓
Automatically calls searchTpmjsTools
↓
Search API returns matching tools
(packageName, exportName, version)
↓
loadToolsBatch() called for each tool
↓
For each tool:
1. Check local cache
2. If not cached:
→ POST to Railway: /load-and-describe
← Get back: description + inputSchema
3. Create wrapper tool with:
- description from Railway
- inputSchema from Railway
- execute() → calls Railway /execute-tool
4. Cache wrapper locally
↓
All tools available to agent
↓
Agent calls tool (wrapper)
↓
Wrapper → POST to Railway: /execute-tool
↓
Railway imports from esm.sh and executes
↓
Result returned to agent
↓
Agent uses result to answer user
```
## 🔍 Debugging
Check console logs for:
- `📦 Loading from Railway` - Tool loading initiated
- `✅ Tool loaded from Railway` - Tool schema received
- `🚀 Executing ... remotely` - Tool execution initiated
- `✅ Tool executed successfully` - Tool execution complete
- `❌ Railway service error` - Connection failed
- `❌ Failed to load tool` - Import failed
## 📁 Files Modified
1. `packages/tools/search-registry/src/index.ts` - Search tool
2. `packages/tools/search-registry/package.json` - AI SDK version
3. `apps/web/src/app/api/tools/search/route.ts` - Search endpoint
4. `apps/playground/src/app/api/chat/route.ts` - Pre-flight search
5. `apps/playground/src/lib/dynamic-tool-loader.ts` - Railway integration
6. `apps/playground/next.config.ts` - Added urlImports (unused)
7. `apps/playground/src/lib/tool-loader.ts` - Removed firecrawl
## 🚀 Next Steps
1. **Deploy Railway endpoints** using code from `RAILWAY_DYNAMIC_TOOL_LOADER.md`
2. **Set environment variables** in Vercel
3. **Test locally** with Railway service running on localhost:3001
4. **Deploy to production** and test with real tools
5. **Monitor logs** for any issues
## 💡 Key Insights
- **Next.js Limitation**: Cannot do dynamic HTTP imports due to bundler
- **Railway Solution**: Plain Node.js with `--experimental-network-imports`
- **Caching Strategy**: Two-level cache (local wrapper + Railway module)
- **Execution Model**: Remote execution in Railway, not Next.js
- **Security**: Tools execute in Railway sandbox, not Vercel
- **Performance**: First load ~1-2s (import), cached loads <10ms
## 📚 Related Documentation
- `DYNAMIC_IMPORT_ISSUE.md` - Problem analysis and ChatGPT response
- `RAILWAY_DYNAMIC_TOOL_LOADER.md` - Full Railway implementation guide
- Plan file: `~/.claude/plans/jiggly-inventing-dragon.md`
---
**Status**: Ready for Railway deployment
**Blocker**: Railway `/load-and-describe` and `/execute-tool` endpoints need implementation
**ETA**: 30-60 minutes to implement Railway endpoints + test

331
LAUNCH_REVIEW.md Normal file
View file

@ -0,0 +1,331 @@
# TPMJS Launch Review & Checklist
**STATUS: COMPLETED** - All critical issues have been fixed.
A comprehensive review of all public-facing content for Hacker News launch readiness.
---
## Executive Summary
**Overall Readiness: 7/10 - Needs Work Before Launch**
The website has excellent technical content and professional design, but fails the "5-second test" - a first-time visitor cannot quickly understand what TPMJS is or why they need it. The documentation is strong for existing users but assumes too much prior knowledge about AI agents and tooling.
### Critical Issues (Must Fix)
1. **Landing page doesn't explain what TPMJS is** - Hero section uses jargon without definition
2. **"Tool" vs "Package" never defined** - Core concepts assumed, not explained
3. **Knowledge gaps** - Assumes familiarity with AI agents, Zod, semantic search
4. **Category inconsistency** - HOW_TO_PUBLISH and NPM_MIRROR have different category lists
5. **NPM_MIRROR.md conflicts with other docs** - Appears outdated, creates confusion
### What's Working Well
- Publishing guide (HOW_TO_PUBLISH_A_TOOL.md) is excellent
- How It Works page has great technical depth
- Developer testimonials are concrete with real metrics
- No obvious AI-generated language on website
- Code examples are practical and well-placed
---
## The 5-Second Test: FAILED
**Question:** Can a developer understand what TPMJS is within 5 seconds of landing on the homepage?
**Answer:** No.
### What They See First
```
TOOL REGISTRY FOR AI AGENTS
Discover, share, and integrate tools that give your agents superpowers
```
### What's Missing
- What is a "tool" in this context?
- What is an "AI agent"?
- Why would I use this vs npm directly?
- Is this a package manager? A marketplace? An SDK?
### The "Aha Moment" is Unclear
A visitor still doesn't know:
- WHO should use TPMJS (tool builders? agent developers? both?)
- WHEN they would use it (at development time? runtime?)
- HOW it differs from regular npm packages
- WHY they can't just install packages normally
---
## Page-by-Page Clarity Ratings
| Page | Clarity | Human Feel | Issues |
|------|---------|------------|--------|
| **Landing Page** | 5/10 | Yes | No 5-second explanation, jargon-heavy |
| **Hero Section** | 3/10 | Yes | "Tool registry" undefined, circular language |
| **Problem Section** | 7/10 | Yes | Best section - concrete pain points |
| **Vision Section** | 5/10 | Yes | "Semantic search" unexplained |
| **Developer Stories** | 7/10 | Yes | Good metrics, but code unexplained |
| **Publish Section** | 6/10 | Yes | Assumes visitor is a tool builder |
| **How It Works** | 9/10 | Excellent | Minor density issues |
| **FAQ** | 8/10 | Yes | Missing some common questions |
| **Publish Guide** | 8.5/10 | Yes | Tier system could be clearer upfront |
| **Spec Page** | 8.5/10 | Yes | Assumes Zod/AI SDK knowledge |
| **Docs Page** | 9/10 | Excellent | Overwhelming length |
| **SDK Page** | 8.5/10 | Yes | Assumes Vercel AI SDK familiarity |
| **Privacy** | 8/10 | Yes | Hardcoded email address |
| **Terms** | 8/10 | Yes | Hardcoded date |
---
## Documentation Clarity Ratings
| Document | Clarity | Necessary | Critical Issues |
|----------|---------|-----------|-----------------|
| README.md | 8/10 | YES | Missing "what is TPMJS" explanation |
| HOW_TO_PUBLISH_A_TOOL.md | 9/10 | YES | Minor - excellent overall |
| DEPLOYMENT.md | 8/10 | YES | Confusing exit code explanation |
| QUALITY-GATES.md | 7/10 | OPTIONAL | Could merge into README |
| MANUAL_TOOLS.md | 8.5/10 | YES | Good for maintainers |
| NPM_MIRROR.md | 6.5/10 | **REMOVE** | **Conflicts with other docs, appears outdated** |
---
## Knowledge Gaps (Things Visitors Won't Understand)
### Not Explained Anywhere
1. **What is an "AI Agent"?** - The entire site assumes you know this
2. **What is a "Tool" vs a "Package"?** - Used interchangeably, never defined
3. **Why semantic search matters** - Just says "semantic" without explaining benefit
4. **What frameworks are supported** - Mentioned in FAQ but not prominently
5. **The Package → Tool relationship** - Can one package have multiple tools?
### Assumed Technical Knowledge
- Zod schemas (used throughout, never introduced)
- AI SDK tool format (referenced as "standard" but what standard?)
- esm.sh and Deno sandboxing (mentioned in How It Works)
- BM25 ranking algorithm (mentioned in docs)
### Missing Use Cases
- "Use TPMJS when..." section doesn't exist
- No comparison to alternatives (why not just npm?)
- No "before/after" showing the problem solved
---
## Human-Written Assessment
### Reads Like Human: YES ✓
- Developer stories use specific metrics ("500 lines to 3")
- Technical explanations show genuine understanding
- Problem section addresses real pain points
- No buzzword soup or meaningless marketing phrases
### Minor AI-Sounding Phrases Found
| Location | Phrase | Issue |
|----------|--------|-------|
| NPM_MIRROR.md:7 | "automated NPM-integrated registry" | Marketing speak |
| NPM_MIRROR.md:27 | "✨ Listed automatically" | Emoji in technical doc |
| NPM_MIRROR.md:500 | "Built with ❤️" | Remove emoji |
| HOW_TO_PUBLISH:389 | "AI-friendly descriptions" | Vague - what makes it "AI-friendly"? |
| Vision Section | "gives agents superpowers" | Metaphor without substance |
---
## Critical Inconsistencies Found
### Category Lists Don't Match
**HOW_TO_PUBLISH_A_TOOL.md says:**
```
text-analysis, code-generation, data-processing,
image-generation, audio-processing, search, integration, other
```
**NPM_MIRROR.md says:**
```
web-scraping, data-processing, file-operations, communication,
database, api-integration, image-processing, text-analysis,
automation, ai-ml, security, monitoring
```
**These are completely different!** Which is correct?
### Quality Score Formula Conflicts
- HOW_TO_PUBLISH: "Tier: Rich (1.0) > Basic (0.5) > Minimal (0.25)"
- MANUAL_TOOLS: "Rich tier tools get 4x quality score multiplier"
- NPM_MIRROR: Different formula entirely
### Field Names Inconsistent
- `exportName` used in MANUAL_TOOLS but not in HOW_TO_PUBLISH
- Deprecated fields (`parameters`, `returns`) mentioned but unclear when deprecated
---
## Hardcoded Values to Fix
| File | Issue | Line |
|------|-------|------|
| FAQ, Privacy, Terms | `thomasalwyndavis@gmail.com` hardcoded | Multiple |
| Privacy, Terms | Date "December 14, 2025" hardcoded | Multiple |
| Changelog page | Package list hardcoded in code | ~95-110 |
| Developer Stories | Fictional company names (Support.ai, DocFlow) | homePageData.ts |
---
## Launch Checklist
### Must Fix Before Launch (Blocking) - ALL DONE ✓
- [x] **Rewrite hero section** to explain TPMJS in one sentence
- Current: "TOOL REGISTRY FOR AI AGENTS"
- Suggested: "TPMJS lets AI agents discover and use npm packages as tools at runtime. Publish once to npm, get discovered automatically."
- [x] **Add "What is TPMJS?" section** to landing page
- Define: What is an AI agent?
- Define: What is a "tool" in this context?
- Explain: Why not just use npm directly?
- Show: 3-step "how it works" visual
- [x] **Reconcile category lists** between docs (deleted NPM_MIRROR.md)
- Pick one canonical list
- Update all docs to match
- Add categories to types package
- [x] **Delete or archive NPM_MIRROR.md** (deleted)
- Conflicts with HOW_TO_PUBLISH
- Appears to be old design doc, not current state
- Move to `/docs/internal/` if historical value
- [x] **Fix hardcoded values** (emails → hello@tpmjs.com, dates → December 2024)
- Email addresses → environment variable
- Dates → dynamic or remove
- Package lists → generated from filesystem
### Should Fix (High Priority) - MOSTLY DONE
- [x] **Add "Use TPMJS when..." section** to landing page (covered in "What is TPMJS?" section)
- List concrete scenarios: "Building a chatbot that needs web access"
- "Agent that processes different file formats"
- "Tool that should be discoverable by other agents"
- [x] **Explain Package vs Tool distinction** (covered in "What is TPMJS?" section)
- Add glossary or definitions section
- Clarify: 1 package can have N tools
- [x] **Add framework compatibility section** (mentioned in hero and publish sections)
- Which AI frameworks work with TPMJS?
- Are there adapters needed?
- Show code for each framework
- [ ] **Simplify developer stories code**
- Current code snippet unexplained:
```js
const agent = new Agent({ tools: await tpmjs.search(...) })
```
- Add: Where does `Agent` come from? What's happening here?
- [x] **Add README context** (completely rewritten with clear explanation)
- What is TPMJS for?
- Link to tpmjs.com
- Explain discovery mechanism
### Nice to Have (Post-Launch)
- [ ] Add video walkthrough (30-60 seconds)
- [ ] Interactive playground link from homepage
- [ ] "Compare to alternatives" section
- [ ] Case studies with real company names
- [ ] Quick links sidebar for docs page
- [ ] Status badges for each quality gate
---
## Recommended Hero Section Rewrite
### Current
```
TOOL REGISTRY FOR AI AGENTS
Discover, share, and integrate tools that give your agents superpowers
The registry for AI tools
```
### Suggested
```
MAKE YOUR AI AGENT SMARTER
TPMJS connects your AI agent to 2,500+ npm packages at runtime.
No config files. No manual imports. Just describe what you need.
"Find me a tool that can scrape websites" → Your agent gets web-scraper
"I need to process markdown" → Your agent gets markdown-formatter
Publish your npm package → It's discoverable by every AI agent in 15 minutes.
```
This version:
- Explains what it DOES (connects agents to npm packages)
- Shows HOW it works (natural language → tool)
- States the VALUE (no config, automatic discovery)
- Gives concrete examples
---
## Recommended "What is TPMJS?" Section
Add after hero, before featured tools:
```markdown
## What is TPMJS?
**The Problem:** AI agents need tools (web scraping, file processing, API calls)
but developers must manually configure each one. As the ecosystem grows,
this becomes unmanageable.
**The Solution:** TPMJS is a registry that automatically discovers npm packages
designed for AI agents. Agents can search for tools by description and load them
at runtime.
**For Tool Builders:** Add `tpmjs-tool` keyword to your package.json.
Your tool appears on tpmjs.com within 15 minutes.
**For Agent Developers:** Use semantic search to find tools:
```javascript
import { searchRegistry } from '@tpmjs/sdk';
const tools = await searchRegistry('send emails and slack messages');
// Returns: email-sender, slack-notifier, ...
```
**One registry. Thousands of tools. Zero configuration.**
```
---
## Final Assessment
### Ready for Launch?
**Not yet.** The core product is solid but messaging fails first-time visitors.
### Estimated Fixes
- Hero rewrite: 30 minutes
- "What is TPMJS?" section: 1 hour
- Category reconciliation: 1 hour
- Hardcoded values: 30 minutes
- README updates: 30 minutes
- NPM_MIRROR cleanup: 15 minutes
**Total: ~4 hours of work**
### After Fixes
The site will be launch-ready. The technical content is excellent - it just needs a better front door.
---
## Appendix: Positive Highlights
Things that are already great and should NOT change:
1. **How It Works page** - Excellent technical depth, clear structure
2. **Publishing guide** - Best-in-class documentation, real examples
3. **Problem section** - Concrete pain points, relatable issues
4. **Spec page** - Clear field reference, good validation info
5. **SDK documentation** - Quick start is excellent
6. **Code examples throughout** - Practical, copy-pasteable
7. **Visual design** - Clean, professional, developer-focused
8. **Quality scoring explanation** - Transparent, well-documented

21
LICENSE Normal file
View file

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024-2025 TPMJS
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

500
NPM_MIRROR.md
View file

@ -1,500 +0,0 @@
# TPMJS NPM-Integrated Registry Architecture
> **Automated tool discovery from NPM with zero-click submission**
## Vision
Transform TPMJS from a manual directory into an **automated NPM-integrated registry** where package authors simply publish to NPM with a `tpmjs` field in their `package.json` and their tools are discovered and listed within seconds—no manual submission, no forms, no waiting.
## Quick Start for Package Authors
```json
{
"name": "my-awesome-tool",
"version": "1.0.0",
"keywords": ["tpmjs-tool"],
"tpmjs": {
"category": "web-scraping",
"description": "Extract product data from e-commerce websites with ease",
"example": "const data = await scraper.extract('https://shop.com')"
}
}
```
```bash
npm publish
# ✨ Listed automatically within 15 minutes (keyword) or seconds (changes feed)
```
---
## Architecture Overview
```
NPM Ecosystem
↓
Changes Feed + Keyword Search
↓
Package Validator (Zod)
↓
PostgreSQL Database
↓
Next.js API Routes
↓
TPMJS Web App
```
### Core Components
1. **NPM Sync Service** (Node.js) - Monitors NPM registry for new packages
2. **PostgreSQL Database** - Stores validated tool metadata
3. **Next.js API** - Serves tool data with search/filtering
4. **Web Frontend** - Browse, search, and discover tools
---
## Discovery Mechanism: Hybrid Approach
### Method 1: Keyword Search (Official)
- Search NPM for packages with `tpmjs-tool` keyword
- Runs every 15 minutes via cron
- Packages marked as "Official"
### Method 2: Changes Feed (Automatic)
- Monitors `replicate.npmjs.com/registry/_changes` in real-time
- Detects packages with `tpmjs` field instantly
- Packages marked as "Community" (unless they also have keyword)
### Why Hybrid?
- **Keywords** = Clear opt-in, queryable, respects NPM conventions
- **Changes Feed** = Real-time, catches packages without keywords
- **Together** = Best discoverability with fallback
---
## The "tpmjs" Field: Tiered Schema
### Minimal Tier (Required)
```json
{
"tpmjs": {
"category": "web-scraping",
"description": "Extract structured data from websites using CSS selectors",
"example": "const data = await tool.scrape({ url: 'https://example.com', selector: '.price' })"
}
}
```
**Categories:**
- web-scraping
- data-processing
- file-operations
- communication
- database
- api-integration
- image-processing
- text-analysis
- automation
- ai-ml
- security
- monitoring
### Rich Tier (Optional)
Extend with any of these optional fields:
```json
{
"tpmjs": {
// ... Required fields ...
"parameters": [
{
"name": "url",
"type": "string",
"description": "Target URL to scrape",
"required": true
}
],
"returns": {
"type": "object",
"description": "Extracted data matching the selector"
},
"authentication": {
"required": false,
"type": "api-key",
"envVar": "SCRAPER_API_KEY",
"docsUrl": "https://docs.example.com/auth"
},
"pricing": {
"model": "freemium",
"freeLimit": "100 requests/month",
"paidUrl": "https://example.com/pricing"
},
"frameworks": ["vercel-ai", "langchain", "llamaindex"],
"links": {
"documentation": "https://docs.example.com",
"playground": "https://example.com/try",
"repository": "https://github.com/user/repo"
},
"tags": ["web", "scraping", "html", "css"],
"status": "stable",
"aiAgent": {
"useCase": "Use when agent needs to extract data from websites",
"limitations": "Cannot handle JavaScript-heavy SPAs"
}
}
}
```
---
## Database Schema
### Tools Table
```sql
CREATE TABLE tools (
-- NPM Metadata
npm_package_name VARCHAR(214) UNIQUE NOT NULL,
npm_version VARCHAR(50) NOT NULL,
npm_published_at TIMESTAMP NOT NULL,
npm_description TEXT,
npm_repository JSONB,
npm_homepage TEXT,
npm_license VARCHAR(50),
-- TPMJS Metadata
category VARCHAR(50) NOT NULL,
description TEXT NOT NULL,
example TEXT NOT NULL,
parameters JSONB,
authentication JSONB,
pricing JSONB,
frameworks TEXT[],
links JSONB,
tags TEXT[],
status VARCHAR(20),
-- Discovery
discovery_method VARCHAR(20) NOT NULL, -- 'keyword' | 'changes-feed'
is_official BOOLEAN DEFAULT false,
tier VARCHAR(20) NOT NULL, -- 'minimal' | 'rich'
-- Metrics
npm_downloads_last_month INTEGER DEFAULT 0,
github_stars INTEGER DEFAULT 0,
quality_score DECIMAL(3,2), -- 0.00 to 1.00
-- Timestamps
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
```
---
## Sync Service Architecture
### Workers
**1. Changes Feed Worker**
- Connects to `replicate.npmjs.com/registry/_changes`
- Receives real-time change events
- Fetches package metadata for each change
- Checks for `tpmjs` field
- Validates and inserts to database
**2. Keyword Search Worker**
- Runs every 15 minutes (cron)
- Searches `/-/v1/search?text=keywords:tpmjs-tool`
- Processes all results
- Marks as "Official"
**3. Metrics Worker** (Optional Phase 4)
- Updates download counts from NPM API
- Fetches GitHub stars
- Calculates quality scores
### Package Processing Pipeline
```
1. Fetch package metadata from NPM
2. Extract `tpmjs` field from latest version
3. Validate against Zod schema
4. If valid → Insert/Update database
5. If invalid → Log error
6. If no field → Skip
```
---
## API Routes
### GET /api/tools
Search and list tools
**Query Parameters:**
- `q` - Search query
- `category` - Filter by category
- `official` - Only official tools (true/false)
- `limit` - Results per page (default 20)
- `offset` - Pagination offset
**Response:**
```json
{
"tools": [...],
"pagination": {
"total": 150,
"limit": 20,
"offset": 0,
"hasMore": true
}
}
```
### GET /api/tools/[id]
Get tool details by ID
### POST /api/tools/validate
Validate a `tpmjs` field before publishing
**Request:**
```json
{
"category": "web-scraping",
"description": "...",
"example": "..."
}
```
**Response:**
```json
{
"valid": true,
"tier": "minimal",
"errors": []
}
```
### GET /api/stats
Registry statistics
```json
{
"totalTools": 2847,
"officialTools": 150,
"categories": {
"web-scraping": 320,
"communication": 280,
...
}
}
```
---
## Quality Scoring Algorithm
Tools are scored 0.00 to 1.00 based on:
- **Base validity** (0.3) - Has valid schema
- **Tier** (0.1-0.2) - Rich tier > Minimal tier
- **NPM downloads** (0.2) - Based on monthly downloads
- **GitHub stars** (0.15) - Repository popularity
- **Documentation** (0.1) - Has docs URL
- **Example quality** (0.05) - Example length > 100 chars
Score is used for default sorting and quality indicators.
---
## Implementation Phases
### Phase 1: Foundation (Week 1-2)
- Set up PostgreSQL + Prisma
- Create Zod schemas in `@tpmjs/types`
- Build sync service structure
- Implement NPM API client
### Phase 2: Discovery (Week 2-3)
- Implement changes feed worker
- Implement keyword search worker
- Deploy sync service (Railway/Fly.io)
- Test with real packages
### Phase 3: API & Frontend (Week 3-4)
- Build Next.js API routes
- Update tool listing page
- Update tool detail pages
- Add validation endpoint
### Phase 4: Polish (Week 4-5)
- Add metrics worker
- Create documentation
- Build CLI validator
- Launch to community
### Phase 5: Enhancements (Post-Launch)
- Semantic search (embeddings)
- Usage analytics
- Tool recommendations
- GitHub Actions integration
---
## Infrastructure Requirements
### Sync Service
- **Platform:** Railway or Fly.io
- **Runtime:** Node.js 22+
- **Resources:** 512MB RAM, 1 CPU
- **Cost:** ~$5-10/month
### Database
- **Platform:** Neon Postgres (serverless)
- **Size:** Free tier (start), scale as needed
- **Backups:** Automatic with Neon
- **Cost:** Free tier available, ~$10-20/month for production
### Web App
- **Platform:** Vercel (existing)
- **No changes required**
---
## Monitoring & Health
### Metrics to Track
1. **Sync Health**
- Changes feed uptime
- Packages processed per hour
- Validation success rate
2. **Database**
- Total tools
- Official vs community ratio
- Tier distribution
3. **API**
- Request latency (p95 < 200ms)
- Search performance
- Error rates
### Alerts
- Sync service down > 5 minutes
- Database connection failures
- Validation error rate > 10%
---
## Developer Experience
### Validation Before Publishing
```bash
# Using TPMJS CLI (to be built)
npx tpmjs validate
# Or via API
curl -X POST https://tpmjs.com/api/tools/validate \
-H "Content-Type: application/json" \
-d '{"category":"web-scraping","description":"...","example":"..."}'
```
### Documentation Pages Needed
1. **Getting Started** - Adding TPMJS support
2. **Schema Reference** - Complete field docs
3. **Best Practices** - Tips for quality tools
4. **Examples** - Sample configurations
5. **FAQ** - Common questions
---
## Migration from Mock Data
### Current State
- 12 mock tools in `toolData.ts`
- Client-side search
- Hard-coded categories
### Migration Strategy
1. **Publish Real Packages**
- Create NPM packages for mock tools
- Add `tpmjs` fields
- Publish with `tpmjs-tool` keyword
2. **Update Frontend**
- Replace mock data with API calls
- Keep existing UI components
- Update types to match Prisma models
3. **Gradual Rollout**
- Dual mode (mock + real)
- Real data primary, mock fallback
- Remove mock entirely
---
## Success Metrics
### Technical
- ✓ Discovery latency < 60 seconds
- ✓ API response time < 200ms p95
- ✓ Support 10,000+ tools
- ✓ 99.9% uptime
### User Experience
- ✓ 0-click submission (automatic)
- ✓ Instant validation feedback
- ✓ <100ms search speed
- ✓ 100% mobile features
### Business
- Week 1: 10 official tools
- Month 1: 50 official tools
- Month 3: 200+ tools
- Month 6: 1000+ tools
- 50+ active package authors
---
## Comparison to Vercel's Approach
| Feature | Vercel AI SDK | TPMJS |
|---------|---------------|-------|
| **Submission** | Manual file edit + PR | Automatic via NPM |
| **Discovery** | None | Real-time changes feed |
| **Validation** | Manual review | Automated Zod schema |
| **Updates** | New PR required | Automatic on publish |
| **Search** | Static array | Full-text + categories |
| **Scale** | 6 tools | 1000+ tools ready |
---
## Next Steps
1. Review this architecture plan
2. Approve database schema and API design
3. Set up infrastructure (Railway + Postgres)
4. Start Phase 1: Foundation
5. Launch MVP in 4-5 weeks
---
## References
- [NPM Registry API Docs](https://github.com/npm/registry/blob/main/docs/REGISTRY-API.md)
- [NPM Changes Feed](https://github.com/npm/registry/blob/main/docs/REPLICATE-API.md)
- [Vercel AI Tools Registry](https://github.com/vercel/ai/blob/main/content/tools-registry/registry.ts)
- [TPMJS Architecture Plan](/.claude/plans/goofy-inventing-stearns.md) (Full details)
---
**Built with ❤️ for the AI agent ecosystem**

522
OPENAI_SCHEMA_ERROR.md
View file

@ -1,522 +0,0 @@
# OpenAI Schema Validation Error - AI SDK v6
## ✅ RESOLVED
**Solution:** Use `tool()` and `jsonSchema()` from AI SDK instead of Zod for tool definitions.
## Error Message
```
Error [AI_APICallError]: Invalid schema for function 'helloWorld': schema must be a JSON Schema of 'type: "object"', got 'type: "None"'.
```
## Context
Building a Next.js playground app to test AI SDK v6 tool execution with OpenAI's GPT-4o-mini model. The error occurs when OpenAI validates the tool schema sent in the API request.
## Root Cause
Zod 4.0.0 generates JSON Schema with `allOf` + `$ref` at the root level instead of a direct `type: "object"`. OpenAI's API requires a JSON Schema with `type: "object"` at the root, so it rejects Zod 4 schemas with `type: "None"` error.
## Environment
- **AI SDK Version**: `ai@6.0.0-beta.124`
- **OpenAI Provider**: `@ai-sdk/openai@3.0.0-beta.74`
- **OpenAI Library**: `openai@^6.9.1`
- **Zod Version**: `zod@^4.0.0`
- **Next.js Version**: `next@^16.0.4`
- **Node.js**: Latest
- **TypeScript**: Strict mode enabled
## Tool Definition
Located at: `packages/tools/hello/src/index.ts`
```typescript
import { z } from 'zod';
/**
* Hello World Tool
* Returns a simple "Hello, World!" greeting
*
* This is a proper AI SDK v6 tool that can be used with streamText()
*/
export const helloWorldTool = {
description: 'Returns a simple "Hello, World!" greeting message',
parameters: z.object({
// OpenAI requires at least one optional parameter, can't be completely empty
includeTimestamp: z.boolean().optional().describe('Whether to include a timestamp in the response'),
}),
execute: async ({ includeTimestamp = true }: { includeTimestamp?: boolean }) => {
const response: any = {
message: 'Hello, World!',
};
if (includeTimestamp) {
response.timestamp = new Date().toISOString();
}
return response;
},
};
/**
* Hello Name Tool
* Returns a personalized greeting with the provided name
*
* This is a proper AI SDK v6 tool that can be used with streamText()
*/
export const helloNameTool = {
description: 'Returns a personalized greeting with the provided name',
parameters: z.object({
name: z.string().describe('The name of the person to greet'),
}),
execute: async ({ name }: { name: string }) => {
return {
message: `Hello, ${name}!`,
timestamp: new Date().toISOString(),
};
},
};
```
## Tool Loading
Located at: `apps/playground/src/lib/tool-loader.ts`
```typescript
// Static imports for tools (required for Next.js/webpack)
import { helloWorldTool, helloNameTool } from '@tpmjs/hello';
import { scrapeTool, crawlTool, searchTool } from 'firecrawl-aisdk';
/**
* Load a specific TPMJS tool by package name
*/
export async function loadTpmjsTool(packageName: string): Promise<any> {
try {
// Map package names to their tool functions
switch (packageName) {
case '@tpmjs/hello':
// Hello has multiple tools, return all of them
return {
helloWorld: helloWorldTool,
helloName: helloNameTool,
};
case 'firecrawl-aisdk':
// Firecrawl has multiple tools, return all of them
return {
scrapeTool,
crawlTool,
searchTool,
};
default:
throw new Error(`Unknown tool package: ${packageName}`);
}
} catch (error) {
if (error instanceof Error) {
throw new Error(`Failed to load tool from package ${packageName}: ${error.message}`);
}
throw new Error(`Failed to load tool from package ${packageName}: Unknown error`);
}
}
/**
* Load all installed TPMJS tools
*/
export async function loadAllTools(): Promise<Record<string, any>> {
const installedTools = ['@tpmjs/hello', 'firecrawl-aisdk'];
const tools: Record<string, any> = {};
for (const packageName of installedTools) {
try {
const tool = await loadTpmjsTool(packageName);
// If the tool returns an object with multiple tools (like firecrawl), spread them
if (tool && typeof tool === 'object' && !tool.description) {
Object.assign(tools, tool);
} else {
// Single tool - use a cleaned name (remove hyphens, camelCase)
const toolName = packageName.replace(/-([a-z])/g, (_match, letter) => letter.toUpperCase()).replace(/-/g, '');
tools[toolName] = tool;
}
} catch (error) {
console.error(`Failed to load tool ${packageName}:`, error);
// Continue loading other tools even if one fails
}
}
return tools;
}
```
## API Route
Located at: `apps/playground/src/app/api/chat/route.ts`
```typescript
import { loadAllTools } from '~/lib/tool-loader';
import { openai } from '@ai-sdk/openai';
import { streamText } from 'ai';
import { NextRequest } from 'next/server';
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const maxDuration = 60;
export async function POST(request: NextRequest) {
try {
const body = await request.json();
const { messages } = body;
if (!messages || !Array.isArray(messages)) {
return new Response(JSON.stringify({ error: 'Invalid request: messages array required' }), {
status: 400,
headers: { 'Content-Type': 'application/json' },
});
}
// Load all available tools
const tools = await loadAllTools();
console.log('Loaded tools:', Object.keys(tools));
// Create system message
const systemMessage = {
role: 'system' as const,
content: `You are a helpful AI assistant that can use TPMJS tools to help users.
Available tools:
${Object.entries(tools)
.map(([name, tool]) => `- ${name}: ${tool.description}`)
.join('\n')}
Call tools as needed to answer user questions. Execute tools directly.`,
};
// Stream the AI response with tools
const result = streamText({
model: openai('gpt-4o-mini'),
messages: [systemMessage, ...messages],
tools,
maxSteps: 5,
});
return result.toTextStreamResponse();
} catch (error) {
console.error('Chat API error:', error);
return new Response(
JSON.stringify({
error: error instanceof Error ? error.message : 'Unknown error occurred',
}),
{
status: 500,
headers: { 'Content-Type': 'application/json' },
}
);
}
}
```
## Package Configuration
Located at: `packages/tools/hello/package.json`
```json
{
"name": "@tpmjs/hello",
"version": "0.0.1",
"private": true,
"description": "Example TPMJS tools - Hello World and Hello Name",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"clean": "rm -rf dist",
"type-check": "tsc --noEmit"
},
"keywords": [
"tpmjs-tool",
"ai-sdk",
"hello",
"example"
],
"tpmjs": {
"category": "text-analysis",
"description": "Simple greeting tools - Hello World and personalized Hello Name greetings"
},
"dependencies": {
"ai": "6.0.0-beta.124",
"zod": "^4.0.0"
},
"devDependencies": {
"@tpmjs/tsconfig": "workspace:*",
"typescript": "^5.9.3"
},
"files": [
"dist",
"README.md"
]
}
```
## TypeScript Configuration
Located at: `packages/tools/hello/tsconfig.json`
```json
{
"compilerOptions": {
"target": "ES2020",
"module": "commonjs",
"lib": ["ES2020"],
"outDir": "./dist",
"rootDir": "./src",
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"moduleResolution": "node",
"resolveJsonModule": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}
```
## Compiled Output
Located at: `packages/tools/hello/dist/index.js`
```javascript
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.helloNameTool = exports.helloWorldTool = void 0;
const zod_1 = require("zod");
/**
* Hello World Tool
* Returns a simple "Hello, World!" greeting
*
* This is a proper AI SDK v6 tool that can be used with streamText()
*/
exports.helloWorldTool = {
description: 'Returns a simple "Hello, World!" greeting message',
parameters: zod_1.z.object({
// OpenAI requires at least one optional parameter, can't be completely empty
includeTimestamp: zod_1.z.boolean().optional().describe('Whether to include a timestamp in the response'),
}),
execute: async ({ includeTimestamp = true }) => {
const response = {
message: 'Hello, World!',
};
if (includeTimestamp) {
response.timestamp = new Date().toISOString();
}
return response;
},
};
/**
* Hello Name Tool
* Returns a personalized greeting with the provided name
*
* This is a proper AI SDK v6 tool that can be used with streamTime()
*/
exports.helloNameTool = {
description: 'Returns a personalized greeting with the provided name',
parameters: zod_1.z.object({
name: zod_1.z.string().describe('The name of the person to greet'),
}),
execute: async ({ name }) => {
return {
message: `Hello, ${name}!`,
timestamp: new Date().toISOString(),
};
},
};
```
## Full Error Response from OpenAI
```json
{
"error": {
"message": "Invalid schema for function 'helloWorld': schema must be a JSON Schema of 'type: \"object\"', got 'type: \"None\"'.",
"type": "invalid_request_error",
"param": "tools[0].parameters",
"code": "invalid_function_parameters"
}
}
```
API endpoint: `https://api.openai.com/v1/responses`
Status code: 400
## Problem Analysis
1. **OpenAI expects JSON Schema format** - The `tools[0].parameters` field must be a valid JSON Schema object with `type: "object"`
2. **AI SDK v6 should convert Zod to JSON Schema** - The AI SDK is supposed to automatically convert Zod schemas to JSON Schema when sending to OpenAI, but it's producing `type: "None"` instead
3. **Potential causes**:
- Zod 4.0.0 compatibility issue with AI SDK v6 beta
- AI SDK not properly converting the Zod schema
- Issue with how the tool object is structured
- Problem with how tools are passed to `streamText()`
4. **Already tried**:
- Added at least one parameter (even optional) to helloWorldTool
- Used proper Zod schema with `.describe()` for descriptions
- Followed AI SDK v6 tool definition format exactly
- Built the package successfully (dist folder exists)
## AI SDK v6 Tool Format Reference
According to AI SDK v6 documentation, a tool should be defined as:
```typescript
{
description: string,
parameters: ZodSchema,
execute: async (args) => Promise<any>
}
```
This matches our implementation exactly.
## Questions for ChatGPT
1. Is there a known compatibility issue between AI SDK v6 Beta (6.0.0-beta.124) and Zod 4.0.0?
2. Does the AI SDK v6 require a specific tool registration format when passing to `streamText()`?
3. Should tools be wrapped in a different structure (e.g., using `tool()` helper function)?
4. Is there a way to manually convert Zod schema to JSON Schema that OpenAI accepts?
5. Are there any known issues with using workspace packages (`@tpmjs/hello`) in Next.js API routes with dynamic imports?
6. Should we downgrade to Zod 3.x instead of Zod 4.0.0?
7. Is there a debug mode to see what JSON Schema is being sent to OpenAI?
## Additional Context
- The `firecrawl-aisdk` package works correctly with the same setup
- Build process completes successfully with no TypeScript errors
- The tool is being loaded and passed to `streamText()` correctly
- Error only occurs when OpenAI validates the tool schema
- This is a monorepo using pnpm workspaces and Turborepo
## Related Files
- Tool definition: `packages/tools/hello/src/index.ts`
- Tool loader: `apps/playground/src/lib/tool-loader.ts`
- API route: `apps/playground/src/app/api/chat/route.ts`
- Package config: `packages/tools/hello/package.json`
- Compiled output: `packages/tools/hello/dist/index.js`
## Expected Behavior
Tools should be automatically converted from Zod schema to JSON Schema by AI SDK v6 and accepted by OpenAI's API.
## Actual Behavior
OpenAI rejects the tool schema with error: `got 'type: "None"'` instead of a valid JSON Schema object.
---
## ✅ SOLUTION IMPLEMENTED
### What We Changed
Instead of using Zod schemas with `parameters`, we now use AI SDK's `tool()` helper with `jsonSchema()` for the input schema. This bypasses Zod's JSON Schema conversion entirely.
### Before (Broken with Zod 4)
```typescript
import { z } from 'zod';
export const helloWorldTool = {
description: 'Returns a simple "Hello, World!" greeting message',
parameters: z.object({
includeTimestamp: z.boolean().optional().describe('Whether to include a timestamp'),
}),
execute: async ({ includeTimestamp = true }) => {
// ...
},
};
```
### After (Working with jsonSchema)
```typescript
import { jsonSchema, tool } from 'ai';
type HelloWorldInput = {
includeTimestamp?: boolean;
};
export const helloWorldTool = tool({
description: 'Returns a simple "Hello, World!" greeting message',
inputSchema: jsonSchema<HelloWorldInput>({
type: 'object',
properties: {
includeTimestamp: {
type: 'boolean',
description: 'Whether to include a timestamp in the response',
},
},
additionalProperties: false,
}),
async execute({ includeTimestamp = true }) {
const response: any = {
message: 'Hello, World!',
};
if (includeTimestamp) {
response.timestamp = new Date().toISOString();
}
return response;
},
});
```
### Key Changes
1. **Import from `ai`**: Added `jsonSchema` and `tool` imports
2. **Define TypeScript types**: Created `HelloWorldInput` type for type safety
3. **Use `tool()` wrapper**: Wraps the entire tool definition
4. **Use `jsonSchema()` for schema**: Provides explicit JSON Schema with `type: "object"` at root
5. **Removed Zod dependency**: No longer need `zod` in package.json
### Benefits
- ✅ Works with OpenAI's strict schema validation
- ✅ Explicit control over JSON Schema structure
- ✅ Full TypeScript type safety with generic types
- ✅ No dependency on Zod (one less package to maintain)
- ✅ Follows AI SDK v6 best practices
- ✅ Guaranteed `type: "object"` at root level
### Updated Package Dependencies
```json
{
"dependencies": {
"ai": "6.0.0-beta.124"
}
}
```
Zod is no longer needed in tool packages that use `jsonSchema()`.
### References
- [AI SDK Core: tool](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool)
- [AI SDK Core: jsonSchema](https://ai-sdk.dev/docs/reference/ai-sdk-core/json-schema)
- [GitHub Issue: Zod 4 JSON Schema compatibility](https://github.com/vercel/ai/issues/10240)

98
RAILWAY_DEPLOYMENT_NOTE.md
View file

@ -1,98 +0,0 @@
# Railway Executor - Deployment Status
## Issue Discovered
Node.js does not support HTTP(S) imports by default, even with `--experimental-network-imports` flag (that flag doesn't exist in current Node versions).
## Solutions Considered
1. **Custom ESM Loader** - Complex, requires Node.js 18.19+ with `--loader` flag
2. **fetch + eval** - Security concerns, doesn't handle ES modules properly
3. **Bundler approach** - Would defeat the purpose of dynamic imports
4. **Deno** - Supports HTTP imports natively, but different ecosystem
## Recommended Solution
Since the core issue is that we need truly dynamic runtime imports from HTTP URLs, and Node.js doesn't support this, we have **two viable paths**:
### Option A: Use Deno on Railway (RECOMMENDED)
Deno supports HTTP imports natively:
```typescript
// server.ts (Deno)
import { serve } from "https://deno.land/std@0.208.0/http/server.ts";
const moduleCache = new Map();
async function loadTool(url: string, exportName: string) {
if (moduleCache.has(url)) {
return moduleCache.get(url);
}
// Deno supports this natively!
const module = await import(url);
const tool = module[exportName];
moduleCache.set(url, tool);
return tool;
}
serve(async (req) => {
// ... handle requests
}, { port: 3002 });
```
**Deploy to Railway:**
```bash
# In Railway dashboard:
# - Set Start Command: deno run --allow-net --allow-env server.ts
# - Or use railway.json with deno runtime
```
### Option B: Pre-build Bundle Approach
Instead of truly dynamic imports, pre-fetch and cache tools:
1. Playground searches for tools
2. Backend fetches tool code once and caches it
3. Use `vm2` or similar to execute in sandbox
4. Not truly "dynamic" but works with Node.js
## Current Status
The Railway executor service is **created** but **not deployed** because Node.js doesn't support the required HTTP imports.
**Files created:**
- `apps/railway-executor/package.json`
- `apps/railway-executor/server.js` (incomplete - needs Deno or vm2 approach)
- `apps/railway-executor/README.md`
## Next Steps
**If using Deno (recommended):**
1. Rewrite server.js as server.ts for Deno
2. Deploy to Railway with Deno runtime
3. Test HTTP imports work
4. Update playground to use Railway URL
**If sticking with Node.js:**
1. Install `vm2` package for sandboxed execution
2. Implement fetch + vm2 approach
3. Deploy to Railway
4. Accept limitations (less dynamic, more complex)
## Alternative: Skip Railway, Use Different Architecture
Since the original issue is Next.js bundler limitations, consider:
**Web Workers in Browser** - Load tools client-side using native `import()`
- Pros: No server needed, truly dynamic
- Cons: Exposes API keys, security concerns
**Serverless Functions with Pre-installed Tools** - Deploy each tool as separate function
- Pros: Works with Vercel/Next.js
- Cons: Not truly dynamic, requires redeployment for new tools
---
**Recommendation**: Use Deno on Railway. It's designed for exactly this use case.

376
RAILWAY_DYNAMIC_TOOL_LOADER.md
View file

@ -1,376 +0,0 @@
# Railway Service - Dynamic Tool Loader Implementation
## Overview
This document describes the Railway service implementation needed to support dynamic tool loading from esm.sh in the TPMJS playground.
## Why Railway Service?
Next.js/Turbopack intercepts all `import()` calls and tries to resolve them through its module graph. HTTP URLs like `https://esm.sh/...` are not supported.
**Solution**: Use a plain Node.js service on Railway that:
- Runs with `--experimental-network-imports` flag
- Can dynamically import from HTTP URLs (esm.sh)
- Executes tool functions and returns results
- Is already set up for existing ToolPlayground
## New Endpoint Required
### `POST /load-and-describe`
**Purpose**: Dynamically import a tool package and return its AI SDK tool definition (description, schema) without executing it.
**Request**:
```json
{
"packageName": "firecrawl-aisdk",
"exportName": "webSearchTool",
"version": "0.7.2",
"importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2"
}
```
**Response**:
```json
{
"success": true,
"tool": {
"exportName": "webSearchTool",
"description": "Search the web using Firecrawl",
"inputSchema": {
"type": "object",
"properties": {
"query": { "type": "string", "description": "Search query" }
},
"required": ["query"]
}
}
}
```
**Implementation** (pseudo-code for Railway service):
```javascript
// server.js (Railway service)
import express from 'express';
const app = express();
app.use(express.json());
// Cache for imported modules
const moduleCache = new Map();
app.post('/load-and-describe', async (req, res) => {
const { packageName, exportName, version, importUrl } = req.body;
const cacheKey = `${packageName}::${exportName}`;
try {
let toolModule;
// Check cache first
if (moduleCache.has(cacheKey)) {
console.log(`✅ Cache hit: ${cacheKey}`);
toolModule = moduleCache.get(cacheKey);
} else {
// Dynamic import from esm.sh
const url = importUrl || `https://esm.sh/${packageName}@${version}`;
console.log(`📦 Importing: ${url}`);
const module = await import(url);
toolModule = module[exportName];
if (!toolModule) {
return res.status(404).json({
success: false,
error: `Export "${exportName}" not found in module`
});
}
// Validate it's an AI SDK tool
if (!toolModule.description || !toolModule.execute) {
return res.status(400).json({
success: false,
error: `Invalid AI SDK tool structure`
});
}
// Cache it
moduleCache.set(cacheKey, toolModule);
}
// Extract tool definition (description + schema)
// AI SDK v6 tools have: description, inputSchema, execute
res.json({
success: true,
tool: {
exportName,
description: toolModule.description,
inputSchema: toolModule.inputSchema || toolModule.parameters?.shape || {},
}
});
} catch (error) {
console.error('Failed to load tool:', error);
res.status(500).json({
success: false,
error: error.message
});
}
});
// Start server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
console.log(`Railway tool loader running on port ${PORT}`);
});
```
**Railway Deployment**:
```bash
# Start command in Railway settings:
node --experimental-network-imports server.js
# Or in package.json:
{
"scripts": {
"start": "node --experimental-network-imports server.js"
}
}
```
## Modified Endpoint: `POST /execute-tool`
**Purpose**: Execute a dynamically loaded tool with parameters.
**Request**:
```json
{
"packageName": "firecrawl-aisdk",
"exportName": "webSearchTool",
"version": "0.7.2",
"importUrl": "https://esm.sh/firecrawl-aisdk@0.7.2",
"params": {
"query": "latest AI news"
}
}
```
**Response**:
```json
{
"success": true,
"output": {
"results": [...]
},
"executionTimeMs": 1234
}
```
**Implementation** (pseudo-code):
```javascript
app.post('/execute-tool', async (req, res) => {
const { packageName, exportName, version, importUrl, params } = req.body;
const cacheKey = `${packageName}::${exportName}`;
const startTime = Date.now();
try {
let toolModule;
// Check cache or import
if (moduleCache.has(cacheKey)) {
toolModule = moduleCache.get(cacheKey);
} else {
const url = importUrl || `https://esm.sh/${packageName}@${version}`;
const module = await import(url);
toolModule = module[exportName];
if (!toolModule || !toolModule.execute) {
return res.status(404).json({
success: false,
error: 'Tool not found or invalid'
});
}
moduleCache.set(cacheKey, toolModule);
}
// Execute the tool
const result = await toolModule.execute(params);
res.json({
success: true,
output: result,
executionTimeMs: Date.now() - startTime
});
} catch (error) {
res.status(500).json({
success: false,
error: error.message,
executionTimeMs: Date.now() - startTime
});
}
});
```
## Integration with Playground
### 1. Update `dynamic-tool-loader.ts`
Replace local dynamic imports with Railway service calls:
```typescript
// apps/playground/src/lib/dynamic-tool-loader.ts
const RAILWAY_SERVICE_URL = process.env.RAILWAY_SERVICE_URL || 'http://localhost:3001';
export async function loadToolDynamically(
packageName: string,
exportName: string,
version: string,
importUrl?: string
): Promise<any | null> {
const cacheKey = getCacheKey(packageName, exportName);
// Check local cache first
if (moduleCache.has(cacheKey)) {
console.log(`✅ Cache hit: ${cacheKey}`);
return moduleCache.get(cacheKey);
}
try {
console.log(`📦 Loading from Railway: ${packageName}/${exportName}`);
// Call Railway service to load and describe tool
const response = await fetch(`${RAILWAY_SERVICE_URL}/load-and-describe`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
packageName,
exportName,
version,
importUrl,
}),
});
if (!response.ok) {
console.error(`❌ Railway service error: ${response.status}`);
return null;
}
const data = await response.json();
if (!data.success) {
console.error(`❌ Failed to load tool: ${data.error}`);
return null;
}
// Create a tool wrapper that executes remotely
const tool = {
description: data.tool.description,
inputSchema: data.tool.inputSchema,
execute: async (params: any) => {
console.log(`🚀 Executing ${packageName}/${exportName} remotely`);
const execResponse = await fetch(`${RAILWAY_SERVICE_URL}/execute-tool`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
packageName,
exportName,
version,
importUrl,
params,
}),
});
const result = await execResponse.json();
if (!result.success) {
throw new Error(result.error || 'Tool execution failed');
}
return result.output;
},
};
// Cache the wrapper
moduleCache.set(cacheKey, tool);
console.log(`✅ Loaded and cached: ${cacheKey}`);
return tool;
} catch (error) {
console.error(`❌ Failed to load ${packageName}#${exportName}:`, error);
return null;
}
}
```
### 2. Environment Variables
Add to `.env.local`:
```bash
RAILWAY_SERVICE_URL=https://your-railway-service.up.railway.app
```
Or for local testing with Railway running locally:
```bash
RAILWAY_SERVICE_URL=http://localhost:3001
```
## Testing Locally
### Terminal 1: Run Railway service locally
```bash
cd railway-service
node --experimental-network-imports server.js
```
### Terminal 2: Run playground
```bash
cd tpmjs
pnpm dev --filter=@tpmjs/playground
```
### Test the flow:
```bash
# Test Railway service directly
curl -X POST http://localhost:3001/load-and-describe \
-H "Content-Type: application/json" \
-d '{
"packageName": "firecrawl-aisdk",
"exportName": "webSearchTool",
"version": "0.7.2"
}'
# Then test via playground UI
# Navigate to http://localhost:3000/playground
# Ask: "search the web for latest AI news"
```
## Deployment Checklist
- [ ] Create Railway service with Node.js
- [ ] Add `--experimental-network-imports` flag to start command
- [ ] Deploy `/load-and-describe` endpoint
- [ ] Deploy `/execute-tool` endpoint (or modify existing `/execute`)
- [ ] Set `RAILWAY_SERVICE_URL` in Vercel environment variables
- [ ] Test with real tools from TPMJS registry
- [ ] Monitor Railway logs for import errors
## Benefits
1. ✅ **Works around Next.js limitations** - Imports happen in plain Node
2. ✅ **Reuses existing Railway infrastructure** - No new service needed
3. ✅ **Caching on both sides** - Local cache + Railway cache
4. ✅ **Security** - Tools execute in Railway sandbox, not Next.js
5. ✅ **Scalability** - Railway handles the heavy lifting
## Next Steps
1. Implement Railway service endpoints
2. Update `dynamic-tool-loader.ts` to use Railway
3. Test locally
4. Deploy to Railway + Vercel
5. Celebrate dynamic tool loading! 🎉

57
README.md
View file

@ -1,10 +1,59 @@
# TPMJS Monorepo
# TPMJS
[![CI](https://github.com/YOUR_ORG/tpmjs/actions/workflows/ci.yml/badge.svg)](https://github.com/YOUR_ORG/tpmjs/actions/workflows/ci.yml)
[![CI](https://github.com/tpmjs/tpmjs/actions/workflows/ci.yml/badge.svg)](https://github.com/tpmjs/tpmjs/actions/workflows/ci.yml)
Tool Package Manager for AI Agents - A Turborepo monorepo with strict TypeScript, Next.js 16, and best practices.
**TPMJS is a registry that lets AI agents discover and use npm packages as tools at runtime.**
## Structure
Instead of manually importing and configuring tools, agents search by description and load what they need on-demand. Publish your npm package with the `tpmjs-tool` keyword—it appears on [tpmjs.com](https://tpmjs.com) within 15 minutes.
## Why TPMJS?
- **No config files** - Agents discover tools by describing what they need
- **Always up-to-date** - Tools load from npm at runtime, no manual updates
- **Works with any agent** - Compatible with Vercel AI SDK, LangChain, OpenAI, Claude, etc.
- **Publish once** - Add one keyword to package.json, publish to npm, done
## Quick Start
**For AI agent developers:**
```bash
npm install @tpmjs/sdk
```
```typescript
import { searchRegistry, executeRegistry } from '@tpmjs/sdk';
// Find tools by description
const tools = await searchRegistry({ query: 'parse PDF documents' });
// Execute a tool
const result = await executeRegistry({
toolId: 'pdf-parser/extractText',
input: { url: 'https://example.com/doc.pdf' }
});
```
**For tool publishers:**
```bash
npx @tpmjs/create-basic-tools
```
Or add manually to your package.json:
```json
{
"keywords": ["tpmjs-tool"],
"tpmjs": {
"category": "text-analysis",
"description": "What your tool does"
}
}
```
See [HOW_TO_PUBLISH_A_TOOL.md](./HOW_TO_PUBLISH_A_TOOL.md) for the full guide.
---
## Monorepo Structure
```
apps/

363
STREAMING_EMPTY_RESPONSE.md
View file

@ -1,363 +0,0 @@
# AI SDK v6 Streaming Empty Response Issue
## Problem
Using AI SDK v6 Beta with OpenAI and `streamText()`, the API route returns a 200 OK response, but the streamed response body is **completely empty** when tools are involved.
- **Normal chat** (without tool calls): Works fine, streams text back
- **Tool calls** (when user asks "say hello world"): Returns empty response body, no error messages
## Environment
- **AI SDK Version**: `ai@6.0.0-beta.124`
- **OpenAI Provider**: `@ai-sdk/openai@3.0.0-beta.74`
- **OpenAI Library**: `openai@^6.9.1`
- **Next.js Version**: `next@^16.0.4` (App Router)
- **Runtime**: Node.js (`runtime = 'nodejs'`)
- **Model**: `gpt-4o-mini`
## API Route Implementation
Located at: `apps/playground/src/app/api/chat/route.ts`
```typescript
import { createOpenAI } from '@ai-sdk/openai';
import { streamText, type CoreMessage, tool, jsonSchema } from 'ai';
import { type NextRequest } from 'next/server';
import { z } from 'zod';
import { env } from '~/env';
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const maxDuration = 60;
// Initialize OpenAI provider
const openai = createOpenAI({
apiKey: env.OPENAI_API_KEY,
});
// Request schema
const RequestSchema = z.object({
messages: z.array(
z.object({
role: z.enum(['user', 'assistant', 'system']),
content: z.string(),
})
),
});
// Simple inline test tool to verify streaming works
const testHelloTool = tool({
description: 'Returns a simple hello world greeting',
inputSchema: jsonSchema<{ includeTimestamp?: boolean }>({
type: 'object',
properties: {
includeTimestamp: {
type: 'boolean',
description: 'Whether to include a timestamp',
},
},
additionalProperties: false,
}),
async execute({ includeTimestamp = true }) {
const response: any = { message: 'Hello, World!' };
if (includeTimestamp) {
response.timestamp = new Date().toISOString();
}
return response;
},
});
/**
* POST /api/chat
* Chat with AI agent that can execute TPMJS tools
*/
export async function POST(request: NextRequest) {
try {
const body = await request.json();
const { messages } = RequestSchema.parse(body);
// Use simple inline tool for testing
const tools = {
testHello: testHelloTool,
};
// Create system prompt listing available tools
const toolsList = Object.keys(tools)
.map((name) => `- ${name}: ${tools[name]?.description}`)
.join('\n');
const systemMessage: CoreMessage = {
role: 'system',
content: `You are a helpful AI assistant that can use TPMJS tools to help users.
Available tools:
${toolsList}
Call tools as needed to answer user questions. When a user asks to say hello world or for a greeting, use the testHello tool.`,
};
// Stream the response
const result = streamText({
model: openai('gpt-4o-mini'),
messages: [systemMessage, ...messages],
tools,
});
// Return the stream as SSE
return result.toTextStreamResponse();
} catch (error) {
console.error('Chat API error:', error);
if (error instanceof z.ZodError) {
return new Response(
JSON.stringify({
success: false,
error: 'Invalid request format',
details: error.issues,
}),
{
status: 400,
headers: { 'Content-Type': 'application/json' },
}
);
}
return new Response(
JSON.stringify({
success: false,
error: error instanceof Error ? error.message : 'Unknown error',
}),
{
status: 500,
headers: { 'Content-Type': 'application/json' },
}
);
}
}
```
## Client-Side Hook
Located at: `apps/playground/src/hooks/useChat.ts`
```typescript
'use client';
import { useCallback, useState } from 'react';
export interface ChatMessage {
id: string;
role: 'user' | 'assistant' | 'system';
content: string;
timestamp: Date;
}
export function useChat() {
const [messages, setMessages] = useState<ChatMessage[]>([]);
const [isLoading, setIsLoading] = useState(false);
const [error, setError] = useState<string | null>(null);
const sendMessage = useCallback(async (content: string) => {
if (!content.trim()) return;
// Add user message immediately
const userMessage: ChatMessage = {
id: crypto.randomUUID(),
role: 'user',
content,
timestamp: new Date(),
};
setMessages((prev) => [...prev, userMessage]);
setIsLoading(true);
setError(null);
try {
// Create assistant message placeholder
const assistantMessageId = crypto.randomUUID();
const assistantMessage: ChatMessage = {
id: assistantMessageId,
role: 'assistant',
content: '',
timestamp: new Date(),
};
setMessages((prev) => [...prev, assistantMessage]);
// Send request to API
const response = await fetch('/api/chat', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
messages: [...messages, userMessage].map((m) => ({
role: m.role,
content: m.content,
})),
}),
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
// Read the streaming response
const reader = response.body?.getReader();
const decoder = new TextDecoder();
if (!reader) {
throw new Error('Response body is null');
}
let accumulatedContent = '';
while (true) {
const { done, value } = await reader.read();
if (done) {
break;
}
// Decode the chunk
const chunk = decoder.decode(value, { stream: true });
accumulatedContent += chunk;
// Update the assistant message with accumulated content
setMessages((prev) =>
prev.map((m) =>
m.id === assistantMessageId
? { ...m, content: accumulatedContent }
: m
)
);
}
} catch (err) {
console.error('Error sending message:', err);
setError(err instanceof Error ? err.message : 'Failed to send message');
} finally {
setIsLoading(false);
}
}, [messages]);
const clearChat = useCallback(() => {
setMessages([]);
setError(null);
}, []);
return {
messages,
isLoading,
error,
sendMessage,
clearChat,
};
}
```
## Observed Behavior
### Working Case (Normal Chat)
- User types: "hi"
- API response: 200 OK
- Response body: Streams text chunks successfully
- UI shows: "Hi! How can I help you today?"
### Broken Case (Tool Call)
- User types: "say hello world"
- API response: 200 OK ✅
- Response body: **EMPTY** ❌ (no chunks, no data, nothing)
- UI shows: Empty message bubble
- Console: No errors logged
## HTTP Response Details
```
Request Method: POST
Status Code: 200 OK
URL: http://localhost:3001/api/chat
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
```
The response headers look correct for a streaming response, but the body is completely empty.
## What We've Tried
1. ✅ Fixed OpenAI schema validation error (was `type: "None"`, now uses proper JSON Schema)
2. ✅ Using `tool()` and `jsonSchema()` from AI SDK
3. ✅ Simplified to a single inline test tool
4. ✅ Tool executes without errors (no schema validation issues)
5. ✅ Normal chat works fine (proves streaming infrastructure is correct)
## Questions
1. **Is `toTextStreamResponse()` the correct method for streaming with tools in AI SDK v6?**
- Should we use a different method like `toDataStreamResponse()` for tool calls?
2. **Are we constructing the messages array correctly?**
- We're sending `{ role: 'user' | 'assistant' | 'system', content: string }[]`
- Do we need to include tool call messages or tool result messages?
3. **Does AI SDK v6 require a specific message format for tool calls?**
- Should we be including `toolInvocations` or `tool_calls` in the message history?
- Are we missing required fields in the `CoreMessage` type?
4. **Is the client-side streaming reader correct?**
- We're reading chunks with `response.body.getReader()`
- Should we be parsing SSE events differently for tool calls?
5. **Does `streamText()` with tools require `maxSteps` parameter?**
- Do we need to set `maxSteps: 5` to allow multi-step reasoning?
6. **Are we handling the conversation history correctly?**
- We're sending all previous messages on each request
- Should we be including assistant messages with tool call results?
## AI SDK v6 Documentation References
We're following these patterns from the official docs:
- [streamText() API](https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text)
- [tool() API](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool)
- [Tool Calling Guide](https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling)
But we might be missing something specific about:
- How to handle streaming when tools are executed
- What response format tool calls produce
- How to parse the stream when tools are involved
## Suspected Issue
**The message format might be wrong.** We're sending:
```typescript
const systemMessage: CoreMessage = {
role: 'system',
content: `You are a helpful AI assistant...`,
};
const result = streamText({
model: openai('gpt-4o-mini'),
messages: [systemMessage, ...messages],
tools,
});
```
But `CoreMessage` might need additional fields when tools are involved, or we might need to handle tool call results differently in the conversation history.
## What We Need
1. Correct message format for `streamText()` with tools
2. How to properly stream responses that include tool calls
3. Whether we need different client-side parsing for tool call streams
4. Example of a working Next.js API route using AI SDK v6 with `streamText()` and tools
## Repo Context
- Monorepo using Turborepo + pnpm workspaces
- Next.js 16 App Router with Turbopack
- TypeScript strict mode
- All UI components from internal `@tpmjs/ui` package
- Tools are imported from workspace package `@tpmjs/hello`

400
USECHAT_INPUT_UNDEFINED.md
View file

@ -1,400 +0,0 @@
# useChat Hook Returns Undefined Input Property
## Problem
Using `@ai-sdk/react`'s `useChat` hook, the `input` property is returning `undefined`, causing the application to crash when trying to call `.trim()` on it.
## Error
```
TypeError: Cannot read properties of undefined (reading 'trim')
at ChatInput (src/components/chat/ChatInput.tsx:41:48)
```
**Code that fails:**
```typescript
<Button type="submit" disabled={!input.trim() || isLoading} loading={isLoading} size="lg">
```
## Environment
- **AI SDK Version**: `ai@6.0.0-beta.124`
- **AI SDK React**: `@ai-sdk/react` (latest version installed via pnpm)
- **OpenAI Provider**: `@ai-sdk/openai@3.0.0-beta.74`
- **Next.js**: `16.0.4` (App Router with Turbopack)
- **React**: `19.0.0`
- **Zod**: `4.0.0` (required, not downgrading)
- **TypeScript**: `5.9.3`
## Current Implementation
### Custom useChat Hook Wrapper
Located at: `apps/playground/src/hooks/useChat.ts`
```typescript
'use client';
import { useChat as useAISDKChat } from '@ai-sdk/react';
/**
* Custom chat hook that wraps the official @ai-sdk/react useChat
* Handles SSE streaming with tool calls and UI message protocol
*/
export function useChat() {
const chat = useAISDKChat({
api: '/api/chat',
});
return {
messages: chat.messages,
input: chat.input,
isLoading: chat.isLoading,
error: chat.error,
handleInputChange: chat.handleInputChange,
handleSubmit: chat.handleSubmit,
setInput: chat.setInput,
reload: chat.reload,
stop: chat.stop,
};
}
```
### API Route
Located at: `apps/playground/src/app/api/chat/route.ts`
```typescript
import { createOpenAI } from '@ai-sdk/openai';
import { streamText, convertToModelMessages, stepCountIs, type UIMessage } from 'ai';
import { type NextRequest } from 'next/server';
import { env } from '~/env';
import { loadAllTools } from '~/lib/tool-loader';
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const maxDuration = 60;
// Initialize OpenAI provider
const openai = createOpenAI({
apiKey: env.OPENAI_API_KEY,
});
/**
* POST /api/chat
* Chat with AI agent that can execute TPMJS tools
*/
export async function POST(request: NextRequest) {
try {
const { messages }: { messages: UIMessage[] } = await request.json();
// Load all available TPMJS tools
const tools = await loadAllTools();
// Create system prompt listing available tools
const toolsList = Object.keys(tools)
.map((name) => `- ${name}: ${tools[name]?.description}`)
.join('\n');
const system = `You are a helpful AI assistant that can use TPMJS tools to help users.
Available tools:
${toolsList}
When you use a tool, you MUST always follow up with a natural language answer to the user summarizing the result.`;
// Stream the response with multi-step tool usage enabled
const result = streamText({
model: openai('gpt-4o-mini'),
system,
messages: convertToModelMessages(messages),
tools,
stopWhen: stepCountIs(5), // Allow model to call tools AND generate text response
});
// Return UI message stream with tool calls and text
return result.toUIMessageStreamResponse();
} catch (error) {
console.error('Chat API error:', error);
return new Response(
JSON.stringify({
success: false,
error: error instanceof Error ? error.message : 'Unknown error',
}),
{
status: 500,
headers: { 'Content-Type': 'application/json' },
}
);
}
}
```
### Component Using the Hook
Located at: `apps/playground/src/components/chat/ChatInterface.tsx`
```typescript
'use client';
import { useChat } from '~/hooks/useChat';
import { ChatInput } from './ChatInput';
import { ChatMessages } from './ChatMessages';
export function ChatInterface(): React.ReactElement {
const { messages, input, isLoading, handleInputChange, handleSubmit, setInput } = useChat();
return (
<div className="flex flex-1 flex-col">
<ChatMessages messages={messages} />
<ChatInput
input={input}
isLoading={isLoading}
onInputChange={handleInputChange}
onSubmit={handleSubmit}
setInput={setInput}
/>
</div>
);
}
```
### ChatInput Component
Located at: `apps/playground/src/components/chat/ChatInput.tsx`
```typescript
'use client';
import { Button } from '@tpmjs/ui/Button/Button';
import { Textarea } from '@tpmjs/ui/Textarea/Textarea';
import type { FormEvent } from 'react';
interface ChatInputProps {
input: string;
isLoading: boolean;
onInputChange: (e: React.ChangeEvent<HTMLTextAreaElement>) => void;
onSubmit: (e: FormEvent<HTMLFormElement>) => void;
setInput: (value: string) => void;
}
export function ChatInput({ input, isLoading, onInputChange, onSubmit }: ChatInputProps): React.ReactElement {
const handleKeyDown = (e: React.KeyboardEvent<HTMLTextAreaElement>) => {
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault();
if (input.trim() && !isLoading) {
// Trigger form submission
const form = e.currentTarget.form;
if (form) {
form.requestSubmit();
}
}
}
};
return (
<form onSubmit={onSubmit} className="border-t border-border bg-background p-4">
<div className="mx-auto flex max-w-4xl gap-2">
<Textarea
value={input}
onChange={onInputChange}
onKeyDown={handleKeyDown}
placeholder="Ask me to tell you a fish joke..."
className="min-h-[60px] flex-1 resize-none"
disabled={isLoading}
rows={3}
/>
<Button type="submit" disabled={!input.trim() || isLoading} loading={isLoading} size="lg">
Send
</Button>
</div>
</form>
);
}
```
## Tool Definition (Working)
The tools are defined using `tool()` and `jsonSchema()` from AI SDK to avoid Zod 4 compatibility issues:
Located at: `packages/tools/hello/src/index.ts`
```typescript
import { jsonSchema, tool } from 'ai';
type HelloWorldInput = {
includeTimestamp?: boolean;
};
export const helloWorldTool = tool({
description: 'Returns a simple "Hello, World!" greeting message',
inputSchema: jsonSchema<HelloWorldInput>({
type: 'object',
properties: {
includeTimestamp: {
type: 'boolean',
description: 'Whether to include a timestamp in the response',
},
},
additionalProperties: false,
}),
async execute({ includeTimestamp = true }) {
const response: any = {
message: 'Hello, World!',
};
if (includeTimestamp) {
response.timestamp = new Date().toISOString();
}
return response;
},
});
type HelloNameInput = {
name: string;
};
export const helloNameTool = tool({
description: 'Returns a personalized greeting with the provided name',
inputSchema: jsonSchema<HelloNameInput>({
type: 'object',
properties: {
name: {
type: 'string',
description: 'The name of the person to greet',
},
},
required: ['name'],
additionalProperties: false,
}),
async execute({ name }) {
return {
message: `Hello, ${name}!`,
timestamp: new Date().toISOString(),
};
},
});
```
## What's Working
1. ✅ API route receives requests correctly
2. ✅ Tools are loaded and registered successfully
3. ✅ `streamText()` with `stopWhen: stepCountIs(5)` configured
4. ✅ `toUIMessageStreamResponse()` returns proper SSE stream
5. ✅ curl test shows tools are called correctly with proper JSON Schema
6. ✅ Stream format includes `tool-input-start`, `tool-output-available`, `text-delta` events
## What's NOT Working
1. ❌ `input` property from `useChat` is `undefined`
2. ❌ Application crashes when trying to access `input.trim()`
3. ❌ Can't type in the chat input field
## curl Test (Successful)
```bash
curl -N http://localhost:3001/api/chat \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":"say hello thomas"}]}'
```
**Response:**
```
data: {"type":"start"}
data: {"type":"start-step"}
data: {"type":"tool-input-start","toolCallId":"call_...","toolName":"helloName"}
data: {"type":"tool-input-delta","toolCallId":"call_...","inputTextDelta":"{\""}
data: {"type":"tool-input-delta","toolCallId":"call_...","inputTextDelta":"name"}
data: {"type":"tool-input-delta","toolCallId":"call_...","inputTextDelta":"\":\""}
data: {"type":"tool-input-delta","toolCallId":"call_...","inputTextDelta":"Thomas"}
data: {"type":"tool-input-delta","toolCallId":"call_...","inputTextDelta":"\"}"}
data: {"type":"tool-input-available","toolCallId":"call_...","toolName":"helloName","input":{"name":"Thomas"},"providerMetadata":{...}}
data: {"type":"tool-output-available","toolCallId":"call_...","output":{"message":"Hello, Thomas!","timestamp":"2025-12-03T16:07:31.366Z"}}
data: {"type":"finish-step"}
data: {"type":"start-step"}
data: {"type":"text-start","id":"msg_...","providerMetadata":{...}}
data: {"type":"text-delta","id":"msg_...","delta":"Hello"}
data: {"type":"text-delta","id":"msg_...","delta":","}
data: {"type":"text-delta","id":"msg_...","delta":" Thomas"}
data: {"type":"text-delta","id":"msg_...","delta":"!"}
data: {"type":"text-end","id":"msg_...","providerMetadata":{...}}
data: {"type":"finish-step"}
data: {"type":"finish","finishReason":"stop"}
data: [DONE]
```
The API works perfectly - tools are called, results are returned, text is generated. The issue is purely on the React client side.
## Questions
1. **Is `@ai-sdk/react`'s `useChat` compatible with AI SDK v6 Beta (6.0.0-beta.124)?**
- Should we be using a different version of `@ai-sdk/react`?
- Are there known compatibility issues with AI SDK v6 Beta?
2. **Why is `input` undefined?**
- Does `useChat` require specific initialization options?
- Do we need to provide `initialMessages` or `initialInput`?
- Is there a required prop we're missing?
3. **Is the API route format correct for `@ai-sdk/react`'s `useChat`?**
- Should the API accept a different request format?
- Is `UIMessage[]` the correct type for messages?
- Should we use `toDataStreamResponse()` instead of `toUIMessageStreamResponse()`?
4. **Do we need to handle client-side state differently?**
- Should we initialize `input` with a default value?
- Is there a provider or context missing?
- Do we need to wrap the component tree with any providers?
5. **Is there a version mismatch between packages?**
- `ai@6.0.0-beta.124`
- `@ai-sdk/openai@3.0.0-beta.74`
- `@ai-sdk/react@?` (unknown version)
6. **Does Zod 4 affect the client-side hook?**
- We fixed the server-side tool schemas using `jsonSchema()`
- Could there be client-side Zod 4 issues affecting `useChat`?
## Expected Behavior
The `useChat` hook should return:
- `input: string` - Current input value (should be empty string initially)
- `handleInputChange: (e) => void` - Update input value
- `handleSubmit: (e) => void` - Submit form and send message
- `messages: Message[]` - Array of messages
- `isLoading: boolean` - Loading state
## Actual Behavior
- `input: undefined` ❌
- Everything else appears to be defined
- Crash on first render when trying to access `input.trim()`
## Monorepo Context
- Turborepo monorepo with pnpm workspaces
- Next.js 16 App Router with Turbopack
- TypeScript strict mode
- `@tpmjs/ui` package for UI components
- `@tpmjs/hello` package for tools
- Using workspace protocol (`workspace:*`) for internal dependencies
## Related Documentation
- [AI SDK Core: streamText](https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text)
- [AI SDK React: useChat](https://ai-sdk.dev/docs/reference/ai-sdk-ui/use-chat)
- [AI SDK UI: Stream Protocol](https://ai-sdk.dev/docs/ai-sdk-ui/stream-protocol)
## What We Need
1. Correct version compatibility information for AI SDK v6 Beta + @ai-sdk/react
2. Why `input` is undefined and how to fix it
3. Whether our API route format is correct for the React hook
4. Any missing initialization or configuration for `useChat`
5. Whether there are alternative approaches (custom SSE parsing, different hook, etc.)
We must keep Zod 4 and cannot downgrade. The server-side tools are working correctly with `jsonSchema()` workaround.

324
VERCEL_FIX_CHECKLIST.md
View file

@ -1,324 +0,0 @@
# 🔥 Vercel API Routes Fix - Action Checklist
## Problem Summary
**API routes are not deploying because Vercel is not detecting your project as Next.js.**
When Vercel doesn't detect Next.js, it treats your deployment as a static site and **drops all App Router API routes** from the build output. Pages work because they're static files, but API routes require serverless function generation which only happens when Next.js is properly detected.
---
## ✅ Fix Checklist (Complete in Order)
### 1️⃣ Fix Root Directory
**Where:** Vercel Dashboard → Project `tpmjs-web` → Settings → General → Root Directory
**Current (likely):** Empty, `.`, or wrong path
**Required:** `apps/web` (exactly this, no leading/trailing slashes)
**Validation:**
```
✓ Must be exactly: apps/web
✗ NOT: /apps/web/
✗ NOT: ./apps/web
✗ NOT: tpmjs/apps/web
```
---
### 2️⃣ Set Framework Preset
**Where:** Vercel Dashboard → Project `tpmjs-web` → Settings → General → Framework Preset
**Current (likely):** "Other"
**Required:** "Next.js"
**Why this matters:**
- Framework = "Other" → Uses `@vercel/static-builder` → No API routes
- Framework = "Next.js" → Uses `@vercel/next` → API routes deployed
---
### 3️⃣ Remove Domain from Old Project
**Where:** Vercel Dashboard → Project `v0-tool-registry-page` → Settings → Domains
**Action:** Remove these domains:
- `tpmjs.com`
- `www.tpmjs.com`
**Then:** Verify both domains are ONLY assigned to the `tpmjs` project
**Why this matters:**
The "Redirecting..." message is coming from the old project. Having two projects with the same domain causes shadow routing and API requests hitting the wrong deployment.
---
### 4️⃣ Clear Custom Build Commands
**Where:** Vercel Dashboard → Project `tpmjs-web` → Settings → Build & Development Settings
**Set ALL to default/empty:**
```
Build Command: (empty - let Vercel auto-detect)
Install Command: (empty - let Vercel auto-detect)
Output Directory: .next (default)
```
**Why this matters:**
Custom build commands bypass Vercel's Next.js detection. Vercel should automatically:
- Detect monorepo structure
- Run `pnpm install`
- Run `pnpm build` in the correct workspace
- Use `@vercel/next` builder
**If you must use custom commands, use:**
```
Build Command: pnpm turbo run build --filter=@tpmjs/web
Install Command: pnpm install
```
But try empty first.
---
## 🧪 Verification Steps
### Before Deploying
Run this locally to confirm Next.js detection:
```bash
cd apps/web
vercel build
```
**Expected output should include:**
```
● route (app) /api/health
● route (app) /api/tools
● route (app) /api/sync/changes
λ /api/health
λ /api/tools
λ /api/sync/changes
```
**If you DON'T see this, Vercel won't deploy API routes.**
### After Deploying
1. **Check Build Output:**
```bash
vercel inspect <deployment-url>
```
Should show:
```
Builds
├── λ api/health (XXX KB) [region]
├── λ api/tools (XXX KB) [region]
├── λ api/sync/changes (XXX KB) [region]
├── λ tool/[slug] (XXX KB) [region]
...
```
2. **Test API Routes:**
```bash
# Should return JSON (not timeout, not "Redirecting...")
curl https://tpmjs.com/api/health
# Should return tool data
curl https://tpmjs.com/api/tools
```
---
## 📋 Expected Results
### ✅ Success Indicators
- [ ] `vercel inspect` shows API routes as `λ` functions
- [ ] `curl https://tpmjs.com/api/health` returns JSON
- [ ] `curl https://tpmjs.com/api/tools` returns tool data
- [ ] No "Redirecting..." messages
- [ ] No timeouts on direct Vercel URLs
- [ ] Build logs show "route (app) /api/*"
### ❌ Failure Indicators (Need to revisit steps)
- [ ] Only pages listed in `vercel inspect`, no API routes
- [ ] API endpoints return "Redirecting..."
- [ ] API endpoints timeout (exit code 28)
- [ ] Build logs don't mention API routes
- [ ] Framework Preset still shows "Other"
---
## 🚨 Common Mistakes
### Mistake 1: Wrong Root Directory Format
```
✗ /apps/web/ (leading/trailing slashes)
✗ ./apps/web (relative path notation)
✗ apps/web/ (trailing slash)
✓ apps/web (correct)
```
### Mistake 2: Leaving Custom Build Commands
If you have:
```json
{
"buildCommand": "cd ../.. && turbo build --filter=@tpmjs/web"
}
```
This MIGHT work, but can break Next.js detection. Start with empty and only add if needed.
### Mistake 3: Not Removing Domain from Old Project
If `v0-tool-registry-page` still has `tpmjs.com`, your requests will route to the wrong project randomly based on:
- DNS propagation
- Edge cache
- Vercel's routing priority
### Mistake 4: Not Verifying Framework Preset
"Other" is Vercel's default when it can't detect a framework. This is the #1 cause of missing API routes in monorepos.
---
## 🔧 Troubleshooting
### If API routes STILL don't deploy after all 4 steps:
1. **Check Build Logs:**
- Does it say "Detected Next.js"?
- Does it list "route (app) /api/*"?
- Does it show `@vercel/next` builder?
2. **Check package.json location:**
```
✓ Should exist: apps/web/package.json
✗ Should NOT be at root ONLY
```
3. **Check next.config.ts location:**
```
✓ Should exist: apps/web/next.config.ts
```
4. **Verify pnpm workspace:**
```bash
# Should show @tpmjs/web
pnpm list --depth 0 --filter @tpmjs/web
```
5. **Test local build with Vercel CLI:**
```bash
cd apps/web
vercel build --debug
```
Look for "Framework: nextjs" in output.
---
## 📞 When to Contact Vercel Support
If after completing all 4 steps:
- Build logs show "Detected Next.js"
- Build logs show "route (app) /api/*"
- BUT `vercel inspect` still doesn't list API functions
Then you have a Vercel platform bug. Contact support with:
- This checklist
- Build logs
- `vercel inspect` output
- Link to `API_ROUTES_TIMEOUT_INVESTIGATION.md`
---
## 🎯 Quick Win Test
**Don't want to change production settings yet?**
1. Create a NEW Vercel project
2. Import the SAME repo
3. Set Root Directory to `apps/web`
4. Set Framework Preset to "Next.js"
5. Deploy
If API routes work in the new project → confirms the fix
If API routes still fail → deeper issue (contact support)
---
## ✨ Post-Fix Cleanup
Once API routes are working:
### Optional: Re-add www redirect
Now that API routes work, you can safely add the www redirect back:
**Option A - Vercel Project Settings:**
Vercel Dashboard → Domains → tpmjs.com → Redirect www to apex
**Option B - Next.js config:**
```typescript
// apps/web/next.config.ts
async redirects() {
return [
{
source: '/:path((?!api).*)*', // Exclude /api/*
has: [{ type: 'host', value: 'www.tpmjs.com' }],
destination: 'https://tpmjs.com/:path*',
permanent: true,
},
];
}
```
**Option C - vercel.json (not recommended):**
Only use if you understand the implications.
### Optional: Remove maxDuration exports
The `export const maxDuration = 60;` in your route files isn't needed unless you actually need longer timeouts. Default is 10s (Hobby) or 15s (Pro).
---
## 📊 Summary
| Issue | Root Cause | Fix |
|-------|------------|-----|
| API routes timeout | Vercel doesn't detect Next.js | Set Framework Preset to "Next.js" |
| No λ functions in build | Wrong Root Directory | Set to `apps/web` exactly |
| "Redirecting..." on API calls | Domain on two projects | Remove from old project |
| Build doesn't find API routes | Custom build commands break detection | Clear custom commands |
**Time to fix:** 5 minutes (just changing dashboard settings)
**Deployments needed:** 1 (changes take effect on next deploy)
**Code changes needed:** 0 (this is pure configuration)
---
## 🎉 When It Works
You'll know it's fixed when:
```bash
$ curl https://tpmjs.com/api/health
{"status":"ok","timestamp":"2025-11-28T...","env":{"hasDatabase":true,"nodeEnv":"production"}}
$ curl https://tpmjs.com/api/tools
{"data":[...],"pagination":{...}}
```
And `vercel inspect <url>` shows:
```
Builds
├── λ api/health
├── λ api/tools
├── λ api/stats
... (ALL your API routes)
```
**That's it. No code changes. Just fix the Vercel project configuration.**

404
ZOD_SCHEMA_SERIALIZATION_PROBLEM.md
View file

@ -1,404 +0,0 @@
# Zod Schema Serialization Problem - Dynamic Tool Loading System
## Architecture Overview
We have a dynamic tool loading system with the following architecture:
```
Next.js Playground (Vercel) → Railway Deno Service → esm.sh CDN → npm packages
↓
AI SDK (OpenAI)
```
### Components:
1. **Next.js Playground** (`apps/playground`) - Runs on Vercel, hosts the AI chat interface
2. **Railway Deno Service** (`apps/railway-executor/server.ts`) - Runs on Railway, uses Deno's native HTTP import support
3. **Dynamic Tool Loader** (`apps/playground/src/lib/dynamic-tool-loader.ts`) - Client-side code that calls Railway
### Why We Need This Architecture:
- Next.js/Turbopack intercepts all `import()` calls and cannot import from HTTP URLs (like `https://esm.sh/package@version`)
- Deno natively supports HTTP imports via `import('https://...')`
- Tools are published to npm and loaded dynamically at runtime via esm.sh CDN
- We cannot bundle tools at build time - they must be discovered and loaded dynamically
## The Problem
AI SDK tools require a specific format with `description` and `inputSchema`:
```typescript
// AI SDK v6 Tool Format
const tool = {
description: "Search the web for current information...",
inputSchema: z.object({
query: z.string().describe("Search query"),
numResults: z.number().optional(),
}),
execute: async (params) => {
// ... execution logic
}
}
```
**The Challenge:** We need to send tool definitions from Railway (Deno) to the Playground (Next.js) over HTTP, but Zod schemas cannot be JSON serialized.
### Error 1: `def.shape is not a function`
When we tried to send the Zod schema directly:
```typescript
// Railway server.ts - DOESN'T WORK
return Response.json({
success: true,
tool: {
exportName: "searchTool",
description: "...",
inputSchema: toolModule.inputSchema, // ❌ Zod schema object
}
});
```
The Zod schema object loses all its methods during JSON serialization, causing:
```
TypeError: def.shape is not a function
```
### Error 2: OpenAI Invalid Schema Error
When we removed `inputSchema` entirely:
```typescript
// dynamic-tool-loader.ts - DOESN'T WORK
const tool = {
description: data.tool.description,
// No inputSchema
execute: async (params) => { ... }
};
```
OpenAI API rejects the tool:
```
Error [AI_APICallError]: Invalid schema for function 'firecrawl-aisdk-searchTool':
schema must be a JSON Schema of 'type: "object"', got 'type: "None"'.
```
## Requirements
1. ✅ **Must work with AI SDK v6** - Tools must have `description` + `inputSchema` format
2. ✅ **Must serialize over HTTP** - Railway → Playground communication is via fetch/HTTP
3. ✅ **Must support any Zod schema** - Tools use various Zod types (objects, arrays, unions, etc.)
4. ✅ **Must preserve validation** - The schema needs to work for OpenAI's function calling
5. ✅ **Must be efficient** - Schemas should be cached, not re-serialized on every request
## Current Code
### Railway Server (`apps/railway-executor/server.ts`)
```typescript
async function loadAndDescribe(req: Request): Promise<Response> {
const { packageName, exportName, version, importUrl } = await req.json();
const cacheKey = `${packageName}::${exportName}`;
let toolModule;
if (moduleCache.has(cacheKey)) {
toolModule = moduleCache.get(cacheKey);
} else {
const url = importUrl || `https://esm.sh/${packageName}@${version}`;
const module = await import(url); // Deno supports HTTP imports!
toolModule = module[exportName];
if (!toolModule.description || !toolModule.execute) {
return Response.json({
success: false,
error: 'Invalid AI SDK tool structure'
}, { status: 400 });
}
moduleCache.set(cacheKey, toolModule);
}
// ❌ PROBLEM: How to serialize inputSchema?
return Response.json({
success: true,
tool: {
exportName,
description: toolModule.description,
// Need to send inputSchema here somehow
hasInputSchema: !!toolModule.inputSchema,
},
});
}
async function executeTool(req: Request): Promise<Response> {
const { packageName, exportName, version, importUrl, params } = await req.json();
const cacheKey = `${packageName}::${exportName}`;
let toolModule;
if (moduleCache.has(cacheKey)) {
toolModule = moduleCache.get(cacheKey);
} else {
const url = importUrl || `https://esm.sh/${packageName}@${version}`;
const module = await import(url);
toolModule = module[exportName];
moduleCache.set(cacheKey, toolModule);
}
const result = await toolModule.execute(params || {});
return Response.json({
success: true,
output: result,
});
}
```
### Dynamic Tool Loader (`apps/playground/src/lib/dynamic-tool-loader.ts`)
```typescript
export async function loadToolDynamically(
packageName: string,
exportName: string,
version: string,
importUrl?: string
): Promise<any | null> {
const cacheKey = getCacheKey(packageName, exportName);
if (moduleCache.has(cacheKey)) {
return moduleCache.get(cacheKey);
}
// Call Railway to load and describe tool
const response = await fetch(`${RAILWAY_SERVICE_URL}/load-and-describe`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
packageName,
exportName,
version,
importUrl: importUrl || `https://esm.sh/${packageName}@${version}`,
}),
});
const data = await response.json();
// ❌ PROBLEM: How to reconstruct inputSchema?
const tool = {
description: data.tool.description,
// Need inputSchema here for OpenAI API
execute: async (params: any) => {
const execResponse = await fetch(`${RAILWAY_SERVICE_URL}/execute-tool`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ packageName, exportName, version, importUrl, params }),
});
const result = await execResponse.json();
return result.output;
},
};
moduleCache.set(cacheKey, tool);
return tool;
}
```
## Example Tool Schema
Here's an example of what we need to serialize:
```typescript
import { tool, jsonSchema } from 'ai';
import { z } from 'zod';
// AI SDK v6 format
export const searchTool = tool({
description: 'Search the web for current information...',
inputSchema: jsonSchema<{
query: string;
numResults?: number;
category?: string;
}>({
type: 'object',
properties: {
query: {
type: 'string',
description: 'Search query'
},
numResults: {
type: 'number',
description: 'Number of results',
minimum: 1,
maximum: 20,
default: 10
},
category: {
type: 'string',
enum: ['search', 'web-scraping', 'data-extraction'],
description: 'Filter by category'
}
},
required: ['query']
}),
execute: async ({ query, numResults = 10, category }) => {
// ... implementation
}
});
```
The `inputSchema` is created via `jsonSchema<Type>()` which wraps a JSON Schema object.
## Potential Approaches to Consider
### Option 1: Serialize Zod Schema to JSON Schema
Use `zod-to-json-schema` library to convert Zod schemas to JSON Schema format:
```typescript
import { zodToJsonSchema } from 'zod-to-json-schema';
// In Railway server
const jsonSchema = zodToJsonSchema(toolModule.inputSchema);
return Response.json({ inputSchema: jsonSchema });
```
**Pros:**
- Standard approach
- Widely used library
- Preserves validation rules
**Cons:**
- Adds dependency to Railway service
- May not work with `jsonSchema()` wrapper from AI SDK
- Need to ensure it works with AI SDK v6 format
### Option 2: Extract JSON Schema from AI SDK's jsonSchema()
The AI SDK's `jsonSchema()` function wraps a plain JSON Schema object. Maybe we can extract it:
```typescript
// Investigate the structure of toolModule.inputSchema
console.log(JSON.stringify(toolModule.inputSchema, null, 2));
// Potentially:
const plainSchema = toolModule.inputSchema._def?.schema || toolModule.inputSchema;
```
**Pros:**
- No additional dependencies
- Uses the schema exactly as AI SDK expects it
**Cons:**
- Relies on internal structure (fragile)
- May break with AI SDK updates
### Option 3: Store Schema Separately and Reconstruct
Store the raw JSON Schema definition separately in the tool package:
```typescript
// In tool package
export const searchToolSchema = {
type: 'object',
properties: { ... }
};
export const searchTool = tool({
description: '...',
inputSchema: jsonSchema(searchToolSchema),
execute: async (params) => { ... }
});
```
Then send `searchToolSchema` over HTTP and reconstruct.
**Pros:**
- Clean separation
- Easy to serialize
**Cons:**
- Requires tool authors to export schema separately
- Duplication of schema definition
- Breaks existing tools
### Option 4: Import Tool Directly in Playground (If Possible)
Try to make HTTP imports work in Next.js somehow:
- Webpack configuration hacks?
- Dynamic imports with custom loader?
- Build-time bundling of tools?
**Pros:**
- No serialization needed
- Direct access to tool objects
**Cons:**
- Next.js/Turbopack limitations are fundamental
- Defeats the purpose of dynamic loading
- May not be technically possible
### Option 5: Two-Way Communication - Send Schema Back
Instead of Railway → Playground, have Playground → Railway for schema:
1. Playground asks Railway: "Does this tool exist?"
2. Railway responds: "Yes, here's the description"
3. Playground asks Railway: "Execute this tool with these params"
4. Railway validates params against schema and executes
The schema stays on Railway side, never serialized.
**Pros:**
- Schema never leaves Railway
- No serialization issues
**Cons:**
- OpenAI API requires schema upfront for function calling
- Can't defer schema to execution time
- Doesn't solve the core problem
## Questions for Consideration
1. **Can we use `zod-to-json-schema` with AI SDK v6's `jsonSchema()` wrapper?**
- How does `jsonSchema()` work internally?
- Does it already store a plain JSON Schema somewhere?
2. **Does the AI SDK provide any serialization utilities?**
- Is there a built-in way to serialize tool definitions?
- Does Vercel have examples of this pattern?
3. **Can we modify the tool format to make serialization easier?**
- Would it break compatibility with existing tools?
- Is there a standardized way tools should export schemas?
4. **Is there a way to inspect the AI SDK's jsonSchema() structure?**
- What properties does it have?
- Can we extract the underlying JSON Schema object safely?
5. **Should we contribute back to the ecosystem?**
- Is this a common problem?
- Should there be a standard for serializable AI SDK tools?
## Current Status
- ✅ Railway service successfully imports tools from esm.sh via HTTP
- ✅ Railway can execute tools and return results
- ✅ Module caching works on Railway side
- ✅ Tool wrapper caching works on Playground side
- ❌ Cannot serialize Zod schemas over HTTP (this blocker)
- ❌ OpenAI API rejects tools without proper inputSchema
## Files to Reference
- `apps/railway-executor/server.ts` - Railway Deno service
- `apps/playground/src/lib/dynamic-tool-loader.ts` - Tool loader client
- `packages/tools/search-registry/src/index.ts` - Example tool using AI SDK v6
## Success Criteria
We need a solution that:
1. Sends complete tool definitions (description + inputSchema + execute capability) from Railway to Playground
2. Works with OpenAI's function calling API (requires valid JSON Schema)
3. Supports all Zod schema types used by AI SDK tools
4. Is maintainable and doesn't break with AI SDK updates
5. Doesn't require changes to existing published tool packages (if possible)

31867
ai-sdk-v6.md
View file

File diff suppressed because it is too large Load diff

2
apps/web/public/.well-known/security.txt
View file

@ -1,4 +1,4 @@
Contact: mailto:thomasalwyndavis@gmail.com
Contact: mailto:hello@tpmjs.com
Preferred-Languages: en
Canonical: https://tpmjs.com/.well-known/security.txt
Policy: https://tpmjs.com/.well-known/security.txt

14
apps/web/src/app/faq/page.tsx
View file

@ -302,11 +302,8 @@ export default function FAQPage(): React.ReactElement {
<li>
<strong className="text-foreground">For security issues:</strong> Email us
directly at{' '}
<a
href="mailto:thomasalwyndavis@gmail.com"
className="text-primary hover:underline"
>
thomasalwyndavis@gmail.com
<a href="mailto:hello@tpmjs.com" className="text-primary hover:underline">
hello@tpmjs.com
</a>
</li>
<li>
@ -370,11 +367,8 @@ export default function FAQPage(): React.ReactElement {
</li>
<li>
<strong className="text-foreground">Email:</strong> Contact us at{' '}
<a
href="mailto:thomasalwyndavis@gmail.com"
className="text-primary hover:underline"
>
thomasalwyndavis@gmail.com
<a href="mailto:hello@tpmjs.com" className="text-primary hover:underline">
hello@tpmjs.com
</a>{' '}
for private inquiries
</li>

10
apps/web/src/app/layout.tsx
View file

@ -26,7 +26,7 @@ export const metadata: Metadata = {
template: '%s | TPMJS',
},
description:
'The registry for AI tools. Discover, share, and integrate tools that give your agents superpowers.',
'Discover and use npm packages as AI agent tools. No config files, automatic discovery, works with any framework.',
icons: {
icon: [
{ url: '/favicon.svg', type: 'image/svg+xml' },
@ -56,7 +56,7 @@ export const metadata: Metadata = {
siteName: 'TPMJS',
title: 'TPMJS - Tool Package Manager for AI Agents',
description:
'The registry for AI tools. Discover, share, and integrate tools that give your agents superpowers.',
'Discover and use npm packages as AI agent tools. No config files, automatic discovery, works with any framework.',
images: [
{
url: '/og-image.png',
@ -72,7 +72,7 @@ export const metadata: Metadata = {
creator: '@tpmjs_registry',
title: 'TPMJS - Tool Package Manager for AI Agents',
description:
'The registry for AI tools. Discover, share, and integrate tools that give your agents superpowers.',
'Discover and use npm packages as AI agent tools. No config files, automatic discovery, works with any framework.',
images: ['/og-image.png'],
},
robots: {
@ -100,7 +100,7 @@ export default function RootLayout({
url: 'https://tpmjs.com',
logo: 'https://tpmjs.com/logo.png',
description:
'The registry for AI tools. Discover, share, and integrate tools that give your agents superpowers.',
'Discover and use npm packages as AI agent tools. No config files, automatic discovery, works with any framework.',
sameAs: ['https://github.com/tpmjs/tpmjs', 'https://x.com/tpmjs_registry'],
};
@ -110,7 +110,7 @@ export default function RootLayout({
name: 'TPMJS',
url: 'https://tpmjs.com',
description:
'The registry for AI tools. Discover, share, and integrate tools that give your agents superpowers.',
'Discover and use npm packages as AI agent tools. No config files, automatic discovery, works with any framework.',
potentialAction: {
'@type': 'SearchAction',
target: {

72
apps/web/src/app/page.tsx
View file

@ -83,6 +83,76 @@ export default async function HomePage(): Promise<React.ReactElement> {
{/* Hero Section - Dithered Design */}
<HeroSection stats={data.stats} />
{/* What is TPMJS? Explainer Section */}
<section className="py-16 bg-surface border-y border-border">
<Container size="xl" padding="lg">
<div className="max-w-4xl mx-auto">
<h2 className="text-2xl md:text-3xl font-bold mb-8 text-foreground">
What is TPMJS?
</h2>
<div className="grid md:grid-cols-2 gap-8 mb-10">
{/* The Problem */}
<div className="p-6 border border-border rounded-lg bg-background">
<h3 className="font-semibold text-lg mb-3 text-foreground flex items-center gap-2">
<span className="text-red-500">✗</span> The Problem
</h3>
<p className="text-foreground-secondary text-sm leading-relaxed">
AI agents need tools (web scraping, file processing, API calls) but developers
must manually import and configure each one. As tooling grows, this becomes
unmanageable—hundreds of imports, version conflicts, and static capabilities.
</p>
</div>
{/* The Solution */}
<div className="p-6 border border-border rounded-lg bg-background">
<h3 className="font-semibold text-lg mb-3 text-foreground flex items-center gap-2">
<span className="text-green-500">✓</span> The Solution
</h3>
<p className="text-foreground-secondary text-sm leading-relaxed">
TPMJS is a registry that automatically discovers npm packages built for AI
agents. Your agent searches by description and loads tools at runtime—no config
files, no manual imports, always up-to-date.
</p>
</div>
</div>
{/* How it works - 3 steps */}
<div className="grid md:grid-cols-3 gap-6">
<div className="text-center p-4">
<div className="w-10 h-10 rounded-full bg-brutalist-accent text-foreground font-bold flex items-center justify-center mx-auto mb-3">
1
</div>
<h4 className="font-semibold mb-2 text-foreground">Publish to npm</h4>
<p className="text-sm text-foreground-secondary">
Add <code className="bg-surface px-1 rounded">tpmjs-tool</code> keyword to your
package.json and publish normally
</p>
</div>
<div className="text-center p-4">
<div className="w-10 h-10 rounded-full bg-brutalist-accent text-foreground font-bold flex items-center justify-center mx-auto mb-3">
2
</div>
<h4 className="font-semibold mb-2 text-foreground">Auto-indexed</h4>
<p className="text-sm text-foreground-secondary">
TPMJS discovers your package within 15 minutes and extracts tool schemas
automatically
</p>
</div>
<div className="text-center p-4">
<div className="w-10 h-10 rounded-full bg-brutalist-accent text-foreground font-bold flex items-center justify-center mx-auto mb-3">
3
</div>
<h4 className="font-semibold mb-2 text-foreground">Agents discover it</h4>
<p className="text-sm text-foreground-secondary">
Any AI agent can now find and use your tool by describing what they need
</p>
</div>
</div>
</div>
</Container>
</section>
{/* Featured Tools Section */}
<section className="py-16 bg-background">
<Container size="xl" padding="lg">
@ -173,7 +243,7 @@ export default async function HomePage(): Promise<React.ReactElement> {
</h2>
<p className="text-lg text-foreground-secondary mb-8">
Share your tool with the AI community. Automatic discovery, quality scoring, and
seamless integration with popular AI frameworks.
integration with Vercel AI SDK, LangChain, and more.
</p>
{/* Generator Highlight Box */}

16
apps/web/src/app/privacy/page.tsx
View file

@ -20,7 +20,7 @@ export default function PrivacyPage(): React.ReactElement {
<p className="text-xl text-foreground-secondary max-w-2xl mx-auto">
How we collect, use, and protect your data
</p>
<p className="text-sm text-foreground-tertiary mt-4">Last updated: December 14, 2025</p>
<p className="text-sm text-foreground-tertiary mt-4">Last updated: December 2024</p>
</div>
{/* Introduction */}
@ -432,11 +432,8 @@ export default function PrivacyPage(): React.ReactElement {
<p className="mt-6 text-foreground-secondary">
To exercise any of these rights, contact us at{' '}
<a
href="mailto:thomasalwyndavis@gmail.com"
className="text-primary hover:underline font-medium"
>
thomasalwyndavis@gmail.com
<a href="mailto:hello@tpmjs.com" className="text-primary hover:underline font-medium">
hello@tpmjs.com
</a>
. We will respond within 30 days.
</p>
@ -542,11 +539,8 @@ export default function PrivacyPage(): React.ReactElement {
<div className="space-y-3">
<div>
<h3 className="font-semibold text-foreground mb-1">Email</h3>
<a
href="mailto:thomasalwyndavis@gmail.com"
className="text-primary hover:underline"
>
thomasalwyndavis@gmail.com
<a href="mailto:hello@tpmjs.com" className="text-primary hover:underline">
hello@tpmjs.com
</a>
</div>

4
apps/web/src/app/sdk/page.tsx
View file

@ -578,8 +578,8 @@ Use registrySearch to find tools, then registryExecute to run them.\`,
<p>
We&apos;re building the{' '}
<span className="text-foreground font-semibold">npm for AI tools</span>. Just as npm
revolutionized JavaScript package sharing, TPMJS aims to create a universal
ecosystem where AI agents can discover, share, and execute tools seamlessly.
changed how developers share JavaScript packages, TPMJS aims to do the same for AI
agent tools—a universal ecosystem where agents discover and use tools on-demand.
</p>
<p>
The <code className="text-primary">registrySearch</code> and{' '}

8
apps/web/src/app/terms/page.tsx
View file

@ -19,7 +19,7 @@ export default function TermsPage(): React.ReactElement {
<h1 className="text-4xl md:text-5xl font-bold mb-4 text-foreground">
Terms of Service
</h1>
<p className="text-lg text-foreground-secondary">Last updated: December 14, 2025</p>
<p className="text-lg text-foreground-secondary">Last updated: December 2024</p>
</div>
{/* Content */}
@ -372,10 +372,10 @@ export default function TermsPage(): React.ReactElement {
</p>
<p>
<a
href="mailto:thomasalwyndavis@gmail.com"
href="mailto:hello@tpmjs.com"
className="text-primary hover:underline font-medium"
>
thomasalwyndavis@gmail.com
hello@tpmjs.com
</a>
</p>
</div>
@ -416,7 +416,7 @@ export default function TermsPage(): React.ReactElement {
We&apos;re here to help. Reach out if you need clarification on anything.
</p>
<div className="flex flex-col sm:flex-row gap-4 justify-center">
<a href="mailto:thomasalwyndavis@gmail.com">
<a href="mailto:hello@tpmjs.com">
<button
type="button"
className="px-6 py-3 bg-primary text-primary-foreground rounded-lg font-medium hover:bg-primary/90 transition-colors"

3
apps/web/src/app/tool/tool-search/page.tsx
View file

@ -138,7 +138,8 @@ export default function ToolSearchPage(): React.ReactElement {
<div className="space-y-4 mb-8">
<h1 className="text-4xl font-bold text-foreground">Tool Registry</h1>
<p className="text-lg text-foreground-secondary">
Discover, share, and integrate tools that give your AI agents superpowers.
Search npm packages indexed as AI agent tools. Filter by category, health status, or
keyword.
</p>
</div>

2
apps/web/src/components/AppFooter.tsx
View file

@ -11,7 +11,7 @@ export function AppFooter(): React.ReactElement {
<p className="text-sm text-foreground-secondary">© 2025 TPMJS. All rights reserved.</p>
<div className="flex items-center gap-4 text-sm">
<a
href="mailto:thomasalwyndavis@gmail.com"
href="mailto:hello@tpmjs.com"
className="text-foreground-secondary hover:text-foreground transition-colors"
>
Contact

2
apps/web/src/components/SDKFlowDiagram.tsx
View file

@ -530,7 +530,7 @@ export function SDKFlowDiagram(): React.ReactElement {
<div className="text-foreground-secondary">
<span className="font-semibold text-foreground">Your existing tools</span> — Any AI
SDK tools you&apos;ve already built or installed. These work alongside the registry
tools seamlessly.
tools without conflicts.
</div>
)}
{hoveredNode === 'registry-search' && (

24
apps/web/src/components/home/HeroSection.tsx
View file

@ -62,14 +62,23 @@ export function HeroSection({ stats }: HeroSectionProps): React.ReactElement {
{/* Main Heading */}
<div className="max-w-7xl">
<h1
className="mb-8 font-bold leading-none tracking-tight text-foreground"
style={{ fontSize: 'clamp(48px, 10vw, 96px)' }}
className="mb-6 font-bold leading-none tracking-tight text-foreground"
style={{ fontSize: 'clamp(42px, 8vw, 80px)' }}
>
TOOL REGISTRY FOR AI AGENTS
NPM PACKAGES YOUR AI AGENT CAN DISCOVER
</h1>
{/* Clear value prop */}
<p className="mb-8 max-w-3xl text-xl md:text-2xl font-medium leading-relaxed text-foreground-secondary tracking-tight">
TPMJS indexes npm packages as tools that AI agents can find and use at runtime.
<br />
<span className="text-foreground">
No config files. No manual imports. Just describe what you need.
</span>
</p>
{/* Live Metrics Strip */}
<div className="mb-12 flex flex-wrap items-center gap-3 border-l-[6px] border-brutalist-accent pl-6 font-mono text-base md:text-lg font-bold uppercase tracking-wider">
<div className="mb-10 flex flex-wrap items-center gap-3 border-l-[6px] border-brutalist-accent pl-6 font-mono text-base md:text-lg font-bold uppercase tracking-wider">
<div className="flex items-center gap-2">
<span className="text-foreground">{formatNumber(stats.packageCount)}</span>
<span className="text-foreground-secondary">PACKAGES</span>
@ -81,13 +90,6 @@ export function HeroSection({ stats }: HeroSectionProps): React.ReactElement {
</div>
</div>
{/* Subheading */}
<p className="mb-12 max-w-3xl text-xl md:text-2xl font-medium leading-relaxed text-foreground-secondary tracking-tight">
Discover, share, and integrate tools that give your agents superpowers.
<br />
The registry for AI tools.
</p>
{/* Brutalist Search Interface */}
<div className="max-w-3xl">
<div className="relative">

2
apps/web/src/components/home/VisionSection.tsx
View file

@ -63,7 +63,7 @@ export function VisionSection(): React.ReactElement {
</div>
<div className="mt-12 text-center font-mono text-sm text-brutalist-accent uppercase tracking-wider">
The registry that gives agents superpowers
One registry. Thousands of tools. Zero configuration.
</div>
</Container>
</section>

61
broken-tools.md
View file

@ -1,61 +0,0 @@
# Broken Tools Classification
This document tracks the different types of tool failures encountered in the TPMJS executor and the strategies for handling them.
## Error Categories
| Error Type | Example | Root Cause | Strategy |
|------------|---------|------------|----------|
| **Invalid structure** | `fish-joke-generator`, `@tpmjs/text-transformer` | Not an AI SDK tool (missing `description` or `execute`) | Mark as BROKEN, filter from search results |
| **Module not found** | `@thomasdavis/cows@0.0.1` | Package doesn't exist on npm/esm.sh | Mark as BROKEN, consider removing from registry |
| **Factory function** | `@tavily/ai-sdk/tavilySearch` | Tool is a factory that needs config to initialize | Need to detect and call with appropriate config |
| **Missing env var** | `@exalabs/ai-sdk/webSearch` | Requires API key (e.g., `EXA_API_KEY`) not provided | Import: HEALTHY, Execution: BROKEN with clear error message |
| **Missing execution context** | `@parallel-web/ai-sdk-tools/extractTool` | Tool expects `{ abortSignal }` as 2nd arg to `execute()` | Fix executor to pass execution context |
## Detailed Examples
### Invalid Structure
```
❌ Invalid AI SDK tool structure: {
hasDescription: false,
hasExecute: false,
hasInputSchema: false,
keys: ["FishJokeSchema", "fishJoker", "createFishJoker", ...]
}
```
These packages export utility functions or schemas, not AI SDK tools.
### Module Not Found
```
❌ Failed to load tool: TypeError: Module not found "https://esm.sh/@thomasdavis/cows@0.0.1"
```
Package was registered but doesn't exist on npm or was unpublished.
### Factory Function
```
❌ Tool "tavilySearch" is a factory function but couldn't be initialized.
Tried: no-args, config object, and single-arg patterns.
Hint: This tool may require specific configuration. Check package documentation.
```
Tool exports a factory like `tavilySearch({ apiKey })` instead of a ready-to-use tool object.
### Missing Env Var
```
❌ EXA_API_KEY is required. Set it in environment variables or pass it in config.
```
Tool loaded successfully but execution fails without required credentials.
### Missing Execution Context
```
❌ Tool execution failed: TypeError: Cannot destructure property 'abortSignal' of 'undefined' as it is undefined.
at Object.execute (https://esm.sh/@parallel-web/ai-sdk-tools@0.1.6/...)
```
AI SDK tools expect `execute(params, { abortSignal, ... })` but executor only passes params.
## Resolution Status
- [x] Invalid structure - Health check marks as BROKEN ✓
- [x] Module not found - Health check marks as BROKEN ✓
- [ ] Factory function - Partial support (tries common patterns)
- [x] Missing env var - Import: HEALTHY, Execution: HEALTHY (config issue, not broken)
- [x] Missing execution context - **Fixed in executor** (commit 0804f1b)

12
package.json
View file

@ -1,6 +1,18 @@
{
"name": "@tpmjs/monorepo",
"version": "0.0.0",
"description": "Tool Package Manager for AI Agents - discover, share, and integrate tools",
"author": "TPMJS",
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/tpmjs/tpmjs.git"
},
"homepage": "https://tpmjs.com",
"bugs": {
"url": "https://github.com/tpmjs/tpmjs/issues"
},
"keywords": ["tpmjs", "ai", "agents", "tools", "mcp", "typescript", "npm"],
"private": true,
"type": "module",
"engines": {

10
packages/env/package.json vendored
View file

@ -1,6 +1,16 @@
{
"name": "@tpmjs/env",
"version": "0.1.1",
"description": "Environment variable validation with Zod for TPMJS",
"author": "TPMJS",
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/tpmjs/tpmjs.git",
"directory": "packages/env"
},
"homepage": "https://tpmjs.com",
"keywords": ["tpmjs", "typescript", "environment", "zod", "validation"],
"type": "module",
"exports": {
".": {

10
packages/types/package.json
View file

@ -1,6 +1,16 @@
{
"name": "@tpmjs/types",
"version": "0.1.1",
"description": "Shared TypeScript types and Zod schemas for TPMJS",
"author": "TPMJS",
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/tpmjs/tpmjs.git",
"directory": "packages/types"
},
"homepage": "https://tpmjs.com",
"keywords": ["tpmjs", "typescript", "types", "zod", "schemas"],
"type": "module",
"exports": {
"./tool": {

10
packages/ui/package.json
View file

@ -1,6 +1,16 @@
{
"name": "@tpmjs/ui",
"version": "0.1.3",
"description": "React component library for TPMJS applications",
"author": "TPMJS",
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/tpmjs/tpmjs.git",
"directory": "packages/ui"
},
"homepage": "https://tpmjs.com",
"keywords": ["tpmjs", "react", "components", "typescript", "ui"],
"type": "module",
"exports": {
"./Button/Button": {

10
packages/utils/package.json
View file

@ -1,6 +1,16 @@
{
"name": "@tpmjs/utils",
"version": "0.1.1",
"description": "Utility functions for TPMJS applications",
"author": "TPMJS",
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/tpmjs/tpmjs.git",
"directory": "packages/utils"
},
"homepage": "https://tpmjs.com",
"keywords": ["tpmjs", "typescript", "utilities", "tailwind"],
"type": "module",
"exports": {
"./cn": {