tpmjs/CLAUDE.md

Monorepo Setup

This project uses a Turborepo monorepo architecture with the following structure:

Packages

Published to npm (@tpmjs scope):

• @tpmjs/ui - React component library with .ts-only components
• @tpmjs/utils - Utility functions (cn, format, etc.)
• @tpmjs/types - Shared TypeScript types and Zod schemas
• @tpmjs/env - Environment variable validation with Zod

Internal tooling (private):

• @tpmjs/config - Shared configurations (Biome, ESLint, Tailwind, TypeScript)
• @tpmjs/eslint-config - ESLint configuration with module boundary rules
• @tpmjs/tailwind-config - Tailwind configuration with design tokens
• @tpmjs/tsconfig - TypeScript configurations (base, nextjs, react-library)
• @tpmjs/test - Vitest shared configuration
• @tpmjs/mocks - MSW mock server for testing
• @tpmjs/storybook - Component documentation and showcase

Applications

• @tpmjs/web - Next.js 16 App Router application (main website)

Architecture Principles

1. Design System First

Always use @tpmjs/ui components instead of raw HTML elements. This ensures visual consistency, accessibility, and maintainability across the application.

// Good - use design system components
import { Button } from '@tpmjs/ui/Button/Button';
import { Input } from '@tpmjs/ui/Input/Input';
import { Table, TableRow, TableCell } from '@tpmjs/ui/Table/Table';

<Button onClick={handleClick}>Submit</Button>
<Input value={value} onChange={onChange} />

// Bad - raw HTML elements
<button onClick={handleClick}>Submit</button>
<input value={value} onChange={onChange} />

When to create/update UI components:

• If a pattern is used in 2+ places, create a reusable component in @tpmjs/ui
• If existing component styling doesn't match the design, update the component (not the usage site)
• If you need virtualization (e.g., react-virtuoso), match the design system's styling classes

Common components to use:

• Button - all clickable actions
• Input, Select, Textarea - form inputs
• Table, TableRow, TableCell - data tables
• Card - content containers
• Badge - status indicators
• Icon - all icons (not inline SVGs)
• Spinner, Skeleton - loading states

2. No Barrel Exports

Components are imported directly without index.ts files:

// Good
import { Button } from '@tpmjs/ui/Button/Button';

// Bad (not allowed)
import { Button } from '@tpmjs/ui';

Benefits:

• Clearer dependency graphs
• Better tree-shaking
• Prevents circular dependencies
• Explicit imports

3. Module Boundaries

ESLint enforces strict module boundaries:

• Apps can only import from published packages
• Packages cannot import from apps
• UI package cannot import from utils (stays dependency-free)

4. Shared Configurations

All configuration is centralized in packages/config/:

• Biome - Formatting + basic linting
• ESLint - Semantic rules and module boundaries
• Tailwind - Design tokens and shared theme
• TypeScript - Multiple configs for different contexts

Development Workflow

# Install dependencies
pnpm install

# Run development servers
pnpm dev

# Build all packages
pnpm build

# Run tests
pnpm test

# Lint and format
pnpm lint
pnpm format

Detailed Development Commands

This section documents the complete testing, building, and development workflow used in this monorepo.

Testing Individual Packages

Use the --filter flag to target specific packages:

# Type-check a single package
pnpm --filter=@tpmjs/npm-client type-check
pnpm --filter=@tpmjs/ui type-check
pnpm --filter=@tpmjs/web type-check

# Run tests in a single package
pnpm --filter=@tpmjs/ui test
pnpm --filter=@tpmjs/web test

# Lint a single package
pnpm --filter=@tpmjs/web lint

Testing All Packages

Commands from the root run across all packages via Turborepo:

# Type-check all packages (runs via Turborepo)
pnpm type-check

# Lint all packages (runs via Turborepo)
pnpm lint

# Format all files with Biome
pnpm format

# Test all packages (runs via Turborepo)
pnpm test

Building Packages

Build commands respect dependency order automatically:

# Build a single package (and its dependencies)
pnpm --filter=@tpmjs/ui build
pnpm --filter=@tpmjs/types build

# Build all packages
pnpm build

# Build and watch for changes
pnpm --filter=@tpmjs/ui dev

Database Commands (Prisma)

The @tpmjs/db package uses Prisma for database management:

# Generate Prisma client (required after schema changes)
pnpm --filter=@tpmjs/db db:generate

# Push schema changes to database (dev)
pnpm --filter=@tpmjs/db db:push

# Create and apply migrations (production)
pnpm --filter=@tpmjs/db db:migrate

# Open Prisma Studio (database GUI)
pnpm --filter=@tpmjs/db db:studio

# Seed the database
pnpm --filter=@tpmjs/db db:seed

Important: Always run pnpm --filter=@tpmjs/db db:generate after modifying schema.prisma to regenerate the Prisma client. Without this, TypeScript will show errors for database types.

Development Servers

# Run Next.js dev server for web app
pnpm dev --filter=@tpmjs/web

# Run all dev servers (if multiple apps)
pnpm dev

# Run Storybook for component development
pnpm --filter=@tpmjs/storybook dev

Pre-commit Hooks (Lefthook)

Git commits automatically trigger these checks via Lefthook:

1. Format - Biome formats all staged files
2. Lint - Runs pnpm lint across all packages
3. Type-check - Runs pnpm type-check across all packages

If any check fails, the commit is blocked. The hooks ensure code quality before changes reach CI.

Note: Pre-commit hooks run the same checks as CI, so if they pass locally, CI should pass too.

Turborepo Caching

Turborepo caches task outputs for faster rebuilds:

• Cache hits: Tasks show cache hit, replaying logs - no actual work done
• Cache miss: Tasks execute normally and outputs are cached
• Invalidation: Cache invalidates when inputs change (source files, dependencies, env vars)

# Clear Turborepo cache if needed
pnpm turbo clean

# Force rebuild without cache
pnpm build --force

Common Workflows

After pulling new changes:

pnpm install              # Install new dependencies
pnpm db:generate          # Regenerate Prisma client if schema changed
pnpm type-check           # Verify everything type-checks
pnpm dev --filter=@tpmjs/web  # Start dev server

Creating a new package:

# 1. Create package directory and files
mkdir -p packages/my-package/src
cd packages/my-package

# 2. Create package.json with proper name and workspace dependencies
# 3. Create tsconfig.json extending @tpmjs/tsconfig

# 4. Install dependencies from root
cd ../..
pnpm install

# 5. Type-check the new package
pnpm --filter=@tpmjs/my-package type-check

Testing before committing:

# Run the same checks that pre-commit hooks will run
pnpm format               # Format all files
pnpm lint                 # Lint all packages
pnpm type-check           # Type-check all packages

# Then commit - hooks should pass quickly
git add .
git commit -m "your message"

Troubleshooting

"Cannot find module '@prisma/client'"

• Run pnpm --filter=@tpmjs/db db:generate to generate the Prisma client
• The Prisma client must be generated after any schema changes or fresh installs

"Type error in package that imports from another package"

• Build the dependency first: pnpm --filter=@tpmjs/types build
• Or build all packages: pnpm build
• Turborepo handles this automatically when using pnpm build

"Biome formatting errors in pre-commit"

• Run pnpm format to auto-fix formatting issues
• Biome will format all files according to the config

"ESLint warnings about module boundaries"

• Check that you're not importing from apps in packages
• Check that imports follow the no-barrel-exports rule
• Example: Use @tpmjs/ui/Button/Button not @tpmjs/ui

"Turborepo cache shows stale outputs"

• Clear cache with pnpm turbo clean
• Force rebuild with pnpm build --force

"Dev server won't start"

• Check that all dependencies are installed: pnpm install
• Check that Prisma client is generated: pnpm db:generate
• Check for port conflicts (Next.js default: 3000)

Publishing Flow

1. Make changes to packages
2. Create changeset: pnpm changeset
3. Version packages: pnpm changeset:version
4. Publish to npm: pnpm changeset:publish
5. Push with tags: git push --follow-tags

Tech Stack

• Build System: Turborepo
• Package Manager: pnpm
• TypeScript: Strict mode, composite projects
• React: v19
• Next.js: v16 App Router
• Styling: Tailwind CSS
• Testing: Vitest + Testing Library
• Linting: Biome + ESLint
• Documentation: Storybook
• CI/CD: GitHub Actions + Changesets
• Git Hooks: Lefthook

Debugging CI/CD with CLI Tools

When debugging CI failures or deployment issues, use command-line tools for efficient investigation:

GitHub CLI (gh)

Debug GitHub Actions CI runs:

# List recent workflow runs
gh run list --limit 10

# View specific run details
gh run view <run-id>

# View failed job logs
gh run view <run-id> --log-failed

# View specific job logs
gh run view <run-id> --job <job-id> --log

# Rerun failed jobs
gh run rerun <run-id> --failed

Common debugging workflow:

1. gh run list - Find the failed run ID
2. gh run view <run-id> --log-failed - See what failed
3. Fix the issue locally
4. Push and monitor: gh run watch

Vercel CLI

Debug deployments and preview environments:

# List deployments
vercel ls

# View deployment details
vercel inspect <deployment-url>

# View deployment logs
vercel logs <deployment-url>

# Pull environment variables
vercel env pull

# Link local project to Vercel project
vercel link

Common debugging workflow:

1. vercel ls - Find the deployment URL
2. vercel inspect <url> - Check deployment status and build logs
3. vercel logs <url> - View runtime logs
4. Compare env vars: vercel env pull and check .env.local

Tips

• Use gh and vercel CLIs to debug without leaving the terminal
• Check CI logs before making blind fixes
• Vercel deployments are blocked until GitHub Actions pass (configured in vercel.json)
• Pre-commit/pre-push hooks run the same checks as CI - if they pass locally, CI should pass too

---

Case Study: Fixing API Route Timeouts on tpmjs.com

This is a detailed account of debugging and fixing API route timeouts in production. The investigation revealed critical insights about deploying Turborepo monorepos to Vercel with Prisma.

The Problem

After deploying tpmjs.com to production, all API endpoints were timing out:

$ curl https://tpmjs.com/api/health
# Request timed out after 60 seconds

$ curl https://tpmjs.com/api/tools
# Request timed out after 60 seconds

The Next.js UI worked perfectly - pages loaded, navigation functioned - but every API route request resulted in a timeout. No errors appeared in Vercel logs, and the requests never even reached the serverless functions.

Initial Investigation

Step 1: Verify Build Output

vercel inspect <deployment-url>

The build showed pages but no API routes listed as lambda functions:

Builds
  ├── ○ / (static page)
  ├── ○ /playground (static page)
  └── ○ /tool/[slug] (static page)

# Expected to see:
  ├── λ api/health
  ├── λ api/tools
  └── λ api/sync/changes

This confirmed Vercel wasn't treating the project as Next.js - it was using static site generation and dropping all API routes.

Step 2: Check Vercel Configuration

Examined apps/web/vercel.json:

{
  "$schema": "https://openapi.vercel.sh/vercel.json",
  "buildCommand": "cd ../.. && turbo build --filter=@tpmjs/web",
  "installCommand": "pnpm install"
}

The build command looked correct, but the custom build command was bypassing Vercel's Next.js detection.

Root Cause #1: Workspace Dependencies Not Built

Deployed the site and checked the build logs. Found this critical error:

Module not found: Can't resolve '@tpmjs/env'
Module not found: Can't resolve '@tpmjs/types/tpmjs'
Module not found: Can't resolve '@tpmjs/ui/Badge/Badge'
Package @prisma/client can't be external

49 module resolution errors - the workspace packages weren't being built before the web app tried to import them.

The Fix:

Changed the build command from:

"buildCommand": "cd ../.. && turbo build --filter=@tpmjs/web"

To:

"buildCommand": "cd ../.. && pnpm install && pnpm --filter=@tpmjs/web... build"

The ... suffix in --filter=@tpmjs/web... tells pnpm to build ALL dependencies first:

1. Build @tpmjs/env
2. Build @tpmjs/types
3. Build @tpmjs/ui
4. Build @tpmjs/utils
5. Build @tpmjs/db (Prisma generate)
6. Finally build @tpmjs/web

After this change, the build succeeded and API routes appeared as lambda functions in vercel inspect.

Root Cause #2: Prisma Cold Start Performance

With the build fixed, API routes were deployed but still timing out. Testing revealed:

# Health endpoint (no database) - WORKS
$ curl https://tpmjs.com/api/health
{"status":"ok","timestamp":"2025-11-28T11:32:29.295Z"}

# Tools endpoint (with database) - TIMEOUT
$ curl https://tpmjs.com/api/tools
# ...60 second timeout

Local Database Performance Test:

// Test the exact queries used in production
const count = await prisma.tool.count();        // 3.244s ⚠️
const tools = await prisma.tool.findMany({
  orderBy: [
    { qualityScore: 'desc' },
    { npmDownloadsLastMonth: 'desc' },
    { createdAt: 'desc' },
  ],
  take: 20,
});                                             // 593ms ⚠️

The parallel count() + findMany() queries were taking 3.8 seconds due to Prisma cold start in serverless environments.

The Fix:

Optimized the endpoint in two ways:

1. Removed expensive count query - Using count() in every request is slow and usually unnecessary:

// Before: Slow parallel queries
const [tools, totalCount] = await Promise.all([
  prisma.tool.findMany({ where, take: limit, skip: offset }),
  prisma.tool.count({ where }),  // ← 3+ seconds!
]);

// After: Fast single query with limit+1 technique
const tools = await prisma.tool.findMany({
  where,
  take: limit + 1,  // Fetch one extra to check if more exist
  skip: offset,
});

const hasMore = tools.length > limit;
const actualTools = hasMore ? tools.slice(0, limit) : tools;

2. Reduced max page size from 100 to 50 items for better performance

Results:

$ curl https://tpmjs.com/api/tools
{
  "success": true,
  "data": [...],  # Returns in <1 second
  "pagination": {
    "limit": 20,
    "offset": 0,
    "hasMore": false
  }
}

Additional Optimizations Applied

Added maxDuration to all API routes:

export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const maxDuration = 60;  // ← Prevent premature timeouts

Verified database indexes exist:

model Tool {
  // ... fields ...

  @@index([category])
  @@index([isOfficial])
  @@index([qualityScore])
  @@index([npmDownloadsLastMonth])
  @@index([createdAt])
}

All necessary indexes were present - the issue was cold start latency, not missing indexes.

Key Lessons Learned

1. Monorepo Build Order Matters

When deploying Turborepo monorepos to Vercel, workspace dependencies MUST be built first:

# ❌ Wrong - only builds the web app
pnpm --filter=@tpmjs/web build

# ✅ Correct - builds dependencies first
pnpm --filter=@tpmjs/web... build

2. Prisma in Serverless = Slow First Request

Prisma Client initialization in serverless environments adds 1-3 seconds of latency on cold starts. Strategies to mitigate:

• Use connection pooling (Neon, PlanetScale)
• Eliminate unnecessary queries (especially count())
• Cache query results when possible
• Consider Prisma Accelerate for critical paths

3. Progressive Debugging Approach

Start simple and progressively add complexity:

// Step 1: Does endpoint respond at all?
export async function GET() {
  return NextResponse.json({ status: 'ok' });
}

// Step 2: Can we connect to database?
export async function GET() {
  return NextResponse.json({
    hasDatabase: !!process.env.DATABASE_URL,
  });
}

// Step 3: Can we query the database?
export async function GET() {
  const count = await prisma.tool.count();
  return NextResponse.json({ count });
}

// Step 4: Full implementation with optimizations

4. Use Vercel CLI for Debugging

# Check what's actually deployed
vercel inspect <deployment-url>

# Look for lambda functions (λ)
Builds
  ├── λ api/health ✅
  ├── λ api/tools ✅

# If you see only static pages (○), API routes aren't deployed

Final Configuration

apps/web/vercel.json:

{
  "$schema": "https://openapi.vercel.sh/vercel.json",
  "buildCommand": "cd ../.. && pnpm install && pnpm --filter=@tpmjs/web... build",
  "installCommand": "pnpm install"
}

API Route Template:

import { prisma } from '@tpmjs/db';
import { NextResponse } from 'next/server';

export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const maxDuration = 60;

export async function GET() {
  try {
    // Avoid count() - use limit+1 technique instead
    const items = await prisma.tool.findMany({
      orderBy: { qualityScore: 'desc' },
      take: 21,  // Request 1 more than needed
    });

    const hasMore = items.length > 20;
    const data = hasMore ? items.slice(0, 20) : items;

    return NextResponse.json({
      success: true,
      data,
      pagination: { hasMore },
    });
  } catch (error) {
    return NextResponse.json(
      { success: false, error: error.message },
      { status: 500 }
    );
  }
}

Performance Metrics

Before Optimization:

• /api/health: Timeout (60s+)
• /api/tools: Timeout (60s+)
• Build: Failed (module resolution errors)

After Optimization:

• /api/health: ~50ms ✅
• /api/tools: ~800ms ✅
• Build: Success (all deps built) ✅

Conclusion

API timeouts in serverless environments often stem from build configuration issues or database cold starts. For Turborepo + Vercel + Prisma:

1. Use pnpm --filter=package... to build dependencies
2. Avoid count() queries in hot paths
3. Add maxDuration to API routes
4. Use vercel inspect to verify lambda deployment
5. Test database performance locally before deploying

The full working implementation is live at tpmjs.com.

---

NPM Package Syncing System

TPMJS.com automatically mirrors npm packages with the tpmjs keyword to keep the tool registry up-to-date. This section documents how the syncing system works.

Overview

The sync system uses three automated strategies running on Vercel Cron to discover and update TPMJS tools:

1. Changes Feed - Monitors npm's real-time changes feed for all package updates
2. Keyword Search - Actively searches npm for packages with the tpmjs keyword
3. Metrics Sync - Updates download stats and calculates quality scores

Sync Endpoints

All sync endpoints are located in apps/web/src/app/api/sync/:

1. Changes Feed Sync (/api/sync/changes)

Purpose: Monitors npm's changes feed to catch new packages and updates in real-time.

Schedule: Every 2 minutes (*/2 * * * *)

How it works:

1. Fetches the last checkpoint sequence number from the database
2. Calls npm's /_changes endpoint with since=<lastSeq> (limit 100 per run)
3. For each changed package, fetches full metadata with fetchLatestPackageWithMetadata()
4. Validates that the package has a valid tpmjs field using validateTpmjsField()
5. Upserts the tool to the database with discoveryMethod: 'changes-feed'
6. Updates the checkpoint with the new sequence number for next run

Key Features:

• Uses checkpoints to track progress and avoid reprocessing
• Processes up to 100 changes per run to avoid timeouts
• Logs all sync operations to syncLog table
• Requires Authorization: Bearer <CRON_SECRET> header

Example Response:

{
  "success": true,
  "data": {
    "processed": 5,
    "skipped": 93,
    "errors": 0,
    "lastSeq": "12345678",
    "pending": 1250,
    "durationMs": 2834
  }
}

2. Keyword Search Sync (/api/sync/keyword)

Purpose: Actively searches npm for packages with the tpmjs keyword.

Schedule: Every 15 minutes (*/15 * * * *)

How it works:

1. Searches npm registry for packages with keyword tpmjs (up to 250 results)
2. Fetches full metadata for each package
3. Validates the tpmjs field
4. Upserts tools with discoveryMethod: 'keyword'
5. Updates checkpoint with last run timestamp

Key Features:

• Catches packages that might be missed by changes feed
• Useful for backfilling existing packages
• Processes up to 250 packages per run

Example Response:

{
  "success": true,
  "data": {
    "processed": 12,
    "skipped": 3,
    "errors": 0,
    "packagesFound": 15,
    "durationMs": 4521
  }
}

3. Metrics Sync (/api/sync/metrics)

Purpose: Updates download statistics and calculates quality scores for all tools.

Schedule: Every hour (0 * * * *)

How it works:

1. Fetches all tools from the database
2. For each tool, calls fetchDownloadStats() to get last 30 days of downloads
3. Calculates quality score based on:
   • Tier (rich = 0.6, minimal = 0.4)
   • Downloads (logarithmic scale, max 0.3)
   • GitHub stars (logarithmic scale, max 0.1)
4. Updates npmDownloadsLastMonth and qualityScore fields

Quality Score Formula:

function calculateQualityScore(params: {
  tier: string;
  downloads: number;
  githubStars: number;
}): number {
  const tierScore = tier === 'rich' ? 0.6 : 0.4;
  const downloadsScore = Math.min(0.3, Math.log10(downloads + 1) / 10);
  const starsScore = Math.min(0.1, Math.log10(githubStars + 1) / 10);
  return Math.min(1.0, tierScore + downloadsScore + starsScore);
}

Example Response:

{
  "success": true,
  "data": {
    "processed": 25,
    "skipped": 0,
    "errors": 0,
    "totalTools": 25,
    "durationMs": 8234
  }
}

Automated Sync Configuration

The sync system can run via two methods:

Option 1: Vercel Cron (Primary)

Cron jobs are configured in vercel.json at the repository root:

{
  "crons": [
    {
      "path": "/api/sync/changes",
      "schedule": "*/2 * * * *"
    },
    {
      "path": "/api/sync/keyword",
      "schedule": "*/15 * * * *"
    },
    {
      "path": "/api/sync/metrics",
      "schedule": "0 * * * *"
    }
  ]
}

Pros:

• Native Vercel integration
• Automatic authentication with CRON_SECRET
• Same infrastructure as the app
• No setup required (works automatically on deploy)

Option 2: GitHub Actions (Backup)

A GitHub Actions workflow (.github/workflows/sync.yml) provides redundancy:

name: NPM Package Sync

on:
  schedule:
    - cron: '*/2 * * * *'   # Changes feed
    - cron: '*/15 * * * *'  # Keyword search
    - cron: '0 * * * *'     # Metrics
  workflow_dispatch:        # Manual trigger

Pros:

• Redundancy if Vercel Cron fails
• Manual trigger via GitHub UI
• Free on GitHub (included in free tier)
• Runs from GitHub's infrastructure

Setup:

1. Add secrets to GitHub repository settings:

   • VERCEL_PRODUCTION_URL - Your production URL (e.g., https://tpmjs.com)
   • CRON_SECRET - Same secret used in Vercel environment variables

2. Enable GitHub Actions in repository settings

3. The workflow will run automatically on schedule OR manually via:

   • GitHub Actions tab → NPM Package Sync → Run workflow → Select sync type

Schedule Breakdown:

• Changes feed: Every 2 minutes (30 times per hour)
• Keyword search: Every 15 minutes (4 times per hour)
• Metrics: Every hour (once per hour)

Recommendation: Use Vercel Cron as primary and GitHub Actions as backup. Both can run simultaneously - the sync endpoints are idempotent.

Database Schema

The sync system uses these Prisma models:

Tool - The main tool registry:

model Tool {
  id                   String   @id @default(cuid())
  npmPackageName       String   @unique
  npmVersion           String
  npmDownloadsLastMonth Int     @default(0)
  qualityScore         Float?
  discoveryMethod      String   // 'changes-feed' | 'keyword'
  tier                 String   // 'minimal' | 'rich'
  // ... other fields

  @@index([qualityScore])
  @@index([npmDownloadsLastMonth])
}

SyncCheckpoint - Tracks sync progress:

model SyncCheckpoint {
  id         String @id @default(cuid())
  source     String @unique // 'changes-feed' | 'keyword-search' | 'metrics'
  checkpoint Json   // { lastSeq: string, lastRun: string, ... }
}

SyncLog - Records all sync operations:

model SyncLog {
  id        String   @id @default(cuid())
  source    String
  status    String   // 'success' | 'partial' | 'error'
  processed Int
  skipped   Int
  errors    Int
  message   String?
  metadata  Json?
  createdAt DateTime @default(now())
}

Manual Sync Triggers

To manually trigger a sync (useful for testing or debugging):

# Trigger changes feed sync
curl -X POST https://tpmjs.com/api/sync/changes \
  -H "Authorization: Bearer $CRON_SECRET"

# Trigger keyword search
curl -X POST https://tpmjs.com/api/sync/keyword \
  -H "Authorization: Bearer $CRON_SECRET"

# Trigger metrics update
curl -X POST https://tpmjs.com/api/sync/metrics \
  -H "Authorization: Bearer $CRON_SECRET"

Note: You need the CRON_SECRET environment variable set in Vercel. The endpoints return 401 Unauthorized without it.

Monitoring Sync Health

Check sync logs in the database:

// Get recent sync operations
const recentSyncs = await prisma.syncLog.findMany({
  orderBy: { createdAt: 'desc' },
  take: 20,
});

// Check last successful sync for each source
const checkpoints = await prisma.syncCheckpoint.findMany();

Sync Log Example:

{
  "id": "clx...",
  "source": "changes-feed",
  "status": "success",
  "processed": 5,
  "skipped": 93,
  "errors": 0,
  "message": "Successfully processed 5 packages",
  "metadata": {
    "durationMs": 2834,
    "lastSeq": "12345678",
    "pending": 1250
  },
  "createdAt": "2025-11-30T12:00:00Z"
}

Error Handling

All sync endpoints follow this error handling pattern:

1. Partial Success: If some packages fail but others succeed, status is partial
2. Complete Failure: If the entire sync fails, status is error
3. Error Messages: First 3 errors are included in the response
4. Logging: All operations are logged to syncLog regardless of success

Example Partial Failure:

{
  "success": true,
  "data": {
    "processed": 5,
    "skipped": 2,
    "errors": 3,
    "durationMs": 5234
  }
}

The sync log will contain:

{
  "status": "partial",
  "message": "Processed with errors: Failed to process pkg1: Network timeout; Failed to process pkg2: Invalid tpmjs field; ..."
}

Configuration

Required environment variables in Vercel:

# Database connection
DATABASE_URL="postgresql://..."

# Cron job authentication
CRON_SECRET="your-secret-key"

Important: Vercel Cron automatically adds the Authorization: Bearer $CRON_SECRET header when calling the endpoints. No manual configuration needed.

Performance Considerations

Timeouts:

• All sync routes have maxDuration: 300 (5 minutes)
• Changes feed processes max 100 packages per run to avoid timeouts
• Keyword search processes max 250 packages per run
• Metrics sync processes all tools but runs only once per hour

Rate Limiting:

• npm API has rate limits - be cautious when testing manually
• Vercel Cron jobs run from Vercel's infrastructure (different IP than dev)
• Consider implementing exponential backoff for npm API errors

Cold Starts:

• First request to each sync endpoint may be slow due to Prisma initialization
• Subsequent requests are faster with warm Prisma Client
• This is acceptable for background cron jobs

Debugging Sync Issues

Check if cron jobs are running:

# View recent deployments
vercel ls

# Check logs for a specific deployment
vercel logs <deployment-url>

# Filter for sync-related logs
vercel logs <deployment-url> | grep sync

Common issues:

1. "Unauthorized" errors: Check that CRON_SECRET is set in Vercel environment variables
2. Timeouts: Reduce batch size in changes feed (currently 100)
3. Missing packages: Check syncLog for errors during processing
4. Stale data: Verify metrics sync is running every hour

Test sync locally:

# Start dev server
pnpm dev --filter=@tpmjs/web

# Trigger sync (requires CRON_SECRET in .env.local)
curl -X POST http://localhost:3000/api/sync/changes \
  -H "Authorization: Bearer $CRON_SECRET"

Package Discovery Flow

Here's how a new TPMJS tool gets discovered:

1. Developer publishes package to npm with tpmjs keyword and tpmjs field in package.json
2. Within 2 minutes: Changes feed sync picks it up from npm's /_changes endpoint
3. Validation: validateTpmjsField() checks that the tpmjs field meets requirements
4. Database Insert: Tool is upserted with initial data
5. Within 1 hour: Metrics sync updates download stats and calculates quality score
6. Visible on tpmjs.com: Tool appears in search results and category pages

Backup Discovery: If changes feed misses a package, the keyword search (every 15 minutes) will catch it.

Future Improvements

Potential enhancements to the sync system:

• Add webhook endpoint for instant npm package notifications
• Implement exponential backoff for npm API rate limits
• Add Slack/Discord notifications for sync failures
• Create admin dashboard to monitor sync health
• Support GitHub stars syncing (requires GitHub API integration)
• Add sync metrics to Vercel Analytics
• Implement differential sync to reduce database writes

---

Verifying Production Deployments

To verify that a deployment has been successfully pushed to production, fetch the /api/health endpoint:

curl https://tpmjs.com/api/health

Response format:

{
  "status": "ok",
  "timestamp": "2025-01-07T13:30:00.000Z",
  "build": {
    "commitSha": "efb2802",
    "commitMessage": "feat: add public chat page for agents",
    "deploymentUrl": "tpmjs-xyz123.vercel.app"
  },
  "env": {
    "hasDatabase": true,
    "nodeEnv": "production"
  }
}

Key fields:

• commitSha - The short git commit hash (7 chars) of the deployed code
• commitMessage - The commit message of the deployed commit
• deploymentUrl - The Vercel deployment URL

Verification workflow:

1. After pushing, check the latest commit: git log --oneline -1
2. Wait for CI and Vercel deployment to complete
3. Fetch /api/health and compare commitSha with your local commit
4. If they match, the deployment is live

Note: Vercel provides these values via environment variables (VERCEL_GIT_COMMIT_SHA, VERCEL_GIT_COMMIT_MESSAGE, VERCEL_URL) which are automatically available at runtime.

---

Health Check Side Effects System

The health check system tests every registered tool by importing it and executing it with test parameters. For tools that create real resources on external platforms (e.g., createService, createSession), this creates orphaned resources that are never cleaned up.

Problem

The health check service (apps/web/src/lib/health-check/health-check-service.ts) runs two checks per tool:

1. Import check (/load-and-describe): Verifies the tool can be loaded and has valid description and inputSchema.
2. Execution check (/execute-tool): Executes the tool with auto-generated test parameters via the Railway executor.

The Railway executor (apps/railway-executor/server.ts) injects real API keys from the package's env vars stored in the database before calling tool.execute(). There is no dry-run mode. This means tools like createService actually create services on the Unsandbox platform.

Solution: healthCheck Config in tpmjs Spec

Tool authors can declare per-tool health check behavior via the healthCheck field in package.json:

{
  "tpmjs": {
    "tools": [
      {
        "name": "createService",
        "healthCheck": {
          "testParams": { "name": "tpmjs-hc-{{timestamp}}" },
          "cleanup": [
            { "tool": "deleteService", "mapping": { "service_id": "service_id" } }
          ]
        }
      },
      {
        "name": "getSession",
        "healthCheck": { "skipExecution": true }
      },
      {
        "name": "listJobs"
      }
    ]
  }
}

Config Options

Field	Type	Description
skipExecution	boolean	Skip execution check entirely. Only verify import. Use for tools that require existing external resources (would 404 with fake IDs).
testParams	Record<string, unknown>	Override auto-generated test parameters with known-good values. String values support {{timestamp}} template variable (replaced with Date.now()).
cleanup	Array<{ tool, mapping }>	Ordered cleanup steps to run after execution. Each step calls a tool from the same package with params mapped from the execution result.

Architecture

Schema: ToolHealthCheckConfigSchema and ToolHealthCheckCleanupStepSchema in packages/types/src/tpmjs.ts.

Database: healthCheckConfig Json? @db.JsonB column on the Tool model in packages/db/prisma/schema.prisma.

Sync: The 3 sync routes (/api/sync/changes, /api/sync/keyword, /api/sync/package) store toolDef.healthCheck as healthCheckConfig during tool upsert.

Execution: checkExecutionHealth() in the health check service reads the config from the DB and:

1. If skipExecution: true → returns HEALTHY immediately, no execution
2. If testParams → uses them (after {{timestamp}} template processing) instead of auto-generated params
3. After successful execution, if cleanup defined → runs each cleanup step in order via the executor
4. Default behavior unchanged for tools without config

Tool Classification Guide

When adding healthCheck config to a new package:

• Read-only/list tools (e.g., listJobs, getLanguages): No config needed. Safe by default.
• Sync execution tools (e.g., execute, run): Add testParams with valid input. No cleanup needed since execution is ephemeral.
• Async execution tools (e.g., executeCodeAsync): Add testParams + cleanup to delete the created job.
• Resource creation tools (e.g., createSession, createService): Add testParams + cleanup to delete the created resource.
• Operations on existing resources (e.g., getSession, freezeService, deleteSnapshot): Add skipExecution: true. These would 404 with fake IDs.

Files Involved

File	Role
packages/types/src/tpmjs.ts	Zod schemas for ToolHealthCheckConfigSchema
packages/db/prisma/schema.prisma	healthCheckConfig column on Tool model
apps/web/src/app/api/sync/changes/route.ts	Stores config during sync
apps/web/src/app/api/sync/keyword/route.ts	Stores config during sync
apps/web/src/app/api/sync/package/route.ts	Stores config during sync
apps/web/src/lib/health-check/health-check-service.ts	Reads config, skip/cleanup logic
packages/tools/official/unsandbox/package.json	Reference implementation with all tools configured

---

Environment Setup

Prerequisites

• Node.js >= 22 (project enforces this in package.json engines field)
• pnpm package manager (install via corepack enable or npm install -g pnpm)

First-Time Setup

# Clone and install
git clone https://github.com/tpmjs/tpmjs.git
cd tpmjs
pnpm install

# Generate Prisma client (required before type-checking)
pnpm --filter=@tpmjs/db db:generate

# Build all packages (needed for cross-package imports)
pnpm build

# Verify everything works
pnpm type-check
pnpm lint

Common Issues

pnpm not found: Run corepack enable to enable pnpm via Node's corepack. Alternatively, use npx -y pnpm as a fallback.

Node version mismatch: The project requires Node >= 22. If you see engine warnings, upgrade Node via nvm: nvm install 22 && nvm use 22.

Prisma client not generated: After schema changes or fresh installs, always run pnpm --filter=@tpmjs/db db:generate. The postinstall hook does this automatically during pnpm install, but manual runs may be needed after schema modifications.

Database migrations: After adding columns to schema.prisma, create and apply a migration: pnpm --filter=@tpmjs/db db:migrate (creates SQL migration file and applies it). For development, pnpm --filter=@tpmjs/db db:push pushes schema changes without creating a migration file.

Note: Vercel provides these values via environment variables (VERCEL_GIT_COMMIT_SHA, VERCEL_GIT_COMMIT_MESSAGE, VERCEL_URL) which are automatically available at runtime.