Commit graph

59 commits

Author SHA1 Message Date
Ajax Davis
ac6fdb87e7 feat(web): add tweet button and OG tags to collection page
- Add ShareButton component for Twitter/X sharing
- Refactor collection page to server component with generateMetadata
- Add proper OpenGraph and Twitter Card meta tags for social sharing
- Extract client-side logic to CollectionDetailClient component
2026-01-19 18:16:17 +10:00
Ajax Davis
ea742c9386 feat: add scenarios system for collection testing
Implements the complete scenarios feature for TPMJS:

API endpoints:
- GET/POST /api/scenarios - list/create scenarios
- GET/PATCH/DELETE /api/scenarios/:id - scenario CRUD
- POST /api/scenarios/:id/run - execute scenario
- GET /api/scenarios/:id/runs - run history
- POST /api/scenarios/check-similarity - vector similarity check
- GET /api/scenarios/featured - featured scenarios
- POST /api/collections/:id/scenarios/generate - AI scenario generation
- GET /api/collections/:id/scenarios - list collection scenarios

Services:
- generate-prompt.ts - AI prompt generation using GPT-4o-mini
- similarity.ts - vector embedding and cosine similarity
- evaluate.ts - LLM-based success evaluation
- execute.ts - scenario execution orchestration

CLI commands:
- tpm scenario list [collection] - list scenarios
- tpm scenario run <collection> - run all scenarios
- tpm scenario test <id> - run single scenario
- tpm scenario generate <collection> - generate scenarios
- tpm scenario info <id> - scenario details

Database:
- Scenario, ScenarioEmbedding, ScenarioRun, ScenarioQuota models
- Streak-based quality scoring
- Daily quota management

Migration script included for converting existing useCases.
2026-01-18 09:31:31 +10:00
Ajax Davis
7e401c82eb chore: update lockfile for judge package 2026-01-17 11:21:58 +10:00
Ajax Davis
0fa4374d9e feat(tools): add @tpmjs/tools-hllm package
Add HLLM API client tools for AI agents with 36 tools:
- Topology execution (executeTopology)
- Chat sessions (list, create, get, update, delete)
- Session messages (add, clear)
- Prompt library (CRUD + usage tracking)
- User profile and stats
- Environment variables
- File management
- Models listing
- Execution logs and agent metrics
- TPMJS tools (list, describe, execute)
- Prompt generation and improvement
- Data export/import
- Health check and public stats
- API keys management

Published as @tpmjs/tools-hllm@0.1.0
2026-01-17 08:35:27 +10:00
Ajax Davis
0489f5cfdb feat(web): implement SWR for instant data fetching
- Add SWR package and global SWRProvider
- Create reusable hooks: useTools, useAgents, useCollections,
  useStats, useLikeStatus, useBundleSize, useActivity
- Update tool-search page to use useTools hook
- Update agents page to use useAgents hook
- Update BundleSize component to use useBundleSize hook
- Update LikeButton with optimistic updates via useLikeStatus
- Add simple about page with creator info
2026-01-16 11:42:55 +10:00
Ajax Davis
c21ed5b41e feat(exe-dev): add exe.dev VM management tools v0.2.3
Add 15 MCP tools for managing exe.dev virtual machines:
- list, create, deleteVm, restart - VM lifecycle management
- exec - execute commands on VMs via SSH
- shareShow, shareSetPublic, shareSetPrivate - visibility control
- sharePort - configure HTTP proxy port
- shareAddUser, shareRemoveUser - user access management
- shareAddLink, shareRemoveLink - shareable link management
- whoami - user account info
- shelleyInstall - install Shelley agent

Key fixes in v0.2.3:
- Properly quote SSH args to prevent local shell interpretation
- Add -- separator for commands to prevent flag parsing issues
- Support base64-encoded SSH key via EXE_DEV_SSH_KEY env var
2026-01-15 09:49:08 +10:00
Ajax Davis
4461bf70c8 feat(unsandbox): add code execution tools for unsandbox API
- Add 4 new tools: executeCodeAsync, getJob, execute, run
- Support 42+ programming languages
- Features: network isolation modes, input files, compiled artifacts, WASM
- Add entity definitions for code execution results
- All tools pass schema and shape validation
2026-01-15 06:38:45 +10:00
Ajax Davis
1c903cd26e feat(chat): use Streamdown for markdown rendering in chat messages
- Add streamdown package for AI-optimized markdown streaming
- Update dashboard and public agent chat pages to render messages with Streamdown
- Enables proper markdown formatting (code blocks, lists, etc.) in chat responses
2026-01-15 05:28:38 +10:00
Ajax Davis
27e7d45646 feat(sprites): add sprites-url-get and sprites-url-set tools
New tools for managing sprite URL access settings:
- sprites-url-get: Get the public URL and auth settings for a sprite
- sprites-url-set: Set URL auth to 'public' or 'sprite' (private)

Both published as @tpmjs/tools-sprites-url-get and @tpmjs/tools-sprites-url-set v0.1.0
2026-01-15 04:56:52 +10:00
Ajax Davis
2cd2b10cd0 refactor: replace exportName with name throughout codebase
- Update TpmjsToolDefinitionSchema to only use 'name' field
- Add 'sandbox' as valid category for sprites tools
- Update all package.json files to use 'name' instead of 'exportName'
- Update documentation and source files accordingly
- Add 11 new sprites tools for sandbox/code-execution
2026-01-14 14:18:36 +10:00
Ajax Davis
c8c1a12f22 feat: complete MCP Bridge implementation
- Add @tpmjs/mcp-client package for connecting to MCP servers
- Add @tpmjs/bridge CLI for bridging local MCP servers to TPMJS
- Add @tpmjs/test-file-writer test MCP server
- Add BridgeConnection and CollectionBridgeTool database models
- Add /api/bridge endpoints for bridge communication
- Add /api/collections/[id]/bridge-tools API for managing bridge tools
- Update MCP handlers to include bridge tools in tools/list
- Add bridge status UI at /dashboard/settings/bridge
- Add interactive bridge tutorial at /docs/tutorials/bridge
2026-01-13 00:59:34 +10:00
Ajax Davis
938a382aab feat: add environment variables support for agents and collections
- Add envVars field to Agent and Collection models in Prisma schema
- Add envVars to UpdateAgentSchema and UpdateCollectionSchema types
- Implement env vars merging logic (agent overrides collection)
- Pass env vars through tool executor to sandbox
- Add UI for managing env vars in agent and collection settings pages
- Update API routes to handle envVars PATCH updates
- Refactor discord tool execute functions for biome compatibility

This allows users to configure tool-specific environment variables (like
DISCORD_BOT_TOKEN) at both the collection and agent level, with agent
settings taking precedence over collection settings.
2026-01-12 01:35:29 +10:00
Ajax Davis
5c9f1a1d0a chore: update dependencies with compatibility fixes
- Update all packages to latest versions via pnpm update --latest
- Downgrade Prisma 7 to 6 (v7 requires schema migration)
- Downgrade Tailwind CSS 4 to 3 (v4 requires PostCSS migration)
- Downgrade Storybook 10 to 8 (addons not available in v10)
- Pin cheerio to 1.0.0-rc.12 via pnpm override (type exports changed)
- Fix AI SDK tool definitions: parameters -> inputSchema
- Fix cheerio types in extract-meta and table-extract tools
- Add explicit type annotations to tool execute functions
- Migrate biome config to v2.3.11 schema

All type-checks, tests, and builds pass.
2026-01-09 22:49:41 +10:00
Ajax Davis
361a49f3d7 feat: redesign public pages with virtualized tables
- Add sonner for toast notifications
- Create CopyButton component for simple copy actions
- Create CopyDropdown component with entity-specific copy options
- Create PackageManagerSelector with localStorage persistence
- Redesign tools page with TableVirtuoso, sort by downloads/likes/recent/name
- Redesign collections page with TableVirtuoso and infinite scroll
- Redesign agents page with TableVirtuoso and infinite scroll
- All tables have fast client-side filtering and copy functionality
2026-01-07 21:55:13 +10:00
Ajax Davis
552f319583 feat: add AI Agents feature with multi-provider support and documentation
- Add Agent, AgentCollection, AgentTool, UserApiKey, Conversation, Message models to Prisma schema
- Create agent types and Zod schemas in @tpmjs/types
- Implement AES-256 API key encryption utilities
- Add CRUD API endpoints for agents, tools, collections, and user API keys
- Create conversation streaming endpoint with SSE events
- Build agent tool builder to merge collections and individual tools
- Add dashboard pages: agents list, new agent form, agent detail/edit, chat interface
- Add API keys settings page for managing provider keys
- Add comprehensive Agents documentation section to /docs
- Update navigation to include Agents link in header and mobile menu
- Add new icons: terminal, puzzle, message, key, info, send

Supported providers: OpenAI, Anthropic, Google, Groq, Mistral
2026-01-02 20:08:52 +10:00
Ajax Davis
828801f463 feat: add MCP endpoint for collections
Add Model Context Protocol (MCP) server endpoint for collections,
allowing users to connect Claude Desktop, Cursor, and other MCP
clients to use tools from their collections.

- POST /api/collections/[id]/mcp - JSON-RPC endpoint for MCP
- GET /api/collections/[id]/mcp - Server info for discovery
- Tool name conversion (e.g., @tpmjs/hello → tpmjs-hello--helloWorldTool)
- Handlers for initialize, tools/list, tools/call methods
- Executes tools via existing sandbox executor

Only public collections are accessible (no auth required).
2026-01-02 04:25:33 +10:00
Ajax Davis
522c198041 fix: add Prisma client to web app for Vercel monorepo compatibility
- Add @prisma/client and prisma as direct dependencies
- Add postinstall and build scripts to generate Prisma client
- Points to schema in packages/db
2026-01-02 00:37:00 +10:00
Ajax Davis
6060492169 feat: add Better Auth with email/password authentication
- Add User, Session, Account, Verification models to Prisma schema
- Create auth configuration with Prisma adapter
- Add Resend email integration for verification emails
- Create sign-in, sign-up, and verify-email pages
- Create user dashboard with profile display
- Add middleware for /dashboard route protection
- Add better-auth and resend dependencies

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-01 20:45:10 +10:00
Ajax Davis
d00ab8afbb feat: add /tool-ideas page with 10K AI-generated tools
- Add virtualized list using react-virtuoso for smooth scrolling
- Filter by category, verb, quality score, and search
- Expandable cards with parameters, returns, AI guidance
- Add redirect from /tools-ideas to /tool-ideas

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-01 17:13:46 +10:00
Ajax Davis
5d2096fb5d feat: add 100+ official TPMJS tools
Implements a comprehensive suite of AI SDK v6 tools across multiple categories:

- Research (5): page-brief, compare-pages, source-credibility, claim-checklist, timeline-from-text
- Web (10): fetch-text, links-catalog, extract-meta, extract-json-ld, redirect-trace, sitemap-read, rss-read, table-extract, robots-policy, url-normalize
- Data (15): csv-parse, csv-stringify, json-repair, json-schema-validate, yaml-parse, yaml-stringify, text-chunk, normalize-whitespace, dedupe-by-key, pivot, rows-filter, rows-sort, rows-group-aggregate, rows-join, schema-infer
- Doc (12): toc-generate, glossary-build, faq-from-text, executive-brief, decision-record-adr, prd-outline, acceptance-criteria, style-rewrite
- Eng (12): diff-text-unified, env-var-docs-generate, dependency-audit-lite, conventional-commit-suggest, markdown-lint-basic, test-case-generate, stacktrace-parse, release-notes, changelog-entry, release-checklist
- Security (7): redact-secrets, secret-scan-text, url-risk-heuristic, csp-compose, hardening-checklist-web, access-control-matrix, data-classification-heuristic
- Stats (9): effect-size-suite, bootstrap-ci, permutation-test, multiple-testing-adjust, linear-regression-ols, logistic-regression, time-series-decompose-lite, anomaly-detect-mad
- Ops (7): slo-draft, runbook-draft, postmortem-draft, postmortem-action-extractor, error-log-triage, coverage-tracker, monitoring-gap-analysis
- Agent (15): prompt-to-workflow-skeleton, workflow-validate-io, workflow-explain, workflow-cost-estimate, tool-call-accuracy-score, eval-fixture-build, guardrail-policy-draft, workflow-auto-repair, tool-selection-plan, novelty-score-workflow, workflow-variant-generate, config-normalize, recipe-*
- Utility (8): base64-encode, base64-decode, hash-text, regex-extract, template-render, date-parse, json-path-query, url-parse
- HTML (3): html-sanitize, html-to-markdown, markdown-to-html

All tools follow AI SDK v6 pattern with tool() and jsonSchema<T>().

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-31 22:55:56 +10:00
Ajax Davis
16004298b2 feat: fully implement 5 research tools with production-ready functionality
Research tools now have complete implementations:

- **page-brief**: Uses @mozilla/readability + jsdom for content extraction,
  sbd for sentence parsing, comprehensive error handling with timeout and
  network error detection

- **compare-pages**: Uses natural library TF-IDF for text similarity,
  detects agreements via high-similarity sentences, identifies conflicts
  using negation pattern analysis

- **source-credibility**: Uses tldts for domain parsing, cheerio for HTML
  analysis, calculates 6 credibility signals (HTTPS, domain reputation,
  author, date, citations, contact info), weighted scoring system

- **claim-checklist**: Uses sbd for sentence boundary detection, regex
  patterns for claim identification (statistics, quotes, historical,
  scientific, factual), priority levels and evidence suggestions

- **timeline-from-text**: Uses chrono-node for date parsing, sbd for
  context extraction, calculates confidence scores, identifies gaps >30 days

All tools follow AI SDK v6 pattern (tool() + jsonSchema()), include proper
TypeScript types, input validation, comprehensive error handling, and
Node.js 18+ fetch requirement verification.

Updated blocks.yml with comprehensive philosophy and domain rules for
quality validation.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-31 21:04:07 +10:00
Ajax Davis
36a0735ab1 feat: add 5 research tools and blocks framework setup
- Add @tpmjs/tools-page-brief for URL content extraction
- Add @tpmjs/tools-compare-pages for cross-source validation
- Add @tpmjs/tools-source-credibility for credibility scoring
- Add @tpmjs/tools-claim-checklist for factual claim extraction
- Add @tpmjs/tools-timeline-from-text for timeline generation
- Move createBlogPost to packages/tools/official/
- Add blocks.yml for Blocks framework validation
- Update pnpm-workspace.yaml to include official tools
2025-12-31 19:47:02 +10:00
Ajax Davis
fd4520dbf6 perf: reduce Neon compute usage with cron + caching optimizations
- Reduce cron frequency: changes 2min→4hr, keyword 15min→6hr, metrics hourly→daily
- Add Prisma directUrl for connection pooling support
- Add Vercel KV caching to /api/tools endpoint (graceful degradation if not configured)
- Add X-Cache header to indicate cache hit/miss
- Add NEON_COMPUTE_OPTIMIZATION.md with full strategy guide

These changes should reduce Neon CU usage from 100+ to ~20-30 CU-hrs/month.
2025-12-31 12:14:00 +10:00
Ajax Davis
29e1b9d9ac feat: add interactive tutorial slideshow app
Create apps/tutorial - a beautiful, animated Next.js slideshow that explains
TPMJS concepts from basic to advanced:

Slides:
- Welcome: Gradient TPMJS title with animated floating orbs
- Problem: Chaotic floating icons showing fragmented AI tools
- Solution: Icons organizing into a neat grid
- How It Works: Flow diagram (npm → Discovery → Registry → AI Agent)
- Discovery: Animated counters showing real-time sync stats
- Integration: Typewriter code animation showing SDK usage
- Quality Scoring: Animated gauge and progress bars
- Tool Detail: Mock tool card with all metadata
- Get Started: CTA buttons linking to tpmjs.com

Features:
- Framer Motion animations with spring physics
- Keyboard navigation (arrows, space, enter)
- Mouse navigation (arrows, progress dots)
- Touch-friendly swipe support
- Dark theme with cyan/purple gradient accents
- Responsive design
2025-12-28 23:41:15 +10:00
Ajax Davis
1efb2623b3 feat: add build-time OG image generation with OpenAI
- Create generate-og-images.ts script for build-time image generation
- Generate 13 static page OG images using gpt-image-1-mini
- Update API route to serve pre-generated images from public/og/
- Add 30-day cache headers and fallback to default image
- Images regenerate if older than 30 days

Run with: pnpm --filter=@tpmjs/web generate-og

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-28 20:45:02 +10:00
Ajax Davis
effd242b07 feat: add AI-generated OG images with OpenAI and Vercel Blob caching
- Add /api/og/[...path] endpoint for dynamic OG image generation
- Use OpenAI gpt-image-1 for image generation with page-specific prompts
- Cache images in Vercel Blob storage with 30-day TTL
- Extract page content for contextual prompts (static pages, tool details)
- Update all page metadata to use dynamic OG image URLs
- Refactor tool detail page to server component for generateMetadata support
- Fall back to static /public/og-image.png on generation errors
2025-12-28 18:21:11 +10:00
Ajax Davis
c189bd366a feat: add comprehensive stats dashboard with D3 charts
- Add /stats page with animated D3 visualizations
- Create reusable chart components: AnimatedCounter, DonutChart, BarChart, AreaChart
- Expand stats API endpoints: /api/stats, /api/stats/health, /api/stats/executions, /api/stats/sync, /api/stats/tools
- Add Stats link to desktop and mobile navigation
- Add AI SDK v6 integration tests with vitest

Dashboard displays:
- Registry overview metrics with count-up animations
- Health distribution donut charts (import/execution)
- Quality score distribution
- Package tier breakdown
- Execution trends area chart with success/error series
- Token usage statistics
- Top categories bar chart
- Recent sync operations status
2025-12-28 17:08:19 +10:00
Ajax Davis
9855425be4 chore: upgrade AI SDK from beta to stable v6
- ai: 6.0.0-beta.124 → 6.0.3
- @ai-sdk/openai: 3.0.0-beta.74 → 3.0.1
- @ai-sdk/react: 3.0.0-beta.131 → 3.0.3

Breaking changes addressed:
- CoreMessage renamed to ModelMessage
- convertToModelMessages is now async (added await)
2025-12-28 16:17:41 +10:00
Ajax Davis
0e5d4d6e3b feat: add Vercel Analytics to web and playground apps
- Install @vercel/analytics in both apps
- Add <Analytics /> component to root layouts
- Enables automatic page view tracking on Vercel
2025-12-17 11:04:02 +10:00
Ajax Davis
3bb8033c69 chore: update dependencies and add unsandbox package 2025-12-16 12:00:22 +10:00
Ajax Davis
5c935dca69 fix(createblogpost): convert to proper AI SDK tool
- Wrap function with tool() from 'ai' package
- Add jsonSchema() for input validation
- Rename export from createBlogPost to createBlogPostTool
- Update package.json with proper tpmjs.tools format
- Bump version to 0.3.0

Fixes executor error: 'Cannot destructure property title of t'
2025-12-12 10:38:46 +10:00
Ajax Davis
ed53b68c75 feat(sdk): replace ASCII diagram with interactive D3 visualization
- Create SDKFlowDiagram component with animated D3 graphics
- Add flowing particle animations along connection paths
- Add hover interactions with tooltips for each node
- Add entrance animations with staggered timing
- Add subtle glow effects and shadows
- Responsive design that adapts to screen width
- Sleek minimal black and white aesthetic with depth effects
2025-12-12 09:43:40 +10:00
Ajax Davis
ee825aa97c feat: add @tpmjs/registrySearch and @tpmjs/registryExecute SDK packages
- Create @tpmjs/registrySearch package for searching tool registry
- Create @tpmjs/registryExecute package for executing tools via sandbox
- Support self-hosted registries via TPMJS_API_URL and TPMJS_EXECUTOR_URL env vars
- Add /sdk documentation page with usage examples and architecture
- Add SDK link to navigation menu
- Include design doc for registry SDK architecture

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-12 07:29:06 +10:00
Ajax Davis
1469b0a060 fix(security): update Next.js to 16.0.8 to address CVE-2025-66478 2025-12-11 11:14:43 +10:00
Ajax Davis
9ff4cd6d05 feat: add markdown-formatter package with defensive patterns
- Create @tpmjs/markdown-formatter with 2 tools:
  - markdownToPlainText: Convert markdown to plain text
  - formatMarkdownTable: Format and align markdown tables
- Includes defensive parameter validation
- Uses AI SDK v6 beta with Zod 4 schemas
- Published v0.2.0 to npm

Testing the full end-to-end workflow:
- Package creation following generator patterns
- Changesets for version management
- npm publishing
- TPMJS registry auto-discovery
2025-12-05 01:31:03 +10:00
Ajax Davis
e6c1be6a53 chore: release emoji-magic v0.2.0 and create-basic-tools v1.0.4 2025-12-05 00:10:18 +10:00
Ajax Davis
38c85bd904 feat: add @tpmjs/create-basic-tools CLI generator
- Interactive CLI generator for scaffolding TPMJS tool packages
- Generates packages with minimum 2 tools (ideally 2-3)
- Zod 4 schemas - uses Zod directly (not jsonSchema wrapper)
- One file per tool in src/tools/<toolName>.ts
- TPMJS validated against official schemas from @tpmjs/types
- Complete package generation ready to publish to npm
- Works both standalone and in monorepo packages/ folders

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-04 23:34:37 +10:00
Ajax Davis
a9e91c02d1 feat: add reusable ToolHealthBadge and ToolHealthBanner components
- Create ToolHealthBadge component in @tpmjs/ui for broken tool indicator
- Create ToolHealthBanner component in @tpmjs/ui for detailed health warnings
- Integrate both components into playground ToolsSidebar:
  - Badge shows in tool cards in left sidebar
  - Banner shows in tool detail modal
- Update search-registry to include health fields in API responses
- Add package.json exports for new health components

These components provide consistent UI for displaying broken tool status
across the application (tool search, tool detail pages, playground).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-04 19:12:07 +10:00
Ajax Davis
f9dfc67de4 feat: add automated Vercel AI registry sync with OpenAI
Add hourly GitHub Action that syncs tools from Vercel's AI SDK registry:

Features:
- Fetches Vercel AI registry from their GitHub
- Uses OpenAI GPT-4 to intelligently convert tool metadata
- Handles multi-export packages (multiple tools per npm package)
- Automatically commits new tools to manual-tools.ts
- Sends Discord notifications with detailed stats
- Extensive logging at every step

Files added:
- sync-vercel-registry.ts - Main sync script with AI conversion
- .github/workflows/sync-vercel-registry.yml - Hourly GitHub Action
- docs/vercel-registry-sync.md - Complete documentation

Requires OPENAI_API_KEY secret in GitHub repository settings.
2025-12-04 17:43:17 +10:00
Ajax Davis
3aef5bb2bb fix: migrate playground to AI SDK v6 beta and resolve TypeScript errors
- Update @ai-sdk/react to v3.0.0-beta.131 for compatibility with AI SDK v6
- Fix tool execute method calls with type assertions in API routes
- Update chat components to use UIMessage types from AI SDK
- Move body option into DefaultChatTransport constructor
- Remove deprecated onResponse option from useChat hook
- Remove unused isCoreTool type guard function
- Fix Button variant from 'primary' to 'default'

Resolves all TypeScript compilation errors and build passes successfully.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-04 13:15:21 +10:00
Ajax Davis
d597a71eb4 feat: add dynamic tool loading system with Railway executor
Implements a complete dynamic tool loading system that allows the playground to discover and load tools from the TPMJS registry at runtime.

**Architecture:**
- Search tool package (@tpmjs/search-registry) - Searches registry for tools
- Search API endpoint (/api/tools/search) - Text-based search with scoring
- Pre-flight tool loading - Automatically searches and loads tools on every message
- Railway executor service (Deno) - Loads tools from esm.sh via HTTP imports
- Dynamic tool loader - Calls Railway to load and execute tools remotely

**Key Components:**

1. Railway Executor (apps/railway-executor/)
   - Deno-based service that natively supports HTTP imports
   - Endpoints: /load-and-describe, /execute-tool, /cache/stats, /cache/clear
   - Deploys to Railway with deno run --allow-net --allow-env server.ts

2. Search Tool Package (packages/tools/search-registry/)
   - AI SDK v6 tool for searching TPMJS registry
   - Uses jsonSchema + inputSchema pattern
   - Searches /api/tools/search endpoint

3. Search API (apps/web/src/app/api/tools/search/)
   - Text-based search with composite scoring
   - Scores: text relevance + quality boost + download boost
   - Returns tool metadata with importUrl for dynamic loading

4. Dynamic Tool Loader (apps/playground/src/lib/dynamic-tool-loader.ts)
   - Calls Railway service to load tools from esm.sh
   - Creates tool wrappers that execute remotely
   - Process-level module cache + per-conversation tracking

5. Pre-flight Loading (apps/playground/src/app/api/chat/route.ts)
   - Automatically searches for tools on every user message
   - Loads top 5 matching tools before agent processes request
   - Merges with static tools for seamless experience

**Technical Decisions:**
- Deno over Node.js: Native HTTP import support without flags
- Remote execution: Tools run in Railway sandbox, not Vercel
- Pre-flight loading: Better UX than two-turn search pattern
- Text search: BM25 had dependency issues, simple scoring works well

**Environment Variables:**
- RAILWAY_SERVICE_URL: https://endearing-commitment-production.up.railway.app

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-04 08:25:53 +10:00
Ajax Davis
141a64d888 feat: implement multi-tool package architecture with manual tool registry
BREAKING CHANGE: Complete refactoring from single-tool to multi-tool package support

Database Schema:
- Split Tool model into Package (1) and Tool (many) with one-to-many relationship
- Package stores npm metadata and package-level tpmjs fields (category, env, frameworks, tier)
- Tool stores individual tool exports with tool-level metadata (exportName, description, parameters, returns, aiAgent)
- Unique constraint on (packageId, exportName) to prevent duplicate tools
- Cascade deletes when packages are removed

Type System:
- Updated tpmjs field schema to support tools array
- Each tool has exportName, description, parameters, returns, aiAgent
- Package-level fields: category, env, frameworks shared across all tools
- Backward compatible with legacy single-tool format (auto-migrates to exportName: "default")

API Updates:
- Updated all /api/tools routes to query Tool model with Package relations
- Updated /api/tools/[slug] to accept package/export path segments
- Updated tool-executor-agent to use actual exportName instead of hardcoded "default"
- Updated metrics sync to calculate quality scores per Tool

Frontend Updates:
- Updated tool search page to display exportName as primary heading
- Updated tool detail pages to show package name as secondary info
- Removed tag-based filtering (tags moved to package level)

Manual Tool Registry:
- Added manual-tools.ts with 23 curated tools from major providers
- Created sync-manual-tools.ts script to sync manual tools to database
- Added MANUAL_TOOLS.md documentation for manual tool system
- Added GitHub workflow for automated daily sync
- Includes tools from: Vercel, Exa, Firecrawl, AWS Bedrock, Perplexity, Tavily, Superagent, Valyu

Playground Updates:
- Updated tool loader to load multiple tools per package
- Added sanitizeToolName for OpenAI API compatibility

Sync System Updates:
- Updated changes feed sync to handle multi-tool packages
- Updated keyword sync to upsert multiple tools per package
- Added orphaned tool deletion when tools removed from package.json

Migration Strategy:
- Database uses same Neon instance for dev and prod
- Schema updated via prisma db push (no migration files yet)
- All data repopulates from npm via sync system

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-04 06:39:58 +10:00
Ajax Davis
0658aea425 feat: implement playground app with AI SDK v6 tool execution
- Create new Next.js app at apps/playground for testing TPMJS tools
- Implement AI SDK v6 patterns with DefaultChatTransport and UIMessage format
- Create template tool package at packages/tools/hello with hello-world and hello-name tools
- Use tool() and jsonSchema() helpers to avoid Zod 4 conversion issues with OpenAI
- Add static tool loading system with switch statement (Next.js/webpack compatible)
- Implement chat interface with tool call visualization showing inputs/outputs
- Support multi-step tool execution with stepCountIs(5)
- Stream responses with toUIMessageStreamResponse() for full tool support
- Add sidebar showing available tools (static list)
- Use parts-based message rendering for text and tool calls
- Integrate firecrawl-aisdk tools (scrape, crawl, search)
- Add theme toggle in header (defaults to light mode)
- Fix responsive layout with max-width for message bubbles
- Use biome-ignore comments for legitimate any types in tool loading

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-04 02:51:02 +10:00
Ajax Davis
268112f96e fix: preserve streamed output and add markdown rendering to playground
**Problem:**
- Tool playground was only showing partial output
- When 'complete' event arrived, it replaced streamed text with final output
- Output was displayed as plain text instead of formatted markdown

**Changes:**
- Fixed streaming bug in ToolPlayground component
  - Removed line that overwrote accumulated text on 'complete' event
  - Now preserves all streamed chunks for full output display
- Added react-markdown with GitHub Flavored Markdown support
  - Install react-markdown and remark-gfm packages
  - Added @tailwindcss/typography plugin for prose styling
  - Replaced plain <pre> with <ReactMarkdown> component
  - Applied prose classes for proper markdown formatting
- Added type="button" to button elements for accessibility

**Files changed:**
- apps/web/src/components/ToolPlayground.tsx
  - Comment out setOutput(data.output) on complete event
  - Import ReactMarkdown and remarkGfm
  - Replace pre element with ReactMarkdown component
  - Add prose styling classes
  - Add type="button" to buttons
- apps/web/tailwind.config.ts
  - Add @tailwindcss/typography plugin

**Result:**
- Full streamed output now displays correctly
- Markdown is rendered with proper formatting (headings, lists, code blocks, etc.)
- Better UX for AI-generated tool responses

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 19:28:53 +10:00
Ajax Davis
a92c2c250c feat: upgrade to AI SDK v6 beta and Zod v4 to fix tool schema errors
OpenAI was rejecting tool definitions with error "schema must be a JSON Schema of 'type: "object"'". This was caused by AI SDK v5 not properly converting Zod schemas to JSON Schema format.

**Changes:**
- Upgrade AI SDK from v5.0.104 to v6.0.0-beta.124
- Upgrade @ai-sdk/openai from v2.0.74 to v3.0.0-beta.22
- Upgrade Zod from v3.25.76 to v4.1.13 across all packages

**AI SDK v6 breaking changes:**
- Tool definition API: `parameters` renamed to `inputSchema`
- Removed `aiTool()` wrapper - use plain object with description, inputSchema, execute
- Streaming API: Use `textStream` async iterator instead of onChunk callback
- Zod schemas now properly converted to JSON Schema for OpenAI

**Zod v4 breaking changes:**
- `z.record()` now requires two arguments: `z.record(keySchema, valueSchema)`
- `z.enum()` params changed: `errorMap` removed, use `message` instead
- Type system improvements require explicit type parameters
- Fixed type errors in @tpmjs/env, @tpmjs/npm-client, @tpmjs/types

**Files changed:**
- apps/web/src/lib/ai-agent/tool-executor-agent.ts
  - Updated tool definition to use `inputSchema` instead of `parameters`
  - Removed `aiTool()` wrapper
  - Fixed streaming to use `textStream` iterator
- packages/env/src/index.ts
  - Updated type constraint from `z.ZodRawShape` to `Record<string, z.ZodTypeAny>`
- packages/npm-client/src/package.ts
  - Fixed `z.record()` calls to include both key and value schemas
  - Added type assertions for record indexing
- packages/types/src/tpmjs.ts
  - Changed `errorMap` to `message` in z.enum() calls

**Testing:**
-  Type-check passes
-  Production build succeeds
-  All routes compile correctly

This fixes the tool execution error where OpenAI rejected tool schemas with invalid format.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 19:14:01 +10:00
Ajax Davis
f0511eb498 fix: sanitize npm package names for OpenAI tool names and update lockfile
Problem: OpenAI tool names must match ^[a-zA-Z0-9_-]+ but npm package names like @tpmjs/createblogpost contain @ and /

Solution: Add sanitizeToolName() function and update pnpm-lock.yaml

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 02:28:42 +10:00
Ajax Davis
14b64f0320 fix: remove VM2 sandboxing to resolve Next.js build errors
**VM2 Removal:**
- Remove VM2 dependency from package-executor
- Rewrite executor to use direct package execution with require()
- Add TODO comment for future sandboxing implementation

**Why this change:**
- VM2 requires runtime filesystem access to bridge.js which doesn't work with Next.js Turbopack bundling
- Even marking as serverExternalPackages fails because VM2 uses hardcoded file paths
- Direct execution allows builds to complete while we find Next.js-compatible sandboxing solution

**Next Steps:**
- Implement proper sandboxing with isolated-vm or similar Next.js-compatible solution
- Add security measures for package execution
- Consider moving package execution to separate microservice

This unblocks CI/CD while maintaining playground functionality.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 00:57:32 +10:00
Ajax Davis
ab46b6e116 feat: add interactive tool playground with AI-powered execution
Implement comprehensive tool testing environment with real package execution, AI agents, and token tracking.

**New Features:**
- Interactive playground UI with 4 tabs (Input, Output, Logs, Token Usage)
- Real npm package execution in VM2 sandbox with security constraints
- AI-powered tool execution using AI SDK v5 and GPT-4 Turbo
- Server-Sent Events (SSE) streaming for real-time progress updates
- Comprehensive 4-category token tracking (Input, Tool Description, Schema, Output)
- Visual token breakdown with colored progress bars
- IP-based rate limiting (10 executions/hour per IP)
- Database persistence of all simulations with full metadata

**Database Schema:**
- New `Simulation` model for execution records
- New `TokenUsage` model for detailed token metrics
- New `ExecutionLog` model for execution event tracking
- Added simulations relation to Tool model

**Package Executor (@tpmjs/package-executor):**
- VM2 sandbox with 5-second timeout
- Blocked dangerous modules (fs, net, http, https, child_process)
- LRU file system cache in /tmp/.tpmjs-cache
- Package installation and caching strategy

**AI Agent Service:**
- TPMJS parameter to Zod schema conversion
- AI SDK tool definition generation
- Token counting using tiktoken library
- GPT-4 Turbo pricing estimation
- Streaming text execution with callbacks

**API Endpoints:**
- POST /api/tools/[...slug]/execute - SSE streaming execution
- GET /api/tools/[...slug]/simulations - Execution history
- Rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining)

**Frontend Components:**
- ToolPlayground - Main playground UI with tabs
- TokenBreakdown - Visual token metrics with colored bars
- Integrated above README section on tool detail pages

**Security:**
- VM2 sandboxing prevents filesystem/network access
- Rate limiting prevents abuse
- IP tracking for usage monitoring
- Timeout protection (60s max API duration, 5s VM timeout)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-30 00:36:05 +10:00
Ajax Davis
6d967f3501 feat: add syntax highlighting and improve README readability
**Syntax Highlighting:**
- Add react-syntax-highlighter with Solarized Light theme
- Proper language detection from markdown code blocks
- Beautiful syntax highlighting for all code examples
- Improved inline code styling with subtle borders

**Enhanced Readability:**
- Larger base typography with prose-lg
- Better contrast for text colors (zinc-700/zinc-300)
- Improved heading spacing and hierarchy
- Enhanced table styling with hover effects and better spacing
- Table headers with uppercase, bold styling
- Table cells with generous padding (px-6 py-4/py-3)
- Row hover effects for better interaction
- Better blockquote styling with blue accents
- Improved list spacing with leading-relaxed
- Enhanced image borders and shadows

**Table Improvements:**
- Professional header styling with background colors
- Better cell padding and spacing
- Hover effects on rows
- Improved borders and shadows
- Responsive overflow handling

All code blocks now have beautiful Solarized Light syntax highlighting, and the overall typography is more readable and professional.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 23:21:40 +10:00
Ajax Davis
32499bbc68 feat: add README rendering and enhanced package metadata display
**Database Schema:**
- Add npmReadme, npmKeywords, npmAuthor, npmMaintainers fields to Tool model

**NPM Client:**
- Add fetchLatestPackageWithMetadata() function to fetch README and top-level metadata
- Export new PackageVersionWithReadme type

**Sync Workers:**
- Update keyword and changes sync to fetch and store README content
- Store author, maintainers, and keywords from package.json

**UI Components:**
- Create Markdown component using react-markdown with GitHub Flavored Markdown
- Add rehype-sanitize for security and remark-gfm for tables/strikethrough support

**Tool Detail Page:**
- Convert from createElement to JSX for better maintainability
- Display README in a dedicated card with proper markdown rendering
- Show NPM keywords, author, and maintainers in sidebar
- Add ThemeToggle to header
- Improve layout with better spacing and organization

This brings the tool detail pages much closer to NPM's package pages,
providing users with comprehensive information about each tool including
the full README documentation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-29 22:41:50 +10:00