feat: implement multi-tool package architecture with manual tool registry

BREAKING CHANGE: Complete refactoring from single-tool to multi-tool package support

Database Schema:
- Split Tool model into Package (1) and Tool (many) with one-to-many relationship
- Package stores npm metadata and package-level tpmjs fields (category, env, frameworks, tier)
- Tool stores individual tool exports with tool-level metadata (exportName, description, parameters, returns, aiAgent)
- Unique constraint on (packageId, exportName) to prevent duplicate tools
- Cascade deletes when packages are removed

Type System:
- Updated tpmjs field schema to support tools array
- Each tool has exportName, description, parameters, returns, aiAgent
- Package-level fields: category, env, frameworks shared across all tools
- Backward compatible with legacy single-tool format (auto-migrates to exportName: "default")

API Updates:
- Updated all /api/tools routes to query Tool model with Package relations
- Updated /api/tools/[slug] to accept package/export path segments
- Updated tool-executor-agent to use actual exportName instead of hardcoded "default"
- Updated metrics sync to calculate quality scores per Tool

Frontend Updates:
- Updated tool search page to display exportName as primary heading
- Updated tool detail pages to show package name as secondary info
- Removed tag-based filtering (tags moved to package level)

Manual Tool Registry:
- Added manual-tools.ts with 23 curated tools from major providers
- Created sync-manual-tools.ts script to sync manual tools to database
- Added MANUAL_TOOLS.md documentation for manual tool system
- Added GitHub workflow for automated daily sync
- Includes tools from: Vercel, Exa, Firecrawl, AWS Bedrock, Perplexity, Tavily, Superagent, Valyu

Playground Updates:
- Updated tool loader to load multiple tools per package
- Added sanitizeToolName for OpenAI API compatibility

Sync System Updates:
- Updated changes feed sync to handle multi-tool packages
- Updated keyword sync to upsert multiple tools per package
- Added orphaned tool deletion when tools removed from package.json

Migration Strategy:
- Database uses same Neon instance for dev and prod
- Schema updated via prisma db push (no migration files yet)
- All data repopulates from npm via sync system

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Ajax Davis 2025-12-04 06:39:58 +10:00
parent fdd1b2c304
commit 0612eac5e2
31 changed files with 3783 additions and 786 deletions

116
sync-hello.ts Normal file
View file

@ -0,0 +1,116 @@
import { PrismaClient } from '@prisma/client';
import { validateTpmjsField } from '@tpmjs/types/tpmjs';
const prisma = new PrismaClient();
async function syncHello() {
try {
console.log('Fetching @tpmjs/hello from npm...');
const response = await fetch('https://registry.npmjs.org/@tpmjs/hello');
const data = await response.json();
const latest = data['dist-tags'].latest;
const pkg = data.versions[latest];
console.log(`\nPackage: ${pkg.name}@${pkg.version}`);
console.log(`Keywords: ${pkg.keywords.join(', ')}`);
console.log(`\ntpmjs field:`, JSON.stringify(pkg.tpmjs, null, 2));
// Validate
const validation = validateTpmjsField(pkg.tpmjs);
if (!validation.valid) {
console.error('\n❌ Invalid tpmjs field:', validation.errors);
process.exit(1);
}
console.log(`\n✅ Valid tpmjs field (${validation.tier} tier)`);
console.log(`Tools to create: ${validation.tools?.length || 0}`);
// Upsert Package
const packageRecord = await prisma.package.upsert({
where: { npmPackageName: pkg.name },
create: {
npmPackageName: pkg.name,
npmVersion: pkg.version,
npmPublishedAt: new Date(pkg.time[pkg.version]),
npmDownloadsLastMonth: 0,
category: validation.packageData!.category,
env: validation.packageData!.env || null,
frameworks: validation.packageData!.frameworks || [],
tier: validation.tier!,
discoveryMethod: 'manual',
isOfficial: pkg.name.startsWith('@tpmjs/'),
},
update: {
npmVersion: pkg.version,
npmPublishedAt: new Date(pkg.time[pkg.version]),
category: validation.packageData!.category,
env: validation.packageData!.env || null,
frameworks: validation.packageData!.frameworks || [],
tier: validation.tier!,
},
});
console.log(`\n✅ Package upserted: ${packageRecord.id}`);
// Get existing tools
const existingTools = await prisma.tool.findMany({
where: { packageId: packageRecord.id },
});
console.log(`\nExisting tools: ${existingTools.length}`);
// Upsert each tool
for (const toolDef of validation.tools || []) {
const tool = await prisma.tool.upsert({
where: {
packageId_exportName: {
packageId: packageRecord.id,
exportName: toolDef.exportName,
},
},
create: {
packageId: packageRecord.id,
exportName: toolDef.exportName,
description: toolDef.description,
parameters: toolDef.parameters ? (toolDef.parameters as any) : undefined,
returns: toolDef.returns ? (toolDef.returns as any) : undefined,
aiAgent: toolDef.aiAgent ? (toolDef.aiAgent as any) : undefined,
qualityScore: null,
},
update: {
description: toolDef.description,
parameters: toolDef.parameters ? (toolDef.parameters as any) : undefined,
returns: toolDef.returns ? (toolDef.returns as any) : undefined,
aiAgent: toolDef.aiAgent ? (toolDef.aiAgent as any) : undefined,
},
});
console.log(`✅ Tool upserted: ${tool.exportName} (${tool.id})`);
}
// Delete orphaned tools
const orphanedTools = existingTools.filter(
(existingTool) =>
!validation.tools?.some((toolDef) => toolDef.exportName === existingTool.exportName)
);
if (orphanedTools.length > 0) {
await prisma.tool.deleteMany({
where: { id: { in: orphanedTools.map((t) => t.id) } },
});
console.log(`\n🗑 Deleted ${orphanedTools.length} orphaned tools`);
}
console.log('\n✅ Sync complete!');
} catch (error) {
console.error('\n❌ Sync failed:', error);
process.exit(1);
} finally {
await prisma.$disconnect();
}
}
syncHello();