docs: add comprehensive NPM package sync documentation and GitHub Actions workflow

**Documentation (CLAUDE.md):**
- Document all three sync endpoints (changes feed, keyword search, metrics)
- Explain Vercel Cron configuration and schedules
- Detail quality score calculation algorithm
- Add database schema documentation for sync tables
- Provide manual sync trigger examples with curl
- Include monitoring and debugging instructions
- Document error handling patterns (partial/complete failures)
- Add package discovery flow diagram
- List future improvements and potential enhancements

**GitHub Actions Workflow (.github/workflows/sync.yml):**
- Add backup sync automation via GitHub Actions cron
- Support manual trigger via workflow_dispatch with sync type selection
- Run changes feed every 2 minutes
- Run keyword search every 15 minutes
- Run metrics sync every hour
- Use concurrency control to prevent overlapping runs
- Call production Vercel endpoints with CRON_SECRET auth

**Key Features:**
- Dual automation strategy: Vercel Cron (primary) + GitHub Actions (backup)
- Idempotent endpoints allow both systems to run simultaneously
- Manual trigger capability for debugging and testing
- Comprehensive documentation for future maintenance

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Ajax Davis 2025-11-30 20:46:57 +10:00
parent 4876b2e4f4
commit 408102602c
2 changed files with 491 additions and 1 deletions

71
.github/workflows/sync.yml vendored Normal file
View file

@ -0,0 +1,71 @@
name: NPM Package Sync
on:
schedule:
# Changes feed: Every 2 minutes
- cron: '*/2 * * * *'
# Keyword search: Every 15 minutes
- cron: '*/15 * * * *'
# Metrics: Every hour
- cron: '0 * * * *'
workflow_dispatch:
inputs:
sync_type:
description: 'Type of sync to run'
required: true
type: choice
options:
- changes
- keyword
- metrics
- all
concurrency:
group: ${{ github.workflow }}-${{ github.event.inputs.sync_type || 'scheduled' }}
cancel-in-progress: false
jobs:
sync-changes:
runs-on: ubuntu-latest
# Run on schedule every 2 minutes OR manual trigger with 'changes' or 'all'
if: |
github.event.schedule == '*/2 * * * *' ||
github.event.inputs.sync_type == 'changes' ||
github.event.inputs.sync_type == 'all'
steps:
- name: Trigger changes feed sync
run: |
curl -X POST "${{ secrets.VERCEL_PRODUCTION_URL }}/api/sync/changes" \
-H "Authorization: Bearer ${{ secrets.CRON_SECRET }}" \
-H "Content-Type: application/json" \
-f -s -S -w "\nHTTP Status: %{http_code}\n"
sync-keyword:
runs-on: ubuntu-latest
# Run on schedule every 15 minutes OR manual trigger with 'keyword' or 'all'
if: |
github.event.schedule == '*/15 * * * *' ||
github.event.inputs.sync_type == 'keyword' ||
github.event.inputs.sync_type == 'all'
steps:
- name: Trigger keyword search sync
run: |
curl -X POST "${{ secrets.VERCEL_PRODUCTION_URL }}/api/sync/keyword" \
-H "Authorization: Bearer ${{ secrets.CRON_SECRET }}" \
-H "Content-Type: application/json" \
-f -s -S -w "\nHTTP Status: %{http_code}\n"
sync-metrics:
runs-on: ubuntu-latest
# Run on schedule every hour OR manual trigger with 'metrics' or 'all'
if: |
github.event.schedule == '0 * * * *' ||
github.event.inputs.sync_type == 'metrics' ||
github.event.inputs.sync_type == 'all'
steps:
- name: Trigger metrics sync
run: |
curl -X POST "${{ secrets.VERCEL_PRODUCTION_URL }}/api/sync/metrics" \
-H "Authorization: Bearer ${{ secrets.CRON_SECRET }}" \
-H "Content-Type: application/json" \
-f -s -S -w "\nHTTP Status: %{http_code}\n"