pig.py/docs/REFACTOR.md

Neopig Module Refactoring Plan

Overview

This document outlines the restructuring of serp.py (3291 lines) and neopig.py (2935 lines) into proper Python packages for improved maintainability, testability, and code organization.

Current State Analysis

serp.py Breakdown (3291 lines)

Section	Lines	Content
TRANSLATIONS dict	1-1111	~1100 lines of i18n strings (26 languages)
i18n helpers	1113-1194	get_lang(), t(), inject_i18n(), NAV_HTML, SEARCH_BOX_HTML
App setup	1195-1233	FastAPI app, Jinja2, static mounts, config globals
Job logging	1234-1277	start_job_logging(), stop_job_logging(), get_job_logs()
Tarball mode	1278-1333	_open_tarball(), read_from_tarball(), find_media_in_tarball()
ArchiveDB	1334-1353	SQLite wrapper for archive mode
Pydantic models	1411-1424	CrawlRequest model
Page routes	1425-2099	/, /crawl, /live, /view, /page, /phantom, /about
API routes	2100-2810	/health, /api/stats, /api/search, /api/media, /api/import/*
Crawl API	2811-3082	/api/crawl/jobs/* CRUD and control
Main entry	3199-3291	init_tarball_mode(), main()

neopig.py Breakdown (2935 lines)

Section	Lines	Content
Imports/globals	1-98	Imports, child process tracking, live queue
Job logging	99-115	Duplicate of serp.py's logging helpers
AppendOnlyStateLog	117-229	Fast crawl state persistence (~113 lines)
State helpers	231-306	get_state_log_path(), rotate_state_file(), logging setup
NeoPig class	307-1751	Core crawler class (~1444 lines)
HTML utilities	1752-1828	trim_html_wrapper(), extract_meta_from_html()
Multiprocess helpers	1830-1954	_process_single_page(), _process_chunk()
backfill_markdown	1955-2121	Async markdown regeneration (~167 lines)
backfill_screenshots	2122-2384	Async screenshot backfill (~262 lines)
CLI main()	2385-2935	Argument parsing + orchestration (~550 lines)

---

Proposed Structure

1. serp/ Module

serp/
├── __init__.py       # Exports app, main()
├── __main__.py       # Entry point: python -m serp
├── app.py            # FastAPI app creation, config, events (~150 lines)
├── i18n.py           # TRANSLATIONS, get_lang(), t(), inject_i18n() (~1200 lines)
├── templates.py      # Jinja2 setup, NAV_HTML, SEARCH_BOX_HTML (~100 lines)
├── models.py         # Pydantic request/response models (~100 lines)
├── archive.py        # Tarball mode: ArchiveDB, read helpers (~150 lines)
├── logging.py        # Job logging: start/stop/get_job_logs() (~50 lines)
└── routes/
    ├── __init__.py   # Router aggregation
    ├── pages.py      # HTML pages: /, /crawl, /live, /view, /about, /phantom (~600 lines)
    ├── search.py     # GET /api/search, /api/search/pages, /random (~150 lines)
    ├── crawl.py      # /api/crawl/* job management (~300 lines)
    ├── live.py       # SSE /api/live/stream (~50 lines)
    ├── media.py      # /media/{hash}, /api/media/{hash} (~200 lines)
    └── import_.py    # /import, /api/import/* upload handling (~300 lines)

Key changes:

• i18n becomes its own module (largest single component)
• Routes split by domain (search, crawl, media, import)
• Archive/tarball mode isolated for clarity
• Shared templates/nav moved to templates.py

2. neopig/ Module

neopig/
├── __init__.py       # Exports NeoPig, CrawlMode, main()
├── __main__.py       # Entry point: python -m neopig
├── crawler.py        # NeoPig class (~1000 lines after cleanup)
├── state.py          # AppendOnlyStateLog, state file helpers (~200 lines)
├── live.py           # Live queue: get_live_queue(), emit_live_media() (~30 lines)
├── cli.py            # main(), argparse, orchestration (~550 lines)
├── logging.py        # TqdmLoggingHandler, setup_logging() (~50 lines)
├── html_utils.py     # trim_html_wrapper(), extract_meta_from_html() (~80 lines)
└── backfill/
    ├── __init__.py
    ├── markdown.py   # backfill_markdown(), multiprocess helpers (~250 lines)
    └── screenshots.py # backfill_screenshots() (~270 lines)

Key changes:

• NeoPig class isolated in crawler.py
• State management extracted to state.py
• Backfill operations grouped in subpackage
• CLI separated from business logic
• HTML utilities extracted for reuse

---

Migration Strategy

Phase 1: Extract Without Breaking (Low Risk)

1. Create package directories

   mkdir -p serp/routes neopig/backfill
   

2. Extract pure utility modules first (no circular deps):

   • serp/i18n.py - TRANSLATIONS dict + functions
   • serp/models.py - Pydantic models
   • neopig/state.py - AppendOnlyStateLog class
   • neopig/html_utils.py - HTML processing functions
   • neopig/live.py - Queue management

3. Create compatibility shims in original files:

   # serp.py (temporary)
   from serp.i18n import TRANSLATIONS, get_lang, t, inject_i18n
   

Phase 2: Extract Routes (Medium Risk)

1. Create route modules with APIRouter:

   # serp/routes/search.py
   from fastapi import APIRouter
   router = APIRouter(tags=["search"])
   
   @router.get("/api/search")
   async def search(...): ...
   

2. Aggregate routers in serp/routes/__init__.py:

   from .search import router as search_router
   from .crawl import router as crawl_router
   # ...
   routers = [search_router, crawl_router, ...]
   

3. Update app.py to include routers:

   for router in routers:
       app.include_router(router)
   

Phase 3: Extract Core Classes (Higher Risk)

1. Move NeoPig class to neopig/crawler.py

   • Update all internal imports
   • Handle circular dependencies with TYPE_CHECKING

2. Move CLI to neopig/cli.py

   • Keep main() callable from both neopig.py and python -m neopig

3. Create package entry points:

   # neopig/__init__.py
   from .crawler import NeoPig
   from .cli import main
   from async_web_fetcher import CrawlMode
   
   __all__ = ['NeoPig', 'CrawlMode', 'main']
   

Phase 4: Deprecate Original Files

1. Convert serp.py to thin wrapper:

   #!/usr/bin/env python3
   """Legacy entry point. Use: python -m serp"""
   from serp import main
   if __name__ == "__main__":
       main()
   

2. Convert neopig.py to thin wrapper:

   #!/usr/bin/env python3
   """Legacy entry point. Use: python -m neopig"""
   from neopig import main
   import asyncio
   if __name__ == "__main__":
       asyncio.run(main())
   

---

Dependency Graph (Post-Refactor)

                    ┌─────────────┐
                    │  database   │
                    └──────┬──────┘
                           │
         ┌─────────────────┼─────────────────┐
         │                 │                 │
         ▼                 ▼                 ▼
   ┌──────────┐     ┌──────────┐     ┌──────────────┐
   │ neopig/  │     │  serp/   │     │ async_web_   │
   │ crawler  │────▶│  routes  │     │ fetcher      │
   └────┬─────┘     └────┬─────┘     └──────────────┘
        │                │
        ▼                ▼
   ┌──────────┐     ┌──────────┐
   │ neopig/  │     │  serp/   │
   │  state   │     │  i18n    │
   └──────────┘     └──────────┘

---

Circular Dependency Prevention

Known Risks

1. serp imports neopig.get_live_queue()

   • Solution: Move get_live_queue() to shared module or neopig/live.py
   • serp imports from neopig.live, not neopig.crawler

2. Job logging duplicated in both files

   • Solution: Consolidate in neopig/logging.py, serp imports from there

3. Database used by both

   • Already separate module - no change needed

TYPE_CHECKING Pattern

For type hints that would cause circular imports:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from neopig.crawler import NeoPig

def some_function(pig: "NeoPig") -> None:
    ...

---

Testing Strategy

Unit Tests Per Module

tests/unit/
├── test_serp_i18n.py       # Translation coverage
├── test_serp_models.py     # Pydantic validation
├── test_neopig_state.py    # AppendOnlyStateLog
├── test_neopig_html.py     # HTML utilities
└── test_neopig_crawler.py  # NeoPig class methods

Integration Tests

tests/integration/
├── test_serp_routes.py     # FastAPI TestClient
├── test_neopig_crawl.py    # End-to-end crawl
└── test_full_pipeline.py   # Crawl + SERP serving

---

Backwards Compatibility

Entry Points Preserved

# These continue to work:
python serp.py --port 31337
python neopig.py https://example.com

# New alternatives:
python -m serp --port 31337
python -m neopig https://example.com

Import Paths Preserved

# These continue to work:
from neopig import get_live_queue
from serp import app

# New alternatives:
from neopig.live import get_live_queue
from serp.app import app

---

Estimated Line Counts (Post-Refactor)

serp/ Package

Module	Lines	Notes
i18n.py	~1200	Mostly TRANSLATIONS dict
app.py	~150	App creation, config, events
templates.py	~100	Jinja2 setup, HTML fragments
models.py	~100	Pydantic models
archive.py	~150	Tarball mode
logging.py	~50	Job log helpers
routes/pages.py	~600	HTML page routes
routes/search.py	~150	Search API
routes/crawl.py	~300	Crawl job API
routes/live.py	~50	SSE stream
routes/media.py	~200	Media serving
routes/import_.py	~300	Import handling
Total	~3350	Slight overhead from structure

neopig/ Package

Module	Lines	Notes
crawler.py	~1000	NeoPig class (trimmed)
state.py	~200	State management
live.py	~30	Queue helpers
cli.py	~550	CLI entry point
logging.py	~50	tqdm-safe logging
html_utils.py	~80	HTML processing
backfill/markdown.py	~250	Markdown backfill
backfill/screenshots.py	~270	Screenshot backfill
Total	~2430	Cleaner than original

---

Open Questions

1. Should i18n be a separate top-level package?

   • Pro: Could be reused by other projects
   • Con: Adds complexity, tightly coupled to serp templates

2. Should backfill be part of neopig or separate?

   • Currently in neopig, but operates on database directly
   • Could be backfill/ at top level if other tools need it

3. Where should shared logging live?

   • Option A: neopig/logging.py (crawler owns it)
   • Option B: New common/ package
   • Option C: Keep duplicated (simplest)

---

Implementation Checklist

• Create serp/ directory structure
• Extract serp/i18n.py (TRANSLATIONS + functions)
• Extract serp/models.py (Pydantic models)
• Extract serp/archive.py (tarball mode)
• Extract serp/templates.py (Jinja2 setup)
• Extract serp/routes/ (all route handlers)
• Create serp/app.py (FastAPI app assembly)
• Create serp/__init__.py and __main__.py
• Create neopig/ directory structure
• Extract neopig/state.py (AppendOnlyStateLog)
• Extract neopig/live.py (queue management)
• Extract neopig/html_utils.py
• Extract neopig/logging.py
• Extract neopig/backfill/ (markdown + screenshots)
• Extract neopig/crawler.py (NeoPig class)
• Extract neopig/cli.py (main function)
• Create neopig/__init__.py and __main__.py
• Convert original files to thin wrappers
• Update all imports in dependent files
• Run full test suite
• Update CLAUDE.md with new structure