engineering/pig.py
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
22 commits 1 branch 0 tags 11 MiB
Commit graph

9 commits

Author SHA1 Message Date
Russell Ballestrini
feb6eed596 Fix crawl UI: async task, real-time progress, better defaults 
- Fix asyncio error: use create_task() directly instead of BackgroundTasks
- Default depth=9, max_pages=-1 for full site crawls
- Add update_crawl_job_stats() for live progress updates
- Poll jobs every 2s while running, show live stats
- Animated progress bar for running jobs
- Fix get_media_type_from_extension() None path crash
 2025-12-30 15:14:49 -05:00
Russell Ballestrini
ee9c3cfc74 ETL backfill with jobs table, background flusher, smooth progress 
- backfill_jobs table tracks ETL progress with atomic increments
- Background flusher thread syncs progress every 50ms (workers never block)
- Thread-local DB connections for page writes (WAL mode)
- tqdm smoothing=0.1 for stable rate display (30-40 pages/sec)
- 4 processes x 6 threads = 24 workers burning all CPUs
- Zero lost counts - final flush verified against return values
 2025-12-30 14:43:25 -05:00
Russell Ballestrini
1346590d39 ETL-style parallel markdown backfill with real-time progress 
- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL)
- ThreadPoolExecutor: 6 threads per process for I/O throughput
- Thread-local SQLite connections with WAL mode
- Manager().Value() for cross-process progress counter
- Real-time tqdm updates polling shared counter every 50ms
- ~19 pages/sec on 4-core system (4887 pages in 2 min)
 2025-12-30 14:22:29 -05:00
Russell Ballestrini
293ed64a3b modified: .gitignore 
modified:   database.py
	modified:   html2md.py
	modified:   neopig.py
	modified:   screenshot.py
	modified:   serp.py
 2025-12-30 11:35:25 -05:00
Russell Ballestrini
416b3cb760 Add self-extracting archives with bundled neopig 
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
 2025-12-30 06:55:20 -05:00
Russell Ballestrini
a1a58fbb50 Add backfill-markdown, download button, fix image hydration 
- backfill-markdown: regenerate markdown with absolute URLs for a domain
- Download button on /view page with slugified filename from alt/title
- html2text baseurl resolves relative URLs during crawl
- Fallback URL lookup by filename for older crawls
- Video hover-to-play on all video elements
- Avatar CSS float:left for text wrapping
- FileResponse inline disposition (view not download)
- Dynamic screenshot delay based on content length
 2025-12-29 15:05:39 -05:00
Russell Ballestrini
222c23bb14 modified: CLAUDE.md 
modified:   archive.py
	modified:   database.py
	modified:   neopig.py
	modified:   screenshot.py
	modified:   serp.py
 2025-12-29 12:13:33 -05:00
Russell Ballestrini
444317a71b modified: async_web_fetcher.py 
modified:   database.py
	modified:   neopig.py
 2025-12-29 05:08:31 -05:00
Russell Ballestrini
6dab26d6fa Initial neopig: async media crawler with MD5 deduplication 
- Multi-target concurrent crawling support
- Content-addressable vault storage (MD5 hash)
- SQLite database with page context for searchability
- Live SERP with real-time polling feed (/live)
- URI naming convention (media_uri, page_uri)
- GIF and image object-fit: contain for proper display
 2025-12-21 17:26:42 -05:00