engineering/pig.py
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
22 commits 1 branch 0 tags 11 MiB
Commit graph

5 commits

Author SHA1 Message Date
Russell Ballestrini
cbed6d9d9a Speed up html2md: lxml parser, combined selectors, content hint 
- Use lxml parser (2-5x faster than html.parser), fallback to html.parser
- Combine 20+ noise selectors into single query (1 DOM traversal vs 20)
- Add hint parameter to skip content detection for known types
- Added lxml to requirements.txt
 2025-12-30 14:49:04 -05:00
Russell Ballestrini
1346590d39 ETL-style parallel markdown backfill with real-time progress 
- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL)
- ThreadPoolExecutor: 6 threads per process for I/O throughput
- Thread-local SQLite connections with WAL mode
- Manager().Value() for cross-process progress counter
- Real-time tqdm updates polling shared counter every 50ms
- ~19 pages/sec on 4-core system (4887 pages in 2 min)
 2025-12-30 14:22:29 -05:00
Russell Ballestrini
293ed64a3b modified: .gitignore 
modified:   database.py
	modified:   html2md.py
	modified:   neopig.py
	modified:   screenshot.py
	modified:   serp.py
 2025-12-30 11:35:25 -05:00
Russell Ballestrini
7b4c8b0241 modified: .gitignore 
modified:   archive.py
	modified:   html2md.py
	modified:   neopig.py
	modified:   screenshot.py
 2025-12-30 09:50:54 -05:00
Russell Ballestrini
416b3cb760 Add self-extracting archives with bundled neopig 
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
 2025-12-30 06:55:20 -05:00