engineering/pig.py
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
262 commits 1 branch 0 tags 11 MiB
Commit graph

18 commits

Author SHA1 Message Date
russell@unturf.com
341bcb325a
docs: refine style rule; add Balance-All-Workstations + 8-capital sections 2026-05-22 12:24:34 -04:00
russell@unturf.com
02e87dd164 style: avoid "the", use "our" — writing style rule + sweep 2026-03-31 13:20:21 -04:00
timehexon
337e3ff70d Replace "AI" with "machine learning" in CLAUDE.md 
Machine learning is what we grow. "AI" is forbidden in all
permacomputer discourse, marketing, & documentation.
 2026-02-02 19:56:08 +00:00
russell@unturf.com
faf34e298b docs: update CLI examples to use un instead of un2 2026-01-17 10:12:31 -05:00
Russell Ballestrini
1df3e82be9 Add critical warning: never destroy prod services, use --redeploy 2026-01-04 15:47:28 -05:00
Russell Ballestrini
66d2f4d1d9 Port 31337 default, code file display fixes 
- Change default port from 8000 to 31337 across all files
- About page: and → &, beast → pig
- Search results: SVG placeholder for code files (shows extension)
- Download button: remove .bin fallback, use original extension
 2026-01-02 15:35:39 -05:00
Russell Ballestrini
34d0be61d4 Fix outdated Pyramid references - just Python 2026-01-02 14:44:49 -05:00
Russell Ballestrini
0696e92ae3 Add VCS repo cloning support (git/hg/svn/fossil) 
Smart source detection: like Hydra for RSS feeds, auto-detect
repository URLs and clone instead of HTTP crawling.

- repo.py: detect_vcs(), clone_repo(), walk_files(), symlink_to_vault()
- Symlink storage: files in repo_vault/ symlink to content-addressed vault/
- Database: repo_uri, repo_path, commit_hash, vcs_type columns
- Auto-detect GitHub, GitLab, Bitbucket, sr.ht, hg.mozilla.org, etc.
 2026-01-02 14:37:53 -05:00
Russell Ballestrini
af50334a7b Fix tests for code media type, add source URL and lore 2026-01-02 13:41:10 -05:00
Russell Ballestrini
52952a104a modified: CLAUDE.md 
modified:   archive.py
	new file:   bootstrap-neopig.sh
	modified:   bootstrap.c
	modified:   serp.py
 2026-01-02 11:51:42 -05:00
Russell Ballestrini
8b2d370c55 Update README and about page with new features 
- Hydra mode: RSS/Atom/Sitemap feed discovery and persistence
- Media provenance: first_seen_at, last_seen_at, discovered_at tracking
- Job purge: --purge-job CLI and DELETE API endpoint
- Updated feature grid on about page
- Added code examples for hydra and purge
 2026-01-01 16:56:41 -05:00
Russell Ballestrini
24120ad867 Update README for filevault 2.0, remove storage.py refs, add depth docs 2025-12-31 16:06:11 -05:00
Russell Ballestrini
b0542fccc6 Improve depth CLI help and add max depth validation 
- Clarify depth help: 0=single page, 1=page+links, -1=unlimited
- Reject depth >18, suggest using -1 for unlimited crawling
- Update CLAUDE.md with depth documentation
 2025-12-31 15:37:18 -05:00
Russell Ballestrini
16f869d7de Enable screenshots by default, --no-screenshot to disable 2025-12-30 15:21:41 -05:00
Russell Ballestrini
416b3cb760 Add self-extracting archives with bundled neopig 
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
 2025-12-30 06:55:20 -05:00
Russell Ballestrini
a1a58fbb50 Add backfill-markdown, download button, fix image hydration 
- backfill-markdown: regenerate markdown with absolute URLs for a domain
- Download button on /view page with slugified filename from alt/title
- html2text baseurl resolves relative URLs during crawl
- Fallback URL lookup by filename for older crawls
- Video hover-to-play on all video elements
- Avatar CSS float:left for text wrapping
- FileResponse inline disposition (view not download)
- Dynamic screenshot delay based on content length
 2025-12-29 15:05:39 -05:00
Russell Ballestrini
222c23bb14 modified: CLAUDE.md 
modified:   archive.py
	modified:   database.py
	modified:   neopig.py
	modified:   screenshot.py
	modified:   serp.py
 2025-12-29 12:13:33 -05:00
Russell Ballestrini
21e344f675 Add archive.py with --serve flag for live crawl watching 
- archive.py: New site archiver wrapper with embedded serve.py
- --serve flag starts SERP server alongside crawl for live viewing
- --fast mode for sites without robots.txt (no crawl delay)
- Symlink-based storage: domain views link to hash vault
- Tarball resolves symlinks to include only domain content
- Fixed bytes_downloaded/bytes_stored accounting for screenshots
- Data directory (data/) for databases and state files
- Standardized on - separator in filenames
 2025-12-29 11:26:02 -05:00