Migrated from https://git2.unturf.com/engineering/unturf/pig.py.git
Russell Ballestrini
6dab26d6fa
- Multi-target concurrent crawling support - Content-addressable vault storage (MD5 hash) - SQLite database with page context for searchability - Live SERP with real-time polling feed (/live) - URI naming convention (media_uri, page_uri) - GIF and image object-fit: contain for proper display |
||
|---|---|---|
| .gitignore | ||
| async_web_fetcher.py | ||
| Caddyfile | ||
| Caddyfile.direct | ||
| database.py | ||
| Makefile | ||
| neopig.py | ||
| requirements.txt | ||
| serp.py | ||
| storage.py | ||
| test_crawl.py | ||