engineering/pig.py
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

README: clarify domain_vault vs filevault, add linkpeek path mapping

Browse source
This commit is contained in:
Russell Ballestrini 2025-12-31 15:46:37 -05:00
parent 99c836e2c8
commit da538ea9b0
1 changed files with 31 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

42
README.md
View file

@ -188,7 +188,19 @@ This spreads files across 256^9 possible directories, keeping the filesystem sna
### domain_vault.py — Triple Vault System
Three separate git-tracked vaults per domain, each with 9-layer deep hash paths:
Three separate git-tracked vaults per domain for full site archival. Unlike `filevault.py` (content-addressed by hash), domain vaults store files by **URL path** to mirror original site structure.
**Key differences from filevault:**
- **URL-based paths** — Files stored at paths matching original URLs
- **Git versioning** — Each domain is a git repo (SSH cloneable)
- **Git LFS** — Media and screenshots use LFS for large files
- **Salted privacy** — Domain directories use salted hashes (set `NEOPIG_VAULT_SALT`)
```
html_vault/{9-layers}/{salted_hash}/about/index.html # Mirrors /about
media_vault/{9-layers}/{salted_hash}/images/logo.png # Mirrors /images/logo.png
linkpeek_vault/{9-layers}/{salted_hash}/about/index.png # Screenshot of /about
```
#### 1. DomainHtmlVault
@ -207,7 +219,7 @@ html_vault = manager.get_html_vault('example.com')
await html_vault.init()
# Archive a page (saves both original and rewritten versions)
is_changed, content_hash = await html_vault.archive_page(
is_changed, chash = await html_vault.archive_page(
url='https://example.com/about',
html=html_content,
media_mappings={'https://cdn.example.com/logo.png': '/media/ab/cd/.../hash.png'}
@ -222,20 +234,20 @@ commit_hash = await html_vault.finish_crawl(stats)
```
**Files:**
- `{path}/index.html.og` — Original HTML with original URIs
- `{path}/index.html` — Rewritten with neopig media paths
- `{url_path}/index.html.og` — Original HTML with original URIs
- `{url_path}/index.html` — Rewritten with neopig media paths
- `crawl_log.json` — Crawl history
#### 2. DomainMediaVault
Stores images/videos with git LFS:
Stores images/videos with git LFS, mirroring original URL paths:
```python
media_vault = manager.get_media_vault('example.com')
await media_vault.init()
is_new, content_hash, file_path = await media_vault.archive_media(
url='https://example.com/images/photo.jpg',
is_new, chash, file_path = await media_vault.archive_media(
url='https://example.com/images/photo.jpg', # Stored at images/photo.jpg
content=image_bytes,
page_url='https://example.com/gallery'
)
@ -245,13 +257,15 @@ content, metadata = await media_vault.get_media(url)
#### 3. DomainLinkpeekVault
Stores page screenshots with git LFS:
Stores page screenshots with git LFS. Each page gets one PNG at a path matching its URL:
```python
linkpeek_vault = manager.get_linkpeek_vault('example.com')
await linkpeek_vault.init()
is_new, content_hash, file_path = await linkpeek_vault.archive_screenshot(
# https://example.com/about -> about/index.png
# https://example.com/blog/post.html -> blog/post.png
is_new, chash, file_path = await linkpeek_vault.archive_screenshot(
url='https://example.com/about',
screenshot_data=png_bytes
)
@ -259,14 +273,20 @@ is_new, content_hash, file_path = await linkpeek_vault.archive_screenshot(
content, metadata = await linkpeek_vault.get_screenshot(url)
```
**Screenshot path mapping:**
| URL | Screenshot Path |
|-----|-----------------|
| `https://example.com/` | `index.png` |
| `https://example.com/about` | `about/index.png` |
| `https://example.com/blog/post.html` | `blog/post.png` |
**Helper Functions:**
| Function | Description |
|----------|-------------|
| `get_vault_salt()` | Get NEOPIG_VAULT_SALT from environment |
| `domain_hash(domain, salted)` | MD5 hash of domain (optionally salted for privacy) |
| `content_hash(content)` | MD5 hash of bytes for deduplication |
| `get_filevault_path(base, domain)` | Generate 9-layer deep path |
| `get_filevault_path(base, domain)` | Generate 9-layer deep path for domain |
| `url_to_filepath(url)` | Convert URL to filesystem path |
| `extract_media_urls(html, base_url)` | Extract all media URLs from HTML |