diff --git a/README.md b/README.md index eac18ec..667d13b 100644 --- a/README.md +++ b/README.md @@ -188,7 +188,19 @@ This spreads files across 256^9 possible directories, keeping the filesystem sna ### domain_vault.py — Triple Vault System -Three separate git-tracked vaults per domain, each with 9-layer deep hash paths: +Three separate git-tracked vaults per domain for full site archival. Unlike `filevault.py` (content-addressed by hash), domain vaults store files by **URL path** to mirror original site structure. + +**Key differences from filevault:** +- **URL-based paths** — Files stored at paths matching original URLs +- **Git versioning** — Each domain is a git repo (SSH cloneable) +- **Git LFS** — Media and screenshots use LFS for large files +- **Salted privacy** — Domain directories use salted hashes (set `NEOPIG_VAULT_SALT`) + +``` +html_vault/{9-layers}/{salted_hash}/about/index.html # Mirrors /about +media_vault/{9-layers}/{salted_hash}/images/logo.png # Mirrors /images/logo.png +linkpeek_vault/{9-layers}/{salted_hash}/about/index.png # Screenshot of /about +``` #### 1. DomainHtmlVault @@ -207,7 +219,7 @@ html_vault = manager.get_html_vault('example.com') await html_vault.init() # Archive a page (saves both original and rewritten versions) -is_changed, content_hash = await html_vault.archive_page( +is_changed, chash = await html_vault.archive_page( url='https://example.com/about', html=html_content, media_mappings={'https://cdn.example.com/logo.png': '/media/ab/cd/.../hash.png'} @@ -222,20 +234,20 @@ commit_hash = await html_vault.finish_crawl(stats) ``` **Files:** -- `{path}/index.html.og` — Original HTML with original URIs -- `{path}/index.html` — Rewritten with neopig media paths +- `{url_path}/index.html.og` — Original HTML with original URIs +- `{url_path}/index.html` — Rewritten with neopig media paths - `crawl_log.json` — Crawl history #### 2. DomainMediaVault -Stores images/videos with git LFS: +Stores images/videos with git LFS, mirroring original URL paths: ```python media_vault = manager.get_media_vault('example.com') await media_vault.init() -is_new, content_hash, file_path = await media_vault.archive_media( - url='https://example.com/images/photo.jpg', +is_new, chash, file_path = await media_vault.archive_media( + url='https://example.com/images/photo.jpg', # Stored at images/photo.jpg content=image_bytes, page_url='https://example.com/gallery' ) @@ -245,13 +257,15 @@ content, metadata = await media_vault.get_media(url) #### 3. DomainLinkpeekVault -Stores page screenshots with git LFS: +Stores page screenshots with git LFS. Each page gets one PNG at a path matching its URL: ```python linkpeek_vault = manager.get_linkpeek_vault('example.com') await linkpeek_vault.init() -is_new, content_hash, file_path = await linkpeek_vault.archive_screenshot( +# https://example.com/about -> about/index.png +# https://example.com/blog/post.html -> blog/post.png +is_new, chash, file_path = await linkpeek_vault.archive_screenshot( url='https://example.com/about', screenshot_data=png_bytes ) @@ -259,14 +273,20 @@ is_new, content_hash, file_path = await linkpeek_vault.archive_screenshot( content, metadata = await linkpeek_vault.get_screenshot(url) ``` +**Screenshot path mapping:** +| URL | Screenshot Path | +|-----|-----------------| +| `https://example.com/` | `index.png` | +| `https://example.com/about` | `about/index.png` | +| `https://example.com/blog/post.html` | `blog/post.png` | + **Helper Functions:** | Function | Description | |----------|-------------| | `get_vault_salt()` | Get NEOPIG_VAULT_SALT from environment | | `domain_hash(domain, salted)` | MD5 hash of domain (optionally salted for privacy) | -| `content_hash(content)` | MD5 hash of bytes for deduplication | -| `get_filevault_path(base, domain)` | Generate 9-layer deep path | +| `get_filevault_path(base, domain)` | Generate 9-layer deep path for domain | | `url_to_filepath(url)` | Convert URL to filesystem path | | `extract_media_urls(html, base_url)` | Extract all media URLs from HTML |