engineering/pig.py
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix tests for code media type, add source URL and lore

Browse source
This commit is contained in:
Russell Ballestrini 2026-01-02 13:41:10 -05:00
parent 4bab2ce503
commit af50334a7b
4 changed files with 841 additions and 120 deletions
Download patch file Download diff file Expand all files Collapse all files

2
CLAUDE.md
View file

@ -48,6 +48,8 @@ pytest tests/ -v
Based on [pig.py](http://russell.ballestrini.net/python-image-grabber-pig-py/) by Russell Ballestrini.
**Source:** [git.unturf.com/engineering/unturf/pig.py](https://git.unturf.com/engineering/unturf/pig.py) (public domain)
## Architecture
```

18
README.md
View file

@ -4,6 +4,8 @@
Based on [pig.py](http://russell.ballestrini.net/python-image-grabber-pig-py/) by Russell Ballestrini
**Source:** [git.unturf.com/engineering/unturf/pig.py](https://git.unturf.com/engineering/unturf/pig.py) (public domain)
---
## The Origin Story
@ -21,7 +23,21 @@ python pig.py https://www.foxhop.net
That was it. Point it at a URL, and it would slurp down every image it could find. No configuration, no complexity — just a hungry little pig gobbling up pixels.
Over a decade later, **neopig** emerged as a spiritual successor — keeping the original's spirit of simplicity while adding the features needed for serious web archival.
### The Great Bitbucket Extinction
Then came the dark times. The original source lived at ~~[bitbucket.org/russellballestrini/pig](https://bitbucket.org/russellballestrini/pig)~~ — click it, we dare you. In 2020, Atlassian swallowed Bitbucket whole and spat out a glorious fountain of Mercurial repositories into the void. The original pig.py, nestled in its cozy hg repo, was atomized in the great purge.
But here's where it gets *weird*.
Russell once wrote that ["programming is like alchemy — instead of exchanging matter, we programmers exchange time."](https://russell.ballestrini.net/programming-is-like-alchemy/) Programs are golems, familiar spirits, magical servants performing repetitive tasks. *"It is more accurate to group programs with technology than magic, but less fun."*
And speaking of alchemy: [Marathon Fusion](https://phys.org/news/2025-07-marathon-fusion-mercury-gold-energy.html) discovered that tokamak breeding blankets — wrapped in Mercury-Lithium alloy, like pigs in a blanket — can transmute Mercury-198 into Gold-197 through chrysopoeia. Fast neutrons trigger (n, 2n) reactions; unstable mercury decays into stable gold within 64 hours. Two metric tons of gold per gigawatt. The alchemists' dream realized, wrapped in radioactive patience (17.7 years of cooling before you can touch your transmuted treasure).
A golden goose born from the ashes of deprecated version control. The old pig was archived, but a new creature stirred in the digital depths...
### The Evolution to neopig
Like a phoenix rising from dead Bitbucket repos, **neopig** emerged — a chimera, a griffin, a more hungry and gluttonous beast than its predecessor ever dreamed of being. Where pig.py sipped politely from single pages, neopig *devours entire domains*.
---

925
serp.py
View file

@ -3120,162 +3120,853 @@ async def phantom_page(lang: str = Cookie(None), accept_language: str = Header(N
ABOUT_CSS = """
.about-content {
max-width: 800px;
margin: 0 auto;
padding: 40px 20px;
line-height: 1.8;
font-size: 17px;
.about-page { width: 100%; min-height: 100vh; background: #0a0a0f; }
/* Hero Section */
.hero-section {
width: 100%;
min-height: 100vh;
display: flex;
flex-direction: column;
justify-content: center;
align-items: center;
background: linear-gradient(180deg, #0a0a0f 0%, #1a0a1a 50%, #0f0a1a 100%);
position: relative;
overflow: hidden;
padding: 60px 20px;
}
.hero-section::before {
content: '';
position: absolute;
top: 0; left: 0; right: 0; bottom: 0;
background: radial-gradient(ellipse at 50% 0%, rgba(255,50,100,0.15) 0%, transparent 60%),
radial-gradient(ellipse at 20% 80%, rgba(100,50,255,0.1) 0%, transparent 40%),
radial-gradient(ellipse at 80% 60%, rgba(255,100,50,0.08) 0%, transparent 40%);
pointer-events: none;
}
.hero-pig {
width: 280px;
height: auto;
filter: drop-shadow(0 0 60px rgba(255,100,100,0.5)) drop-shadow(0 0 120px rgba(255,50,100,0.3));
animation: pulse-glow 4s ease-in-out infinite;
margin-bottom: 40px;
}
@keyframes pulse-glow {
0%, 100% { filter: drop-shadow(0 0 60px rgba(255,100,100,0.5)) drop-shadow(0 0 120px rgba(255,50,100,0.3)); }
50% { filter: drop-shadow(0 0 80px rgba(255,100,100,0.7)) drop-shadow(0 0 160px rgba(255,50,100,0.5)); }
}
.hero-title {
font-size: 6em;
font-weight: 900;
letter-spacing: -0.02em;
background: linear-gradient(135deg, #ff6b6b 0%, #ff8e53 50%, #ff6b9d 100%);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
background-clip: text;
text-shadow: 0 0 80px rgba(255,100,100,0.5);
margin: 0;
text-align: center;
}
.hero-subtitle {
font-size: 1.4em;
color: #888;
margin-top: 20px;
font-weight: 300;
letter-spacing: 0.3em;
text-transform: uppercase;
}
.hero-tagline {
font-size: 1.1em;
color: #666;
margin-top: 30px;
max-width: 600px;
text-align: center;
line-height: 1.8;
}
/* Chapter Sections */
.chapter {
width: 100%;
padding: 120px 40px;
position: relative;
}
.chapter-content {
max-width: 900px;
margin: 0 auto;
}
.chapter-number {
font-size: 8em;
font-weight: 900;
color: rgba(255,100,100,0.08);
position: absolute;
top: 40px;
left: 40px;
line-height: 1;
font-family: 'Georgia', serif;
}
.chapter-title {
font-size: 3em;
font-weight: 800;
color: #ff6b6b;
margin-bottom: 40px;
position: relative;
}
.chapter-title::after {
content: '';
display: block;
width: 100px;
height: 4px;
background: linear-gradient(90deg, #ff6b6b, transparent);
margin-top: 20px;
}
/* Narrative Text */
.narrative {
font-size: 1.3em;
line-height: 2;
color: #c0c0c0;
margin-bottom: 30px;
}
.narrative em { color: #ff8e53; font-style: italic; }
.narrative strong { color: #fff; font-weight: 600; }
.narrative a { color: #ff6b6b; text-decoration: none; border-bottom: 1px solid rgba(255,107,107,0.3); transition: all 0.3s; }
.narrative a:hover { color: #ff8e53; border-bottom-color: #ff8e53; }
/* Dramatic Quote */
.dramatic-quote {
padding: 60px;
margin: 60px 0;
background: linear-gradient(135deg, rgba(255,50,100,0.1) 0%, rgba(100,50,200,0.1) 100%);
border-left: 6px solid #ff6b6b;
position: relative;
}
.dramatic-quote::before {
content: '"';
font-size: 12em;
color: rgba(255,100,100,0.15);
position: absolute;
top: -40px;
left: 20px;
font-family: 'Georgia', serif;
line-height: 1;
}
.dramatic-quote p {
font-size: 1.8em;
font-style: italic;
color: #e0e0e0;
margin: 0;
position: relative;
z-index: 1;
}
.dramatic-quote cite {
display: block;
margin-top: 30px;
font-size: 1em;
color: #888;
font-style: normal;
}
/* Dead Link Styling */
.dead-link {
text-decoration: line-through;
color: #666 !important;
cursor: not-allowed;
position: relative;
}
.dead-link::after {
content: ' [DEAD]';
color: #ff4444;
font-size: 0.7em;
font-weight: bold;
text-decoration: none;
}
/* Dark Chapter */
.chapter-dark {
background: linear-gradient(180deg, #0a0a0f 0%, #150810 50%, #0a0a0f 100%);
}
.chapter-dark .chapter-number { color: rgba(255,50,50,0.1); }
/* Rebirth Chapter - TempleOS cyan */
.chapter-rebirth {
background: linear-gradient(180deg, #0a0a0f 0%, #0a1515 50%, #0a0a0f 100%);
}
.chapter-rebirth .chapter-title { color: #00FFFF; /* TempleOS cyan */ }
.chapter-rebirth .chapter-title::after { background: linear-gradient(90deg, #00FFFF, transparent); }
.chapter-rebirth .chapter-number { color: rgba(0,255,255,0.08); }
/* Beast Section */
.chapter-beast {
background: linear-gradient(180deg, #0a0a0f 0%, #1a0a05 50%, #0a0a0f 100%);
}
.chapter-beast .chapter-title { color: #ff8e53; }
.chapter-beast .chapter-title::after { background: linear-gradient(90deg, #ff8e53, transparent); }
/* Feature Cards - Full Width Grid */
.feature-section {
width: 100%;
padding: 100px 40px;
background: linear-gradient(180deg, #0a0a0f 0%, #0f0f18 50%, #0a0a0f 100%);
}
.feature-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));
gap: 30px;
max-width: 1400px;
margin: 0 auto;
}
.feature-card {
background: linear-gradient(145deg, rgba(30,30,40,0.8) 0%, rgba(20,20,30,0.9) 100%);
padding: 40px;
border-radius: 16px;
border: 1px solid rgba(255,100,100,0.1);
transition: all 0.4s ease;
position: relative;
overflow: hidden;
}
.feature-card::before {
content: '';
position: absolute;
top: 0; left: 0;
width: 100%; height: 4px;
background: linear-gradient(90deg, #ff6b6b, #ff8e53, #4ecdc4);
opacity: 0;
transition: opacity 0.4s;
}
.feature-card:hover {
transform: translateY(-5px);
border-color: rgba(255,100,100,0.3);
box-shadow: 0 20px 60px rgba(0,0,0,0.4), 0 0 40px rgba(255,100,100,0.1);
}
.feature-card:hover::before { opacity: 1; }
.feature-card h3 {
font-size: 1.4em;
color: #ff6b6b;
margin-bottom: 15px;
font-weight: 700;
}
.feature-card p {
color: #999;
line-height: 1.7;
font-size: 1.05em;
}
.feature-card a { color: #4ecdc4; }
/* Code Blocks */
.code-section {
background: #0d0d12;
padding: 40px;
border-radius: 12px;
margin: 40px 0;
overflow-x: auto;
border: 1px solid #222;
}
.code-section pre {
margin: 0;
font-family: 'Fira Code', 'Monaco', monospace;
font-size: 1em;
line-height: 1.8;
color: #e0e0e0;
}
.code-section code { background: none; }
/* Inline Code */
.narrative code {
background: rgba(255,100,100,0.1);
padding: 3px 10px;
border-radius: 4px;
font-family: 'Fira Code', monospace;
font-size: 0.9em;
color: #ff8e53;
}
/* Mirror Section - Special */
.chapter-mirror {
background: linear-gradient(180deg, #0a0a0f 0%, #0a0a1f 30%, #100a1a 70%, #0a0a0f 100%);
position: relative;
}
.chapter-mirror::before {
content: '';
position: absolute;
top: 0; left: 0; right: 0; bottom: 0;
background: repeating-linear-gradient(
90deg,
transparent 0px,
transparent 100px,
rgba(100,100,255,0.02) 100px,
rgba(100,100,255,0.02) 101px
);
pointer-events: none;
}
/* The Forge - TempleOS background */
.chapter-forge {
background: url('/static/vendor/templeos.png') center center no-repeat,
linear-gradient(180deg, #0a0a0f 0%, #0a1515 30%, #0a1515 70%, #0a0a0f 100%);
background-size: contain, cover;
position: relative;
}
.chapter-forge::before {
content: '';
position: absolute;
top: 0; left: 0; right: 0; bottom: 0;
background: rgba(0, 0, 0, 0.75);
pointer-events: none;
}
.chapter-forge .chapter-content {
position: relative;
z-index: 1;
}
.chapter-forge .chapter-title { color: #00FFFF; /* TempleOS cyan */ }
.chapter-forge .chapter-title::after { background: linear-gradient(90deg, #00FFFF, transparent); }
.chapter-forge .chapter-number { color: rgba(0,255,255,0.1); }
/* The Rule - 72 Benedictine principles */
.rule-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(400px, 1fr));
gap: 12px;
margin: 40px 0;
}
.rule-item {
font-size: 1.05em;
line-height: 1.6;
color: #b0b0b0;
padding: 12px 16px;
background: rgba(255,255,255,0.02);
border-radius: 6px;
border-left: 3px solid rgba(255,107,107,0.3);
}
.rule-item strong {
color: #ff6b6b;
font-weight: 700;
}
@media (max-width: 768px) {
.rule-grid { grid-template-columns: 1fr; }
}
.chapter-mirror .chapter-title { color: #8b5cf6; }
.chapter-mirror .chapter-title::after { background: linear-gradient(90deg, #8b5cf6, transparent); }
.chapter-mirror .chapter-number { color: rgba(139,92,246,0.08); }
/* Footer */
.about-footer {
width: 100%;
padding: 80px 40px;
background: #050508;
text-align: center;
}
.about-footer p {
color: #666;
font-size: 1.1em;
margin-bottom: 20px;
}
.about-footer a {
color: #ff6b6b;
font-size: 1.2em;
text-decoration: none;
border-bottom: 2px solid rgba(255,107,107,0.3);
padding-bottom: 4px;
transition: all 0.3s;
}
.about-footer a:hover {
color: #ff8e53;
border-bottom-color: #ff8e53;
}
/* Responsive */
@media (max-width: 768px) {
.hero-title { font-size: 3.5em; }
.hero-pig { width: 200px; }
.chapter { padding: 80px 20px; }
.chapter-number { font-size: 5em; top: 20px; left: 20px; }
.chapter-title { font-size: 2em; }
.narrative { font-size: 1.1em; }
.dramatic-quote { padding: 30px; }
.dramatic-quote p { font-size: 1.3em; }
.feature-section { padding: 60px 20px; }
}
.about-content h1 { font-size: 2.5em; margin-bottom: 10px; color: #ff6b6b; }
.about-content h2 { font-size: 1.6em; margin-top: 40px; margin-bottom: 15px; color: #4ecdc4; border-bottom: 2px solid #333; padding-bottom: 10px; }
.about-content p { margin-bottom: 20px; color: #e0e0e0; }
.about-content a { color: #ff6b6b; text-decoration: none; }
.about-content a:hover { text-decoration: underline; }
.about-content code { background: #2a2a2a; padding: 2px 8px; border-radius: 4px; font-family: 'Fira Code', monospace; font-size: 0.9em; }
.about-content pre { background: #1a1a1a; padding: 20px; border-radius: 8px; overflow-x: auto; border: 1px solid #333; }
.about-content pre code { background: none; padding: 0; }
.about-content ul, .about-content ol { margin-bottom: 20px; padding-left: 30px; }
.about-content li { margin-bottom: 10px; color: #e0e0e0; }
.feature-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); gap: 20px; margin: 30px 0; }
.feature-card { background: #1e1e24; padding: 20px; border-radius: 12px; border: 1px solid #333; }
.feature-card h3 { color: #ff6b6b; margin-bottom: 10px; font-size: 1.1em; }
.feature-card p { color: #aaa; font-size: 0.95em; margin: 0; }
.origin-quote { background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%); border-left: 4px solid #ff6b6b; padding: 20px 25px; margin: 30px 0; border-radius: 0 8px 8px 0; font-style: italic; }
.origin-quote cite { display: block; margin-top: 10px; color: #888; font-style: normal; }
"""
ABOUT_CONTENT = """
<div class="about-content">
<h1>🐷 neopig</h1>
<p><strong>Neo Python Image Grabber</strong> &mdash; A full-domain async media crawler with content-addressable storage, full-text search, and page screenshot capture.</p>
<div class="about-page">
<h2>The Origin Story</h2>
<!-- HERO SECTION -->
<section class="hero-section">
<img src="https://russell.ballestrini.net/uploads/pigpy.png" alt="pig.py" class="hero-pig">
<h1 class="hero-title">neopig</h1>
<p class="hero-subtitle">Neo Python Image Grabber</p>
<p class="hero-tagline">
A chimera born from dead repositories. A beast that devours domains whole.
<br>Where others archive pages, we <em>consume realities</em>.
</p>
</section>
<div class="origin-quote">
"pig.py is a <em>very</em> simple python command line tool to download all the images from a given uri."
<cite>&mdash; Russell Ballestrini, August 22, 2011</cite>
<!-- CHAPTER I: GENESIS -->
<section class="chapter">
<div class="chapter-number">I</div>
<div class="chapter-content">
<h2 class="chapter-title">Genesis</h2>
<div class="dramatic-quote">
<p>pig.py is a <em>very</em> simple python command line tool to download all the images from a given uri.</p>
<cite>&mdash; Russell Ballestrini, August 22, 2011</cite>
</div>
<p class="narrative">
In the beginning, there was <a href="https://russell.ballestrini.net/python-image-grabber-pig-py/" target="_blank">pig.py</a>. A simple creature. Innocent. <em>Hungry.</em>
</p>
<p class="narrative">
Russell Ballestrini crafted it in the summer of 2011 &mdash; a humble Python script with a singular appetite: <strong>images</strong>. Point it at a webpage, and it would slurp down every pixel it could find. No configuration. No complexity. Just a hungry little pig gobbling up the visual fabric of the web.
</p>
<div class="code-section">
<pre><code>python pig.py https://www.foxhop.net</code></pre>
</div>
<p class="narrative">
That was it. The entire interface. The pig asked only for a target, and it <em>fed</em>.
</p>
<p class="narrative">
Russell released it into the public domain &mdash; a gift to anyone who needed to harvest images from the wild web. The source lived at <code>bitbucket.org/russellballestrini/pig</code>, nestled safely in a Mercurial repository. The pig slept soundly in its pen, unaware of the extinction event approaching.
</p>
</div>
</section>
<p>Back in 2011, <a href="https://russell.ballestrini.net/python-image-grabber-pig-py/" target="_blank">Russell Ballestrini created pig.py</a> &mdash; a delightfully simple Python script that did one thing well: download all the images from a webpage. It was placed in the public domain, a gift to anyone who needed to grab images from the web.</p>
<!-- CHAPTER II: THE EXTINCTION -->
<section class="chapter chapter-dark">
<div class="chapter-number">II</div>
<div class="chapter-content">
<h2 class="chapter-title">The Great Bitbucket Extinction</h2>
<p>The original pig.py was elegant in its simplicity:</p>
<p class="narrative">
<strong>Then came the dark times.</strong>
</p>
<pre><code>python pig.py https://www.foxhop.net</code></pre>
<p class="narrative">
In 2020, Atlassian &mdash; the corporate leviathan that had swallowed Bitbucket whole &mdash; decreed that Mercurial repositories would be <em>purged</em>. Millions of hg repos, atomized. Scattered like digital ash across the void.
</p>
<p>That was it. Point it at a URL, and it would slurp down every image it could find. No configuration, no complexity &mdash; just a hungry little pig gobbling up pixels.</p>
<p class="narrative">
The original pig.py, that innocent image-gobbling creature, was among the casualties. Its home at <a href="https://bitbucket.org/russellballestrini/pig" class="dead-link">bitbucket.org/russellballestrini/pig</a> became a tombstone. Click it. We dare you. There's nothing there but the echo of corporate indifference.
</p>
<h2>The Great Bitbucket Extinction</h2>
<p>Then came the dark times. The original source lived at <code><a href="https://bitbucket.org/russellballestrini/pig" style="text-decoration:line-through;color:#666;">bitbucket.org/russellballestrini/pig</a></code> &mdash; click it, we dare you. In 2020, Atlassian swallowed Bitbucket whole and spat out a glorious fountain of Mercurial repositories into the void. The original pig.py, nestled in its cozy hg repo, was atomized in the great purge.</p>
<p>But here's where it gets <em>weird</em>.</p>
<p>Russell once wrote that <a href="https://russell.ballestrini.net/programming-is-like-alchemy/">"programming is like alchemy &mdash; instead of exchanging matter, we programmers exchange time."</a> Programs are golems, familiar spirits, magical servants performing repetitive tasks. <em>"It is more accurate to group programs with technology than magic, but less fun."</em></p>
<p>And speaking of alchemy: <a href="https://phys.org/news/2025-07-marathon-fusion-mercury-gold-energy.html">Marathon Fusion</a> discovered that tokamak breeding blankets &mdash; wrapped in Mercury-Lithium alloy, like pigs in a blanket &mdash; can transmute Mercury-198 into Gold-197 through chrysopoeia. Fast neutrons trigger (n, 2n) reactions; unstable mercury decays into stable gold within 64 hours. Two metric tons of gold per gigawatt. The alchemists' dream realized, wrapped in radioactive patience (17.7 years of cooling before you can touch your transmuted treasure).</p>
<p>A golden goose born from the ashes of deprecated version control. The old pig was archived, but a new creature stirred in the digital depths...</p>
<h2>The Evolution to neopig</h2>
<p>Like a phoenix rising from dead Bitbucket repos, neopig emerged &mdash; a chimera, a griffin, a more hungry and gluttonous beast than its predecessor ever dreamed of being. Where pig.py sipped politely from single pages, neopig <em>devours entire domains</em>.</p>
<p>Over a decade later, this spiritual successor keeps the original's spirit of simplicity while adding the features needed for serious web archival:</p>
<div class="feature-grid">
<div class="feature-card">
<h3>🔄 Async Crawling</h3>
<p>Full-domain recursive crawling with configurable depth, respecting robots.txt and crawl delays.</p>
<div class="dramatic-quote">
<p>They literally killed pig.py and sent him to bitbucket.</p>
<cite>&mdash; The cruel irony of naming your graveyard after a slang term for death</cite>
</div>
<div class="feature-card">
<h3>💾 Content-Addressed Storage</h3>
<p>MD5-based deduplication in a 9-deep vault. Same image from 100 pages? Stored once.</p>
<p class="narrative">
<em>"Sent to the bitbucket"</em> &mdash; programmer slang from the age of punch cards. When you discarded bad data, you threw it in the bit bucket. Atlassian named their service after a trash can, then proved the prophecy true by throwing away everyone's code.
</p>
<p class="narrative">
The pig was dead. Its bones scattered across cached search results and archived blog posts. A ghost in the machine, referenced but unreachable.
</p>
<p class="narrative">
But here's where it gets <em>weird</em>.
</p>
</div>
</section>
<!-- CHAPTER III: ALCHEMY -->
<section class="chapter chapter-rebirth">
<div class="chapter-number">III</div>
<div class="chapter-content">
<h2 class="chapter-title">The Alchemy</h2>
<p class="narrative">
Russell once wrote that <a href="https://russell.ballestrini.net/programming-is-like-alchemy/" target="_blank">"programming is like alchemy &mdash; instead of exchanging matter, we programmers exchange time."</a>
</p>
<p class="narrative">
Programs are <strong>golems</strong>. Familiar spirits. Magical servants performing repetitive tasks so we don't have to. <em>"It is more accurate to group programs with technology than magic, but less fun."</em>
</p>
<p class="narrative">
Years passed. The web continued its relentless churn &mdash; sites going dark, forums shutting down, communities scattering like startled birds. Somewhere, a very angry ex-Ruby developer (Python too) watched the digital decay and remembered the pig.
</p>
<div class="dramatic-quote">
<p>What if we could transmute that dead code into gold?</p>
<cite>&mdash; The thought that started everything</cite>
</div>
<div class="feature-card">
<h3>🔍 Full-Text Search</h3>
<p>Every image indexed with its surrounding context &mdash; page title, alt text, captions, nearby headings.</p>
<p class="narrative">
And speaking of transmutation: <a href="https://phys.org/news/2025-07-marathon-fusion-mercury-gold-energy.html" target="_blank">Marathon Fusion</a> discovered that tokamak breeding blankets &mdash; wrapped in Mercury-Lithium alloy, like <em>pigs in a blanket</em> &mdash; can transmute Mercury-198 into Gold-197 through chrysopoeia. Fast neutrons trigger (n, 2n) reactions; unstable mercury decays into stable gold within 64 hours.
</p>
<p class="narrative">
Two metric tons of gold per gigawatt. The alchemists' dream realized, wrapped in radioactive patience (17.7 years of cooling before you can touch your transmuted treasure).
</p>
<p class="narrative">
A golden goose born from the ashes of deprecated version control. The old pig was archived, but <strong>a new creature stirred in the digital depths...</strong>
</p>
</div>
</section>
<!-- CHAPTER IV: THE BEAST AWAKENS -->
<section class="chapter chapter-beast">
<div class="chapter-number">IV</div>
<div class="chapter-content">
<h2 class="chapter-title">The Beast Awakens</h2>
<p class="narrative">
Like a phoenix rising from dead Bitbucket repos, <strong>neopig</strong> emerged.
</p>
<p class="narrative">
Not a resurrection. Not a mere fork. Something <em>else</em>. A chimera. A griffin. A creature stitched together from the DNA of the original pig and the fevered dreams of someone who had watched too many sites die.
</p>
<div class="dramatic-quote">
<p>A beast with an appetite like Gluttony from Fullmetal Alchemist &mdash; a homunculus that devours everything in its path, absorbing knowledge, power, and form.</p>
</div>
<div class="feature-card">
<h3>📸 Page Screenshots</h3>
<p><a href="https://linkpeek.com">LinkPeek</a> reborn! Full-page captures via <a href="https://github.com/russellballestrini/uri2png">uri2png</a> (wkhtmltoimage, Playwright, etc).</p>
<p class="narrative">
Where pig.py sipped politely from single pages, neopig <strong>devours entire domains</strong>. It doesn't just grab images &mdash; it consumes HTML, screenshots pages, converts content to markdown, indexes every scrap of text, and stores it all in content-addressed vaults that will outlast the heat death of the original servers.
</p>
<p class="narrative">
The code lives now at <a href="https://git.unturf.com/engineering/unturf/pig.py" target="_blank">git.unturf.com/engineering/unturf/pig.py</a>. And it keeps evolving. Every crawl adds new capabilities. Every archived site teaches it new tricks.
</p>
<p class="narrative">
<em>We are adding the ability to backup entire codebases.</em> The beast grows hungrier.
</p>
</div>
</section>
<!-- CHAPTER V: THE MIRROR DIMENSION -->
<section class="chapter chapter-mirror">
<div class="chapter-number">V</div>
<div class="chapter-content">
<h2 class="chapter-title">The Mirror Dimension</h2>
<p class="narrative">
We are about to <strong>invert the git tree into itself</strong>.
</p>
<p class="narrative">
Like Dr. Strange versus Spider-Man in the mirror dimension &mdash; that impossible space where geometry folds back on itself, where buildings become M.C. Escher nightmares and reality reflects endlessly into its own depths.
</p>
<p class="narrative">
neopig doesn't just archive websites. It can archive <em>itself</em>. It can archive the repositories that contain the code that does the archiving. It can crawl git forges and preserve the source code of tools that preserve source code.
</p>
<div class="dramatic-quote">
<p>Our spider-pig gets out of the danger of the mirror void &mdash; where days are years &mdash; and exits absorbing everything like a griffin, like a chimera, like a beast that feeds on the bones of dead platforms.</p>
</div>
<div class="feature-card">
<h3>🐙 Hydra Mode</h3>
<p>Auto-discover RSS/Atom/Sitemap feeds. Persists feed URLs, checks for new content on every crawl.</p>
</div>
<div class="feature-card">
<h3>📊 Media Provenance</h3>
<p>Track first_seen, last_seen, and when each page first linked to media. See reuse across the web.</p>
</div>
<div class="feature-card">
<h3>📝 Markdown Conversion</h3>
<p>Intelligent HTML-to-Markdown with forum post detection, noise removal, and content extraction.</p>
</div>
<div class="feature-card">
<h3>📦 Distributable Archives</h3>
<p>Create self-contained tar.gz packages with embedded search servers for offline browsing.</p>
<p class="narrative">
When a code hosting platform dies (and they all die eventually &mdash; Google Code, Gitorious, BitBucket's hg repos), neopig ensures the knowledge survives. We are the ark. We are the vault. We are the beast that remembers.
</p>
<p class="narrative">
The archives become self-extracting executables. <code>.run</code> files that contain their own server, their own search engine, their own reality. Drop one on a machine with Python and Pyramid, and it <em>wakes up</em>. A sleeping beast, carrying an entire website in its belly, ready to serve it to anyone who asks.
</p>
</div>
</section>
<!-- FEATURES SECTION -->
<section class="feature-section">
<div class="chapter-content" style="max-width: 100%;">
<h2 class="chapter-title" style="text-align: center; margin-bottom: 60px;">The Arsenal</h2>
<div class="feature-grid">
<div class="feature-card">
<h3>Async Domain Crawling</h3>
<p>Full-domain recursive crawling with configurable depth. Respects robots.txt and crawl delays. Polite but <em>relentless</em>.</p>
</div>
<div class="feature-card">
<h3>Content-Addressed Vault</h3>
<p>MD5-based deduplication in a 9-deep hex directory structure. Same image from 100 pages? Stored once. Space is memory.</p>
</div>
<div class="feature-card">
<h3>Full-Text Search</h3>
<p>Every image indexed with its surrounding context &mdash; page title, alt text, captions, nearby headings. Find images by <em>meaning</em>, not filename.</p>
</div>
<div class="feature-card">
<h3>Page Screenshots</h3>
<p><a href="https://linkpeek.com">LinkPeek</a> reborn! Full-page captures via <a href="https://github.com/russellballestrini/uri2png">uri2png</a>. wkhtmltoimage, cutycapt, or Playwright backends.</p>
</div>
<div class="feature-card">
<h3>Hydra Mode</h3>
<p>Auto-discover RSS/Atom/Sitemap feeds. Persists feed URLs, checks for new content on every crawl. Self-healing archive maintenance.</p>
</div>
<div class="feature-card">
<h3>Media Provenance</h3>
<p>Track first_seen, last_seen, and when each page first linked to media. Watch content spread across the web over time.</p>
</div>
<div class="feature-card">
<h3>Markdown Conversion</h3>
<p>Intelligent HTML-to-Markdown with forum post detection, noise removal, and content extraction. Human-readable archives.</p>
</div>
<div class="feature-card">
<h3>Self-Extracting Archives</h3>
<p>Create <code>.run</code> executables that contain entire websites. Drop on any machine, execute, browse. The beast travels light.</p>
</div>
<div class="feature-card">
<h3>Code File Support</h3>
<p>Archive source code, stylesheets, fonts, and scripts alongside media. Complete preservation for developer communities.</p>
</div>
</div>
</div>
</section>
<h2>The Skeleton Key Approach</h2>
<!-- CHAPTER VI: THE SKELETON KEY -->
<section class="chapter">
<div class="chapter-number">VI</div>
<div class="chapter-content">
<h2 class="chapter-title">The Skeleton Key</h2>
<p>neopig uses what we call the "skeleton key" approach to media indexing. For every image, we capture <em>all</em> the text that might help you find it later:</p>
<p class="narrative">
neopig uses what we call the <strong>"skeleton key"</strong> approach to media indexing. For every image, we capture <em>all</em> the text that might help you find it later:
</p>
<ul>
<li><strong>Page context:</strong> Title, description, headings near the image</li>
<li><strong>Image attributes:</strong> Alt text, title, surrounding captions</li>
<li><strong>Link context:</strong> The text of links pointing to the image</li>
<li><strong>Detail pages:</strong> For gallery sites, we follow through to detail pages</li>
</ul>
<p class="narrative">
<strong>Page context:</strong> Title, description, headings near the image.<br>
<strong>Image attributes:</strong> Alt text, title, surrounding captions.<br>
<strong>Link context:</strong> The text of links pointing to the image.<br>
<strong>Detail pages:</strong> For gallery sites, we follow through to detail pages and harvest their metadata too.
</p>
<p>The result? You can search for "sunset over mountains" and find that image even if it was named <code>IMG_4372.jpg</code> with no alt text &mdash; because the page title mentioned it, or someone linked to it with descriptive text.</p>
<p class="narrative">
The result? You can search for <em>"sunset over mountains"</em> and find that image even if it was named <code>IMG_4372.jpg</code> with no alt text &mdash; because the page title mentioned it, or someone linked to it with descriptive text, or a caption three divs away contained the words.
</p>
<h2>Preserving the Web</h2>
<p class="narrative">
Every door opens. Every search succeeds. The skeleton key fits every lock because it carries <em>every possible key</em> inside it.
</p>
</div>
</section>
<p>The web is ephemeral. Sites go dark. Forums shut down. Communities scatter. neopig is built for preservation &mdash; capturing not just the media, but the <em>context</em> that gives it meaning.</p>
<!-- CHAPTER VII: PRESERVATION -->
<section class="chapter chapter-dark">
<div class="chapter-number">VII</div>
<div class="chapter-content">
<h2 class="chapter-title">Why We Archive</h2>
<p>When you archive a site with neopig, you get:</p>
<p class="narrative">
<strong>The web is ephemeral.</strong> Sites go dark. Forums shut down. Communities scatter. Platforms get acquired and gutted. Executives decide that Mercurial isn't profitable and delete millions of repositories with a single corporate memo.
</p>
<ul>
<li>Original HTML with rewritten media links pointing to your local vault</li>
<li>Full-text searchable markdown versions of every page</li>
<li>Screenshots showing exactly how pages looked</li>
<li>A SQLite database you can query, backup, and migrate</li>
<li>Self-extracting archives that work offline forever</li>
</ul>
<p class="narrative">
We have watched beloved communities vanish. Game modding forums. Technical documentation. Art galleries. Fan wikis. Personal blogs with decades of writing. Gone. Not archived by the Wayback Machine. Not cached by Google. Just <em>gone</em>.
</p>
<h2>The Name</h2>
<p class="narrative">
neopig is built for <strong>preservation</strong> &mdash; capturing not just the media, but the <em>context</em> that gives it meaning. When you archive a site with neopig, you get:
</p>
<p><strong>neopig</strong> = <strong>Neo</strong> (new) + <strong>P</strong>ython <strong>I</strong>mage <strong>G</strong>rabber</p>
<p class="narrative">
&bull; Original HTML with rewritten media links pointing to your local vault<br>
&bull; Full-text searchable markdown versions of every page<br>
&bull; Screenshots showing exactly how pages looked<br>
&bull; A SQLite database you can query, backup, and migrate forever<br>
&bull; Self-extracting archives that work offline until the heat death of the universe
</p>
</div>
</section>
<p>A tip of the hat to the original pig.py, with a nod to the Matrix's Neo &mdash; seeing through the surface of the web to the underlying content within.</p>
<!-- CHAPTER VIII: THE FORGE -->
<section class="chapter chapter-forge">
<div class="chapter-number">VIII</div>
<div class="chapter-content">
<h2 class="chapter-title">The Forge</h2>
<h2>Get Started</h2>
<p class="narrative">
This tool was not built for profit. It was <strong>forged</strong>.
</p>
<pre><code># Archive a site and serve it live as you crawl
python neopig.py https://discourse-urho3d.github.io --archive --serve
<p class="narrative">
Forged in the fires of <a href="https://permacomputer.com" target="_blank">permacomputer.com</a> &mdash; the belief that computing should be <em>permanent</em>. That knowledge should not evaporate when a company pivots. That communities should not lose their histories when a platform gets acquired. That the digital commons belongs to <em>everyone</em>, forever.
</p>
# Fast mode for static sites (no crawl delay)
python neopig.py https://russell.ballestrini.net --archive --fast -o ./archives/
<div class="dramatic-quote">
<p>Truth. Harmony. Freedom. Love.</p>
<cite>&mdash; The four pillars of the permacomputer</cite>
</div>
# Hydra mode: discover RSS/Atom/Sitemap feeds for auto-updates
<p class="narrative">
<strong>Truth</strong> &mdash; We preserve what actually existed, not sanitized versions. The original HTML. The exact timestamps. The unedited conversations. History as it happened, not as someone wishes it had.
</p>
<p class="narrative">
<strong>Harmony</strong> &mdash; Every piece connects. Media links to pages. Pages link to context. Context links to meaning. The archive is not a graveyard of disconnected files but a living web of relationships, searchable, traversable, <em>whole</em>.
</p>
<p class="narrative">
<strong>Freedom</strong> &mdash; Public domain. No licenses to parse. No corporate terms of service. Take it. Fork it. Run it. Share it. The code is yours. The archives you create are yours. No one can revoke your access to your own preserved knowledge.
</p>
<p class="narrative">
<strong>Love</strong> &mdash; We archive because we <em>care</em>. About the communities that built these spaces. About the people who poured years into forum posts and wiki pages and blog entries. About the future researchers who will want to understand how we lived and thought and created. This is an act of love for human knowledge.
</p>
<p class="narrative">
The beast is imbued with <em>righteous fire</em>. It does not crawl politely because it fears the powerful &mdash; it respects robots.txt because it honors the wishes of creators. It does not hoard in darkness &mdash; it preserves in light, making archives searchable, shareable, <em>alive</em>.
</p>
<p class="narrative">
When the platforms fall (and they will fall), when the corporate servers go dark (and they will go dark), when the executives decide your community isn't profitable enough to maintain (and they will decide this) &mdash; the permacomputer endures. The archives persist. The knowledge survives.
</p>
<p class="narrative">
<strong>This is what we were forged to do.</strong>
</p>
</div>
</section>
<!-- CHAPTER IX: THE RULE -->
<section class="chapter">
<div class="chapter-number">IX</div>
<div class="chapter-content">
<h2 class="chapter-title">The Rule</h2>
<p class="narrative">
neopig is built on <a href="https://sqlite.org" target="_blank">SQLite</a> &mdash; the most deployed database in existence. SQLite adopted the <a href="https://sqlite.org/codeofethics.html" target="_blank">Rule of St. Benedict</a> as their Code of Ethics. 72 principles from a 6th-century monastery, governing software in the 21st.
</p>
<p class="narrative">
We inherit this lineage. All 72 rules, unabridged:
</p>
<div class="rule-grid">
<div class="rule-item"><strong>1.</strong> Love the Lord God with your whole heart, soul, and strength.</div>
<div class="rule-item"><strong>2.</strong> Love your neighbor as yourself.</div>
<div class="rule-item"><strong>3.</strong> Do not murder.</div>
<div class="rule-item"><strong>4.</strong> Do not commit adultery.</div>
<div class="rule-item"><strong>5.</strong> Do not steal.</div>
<div class="rule-item"><strong>6.</strong> Do not covet.</div>
<div class="rule-item"><strong>7.</strong> Do not bear false witness.</div>
<div class="rule-item"><strong>8.</strong> Honor all people.</div>
<div class="rule-item"><strong>9.</strong> Do not do to another what you would not have done to yourself.</div>
<div class="rule-item"><strong>10.</strong> Deny oneself in order to follow Christ.</div>
<div class="rule-item"><strong>11.</strong> Chastise the body.</div>
<div class="rule-item"><strong>12.</strong> Do not become attached to pleasures.</div>
<div class="rule-item"><strong>13.</strong> Love fasting.</div>
<div class="rule-item"><strong>14.</strong> Relieve the poor.</div>
<div class="rule-item"><strong>15.</strong> Clothe the naked.</div>
<div class="rule-item"><strong>16.</strong> Visit the sick.</div>
<div class="rule-item"><strong>17.</strong> Bury the dead.</div>
<div class="rule-item"><strong>18.</strong> Be a help in times of trouble.</div>
<div class="rule-item"><strong>19.</strong> Console the sorrowing.</div>
<div class="rule-item"><strong>20.</strong> Be a stranger to the world's ways.</div>
<div class="rule-item"><strong>21.</strong> Prefer nothing more than the love of Christ.</div>
<div class="rule-item"><strong>22.</strong> Do not give way to anger.</div>
<div class="rule-item"><strong>23.</strong> Do not nurse a grudge.</div>
<div class="rule-item"><strong>24.</strong> Do not entertain deceit in your heart.</div>
<div class="rule-item"><strong>25.</strong> Do not give a false peace.</div>
<div class="rule-item"><strong>26.</strong> Do not forsake charity.</div>
<div class="rule-item"><strong>27.</strong> Do not swear, for fear of perjuring yourself.</div>
<div class="rule-item"><strong>28.</strong> Utter only truth from heart and mouth.</div>
<div class="rule-item"><strong>29.</strong> Do not return evil for evil.</div>
<div class="rule-item"><strong>30.</strong> Do no wrong to anyone, and bear patiently wrongs done to yourself.</div>
<div class="rule-item"><strong>31.</strong> Love your enemies.</div>
<div class="rule-item"><strong>32.</strong> Do not curse those who curse you, but rather bless them.</div>
<div class="rule-item"><strong>33.</strong> Bear persecution for justice's sake.</div>
<div class="rule-item"><strong>34.</strong> Be not proud.</div>
<div class="rule-item"><strong>35.</strong> Be not addicted to wine.</div>
<div class="rule-item"><strong>36.</strong> Be not a great eater.</div>
<div class="rule-item"><strong>37.</strong> Be not drowsy.</div>
<div class="rule-item"><strong>38.</strong> Be not lazy.</div>
<div class="rule-item"><strong>39.</strong> Be not a grumbler.</div>
<div class="rule-item"><strong>40.</strong> Be not a detractor.</div>
<div class="rule-item"><strong>41.</strong> Put your hope in God.</div>
<div class="rule-item"><strong>42.</strong> Attribute to God, and not to self, whatever good you see in yourself.</div>
<div class="rule-item"><strong>43.</strong> Recognize always that evil is your own doing, and to impute it to yourself.</div>
<div class="rule-item"><strong>44.</strong> Fear the Day of Judgment.</div>
<div class="rule-item"><strong>45.</strong> Be in dread of hell.</div>
<div class="rule-item"><strong>46.</strong> Desire eternal life with all the passion of the spirit.</div>
<div class="rule-item"><strong>47.</strong> Keep death daily before your eyes.</div>
<div class="rule-item"><strong>48.</strong> Keep constant guard over the actions of your life.</div>
<div class="rule-item"><strong>49.</strong> Know for certain that God sees you everywhere.</div>
<div class="rule-item"><strong>50.</strong> When wrongful thoughts come into your heart, dash them against Christ immediately.</div>
<div class="rule-item"><strong>51.</strong> Disclose wrongful thoughts to your spiritual mentor.</div>
<div class="rule-item"><strong>52.</strong> Guard your tongue against evil and depraved speech.</div>
<div class="rule-item"><strong>53.</strong> Do not love much talking.</div>
<div class="rule-item"><strong>54.</strong> Speak no useless words or words that move to laughter.</div>
<div class="rule-item"><strong>55.</strong> Do not love much or boisterous laughter.</div>
<div class="rule-item"><strong>56.</strong> Listen willingly to holy reading.</div>
<div class="rule-item"><strong>57.</strong> Devote yourself frequently to prayer.</div>
<div class="rule-item"><strong>58.</strong> Daily in your prayers, with tears and sighs, confess your past sins to God, and amend them for the future.</div>
<div class="rule-item"><strong>59.</strong> Fulfill not the desires of the flesh; hate your own will.</div>
<div class="rule-item"><strong>60.</strong> Obey in all things the commands of those whom God has placed in authority over you even though they should act otherwise, mindful of the Lord's precept, "Do what they say, but not what they do."</div>
<div class="rule-item"><strong>61.</strong> Do not wish to be called holy before one is holy; but first to be holy, that you may be truly so called.</div>
<div class="rule-item"><strong>62.</strong> Fulfill God's commandments daily in your deeds.</div>
<div class="rule-item"><strong>63.</strong> Love chastity.</div>
<div class="rule-item"><strong>64.</strong> Hate no one.</div>
<div class="rule-item"><strong>65.</strong> Be not jealous, nor harbor envy.</div>
<div class="rule-item"><strong>66.</strong> Do not love quarreling.</div>
<div class="rule-item"><strong>67.</strong> Shun arrogance.</div>
<div class="rule-item"><strong>68.</strong> Respect your seniors.</div>
<div class="rule-item"><strong>69.</strong> Love your juniors.</div>
<div class="rule-item"><strong>70.</strong> Pray for your enemies in the love of Christ.</div>
<div class="rule-item"><strong>71.</strong> Make peace with your adversary before the sun sets.</div>
<div class="rule-item"><strong>72.</strong> Never despair of God's mercy.</div>
</div>
<p class="narrative" style="margin-top: 60px;">
We archive not to hoard, but to <em>give</em>. We preserve not to control, but to <em>liberate</em>. The database that holds your memories runs on a foundation of ancient wisdom &mdash; and so do we.
</p>
</div>
</section>
<!-- CHAPTER X: INVOKE THE BEAST -->
<section class="chapter chapter-beast">
<div class="chapter-number">X</div>
<div class="chapter-content">
<h2 class="chapter-title">Invoke the Beast</h2>
<div class="code-section">
<pre><code># Archive a site and watch it live as the beast feeds
python archive.py https://discourse-urho3d.github.io/ --serve
# Fast mode for static sites (no crawl delay, for sites without robots.txt)
python archive.py https://russell.ballestrini.net --fast -o ./archives/
# Full domain slurp with unlimited depth
python neopig.py https://example.com --mode all --depth -1
# Hydra mode: discover and persist RSS/Atom/Sitemap feeds
python neopig.py https://example.com --hydra
# Re-run with fresh markdown conversion
python neopig.py --fresh --backfill-markdown example.com
# Create a self-extracting executable
make run TARBALL=example.com-20260101.tar.gz
# Purge a job and ALL its data (media, screenshots, pages)
python neopig.py --purge-job 123
# Run the beast (just needs python3 + pyramid)
./example.com-20260101.run
# Opens http://localhost:6543 with full search</code></pre>
</div>
# Upgrade an existing archive with latest neopig/serp
python neopig.py --upgrade-neopig ./archives/example.com-20251231.tar.gz</code></pre>
<p class="narrative">
<strong>neopig</strong> = <strong>Neo</strong> (new) + <strong>P</strong>ython <strong>I</strong>mage <strong>G</strong>rabber
</p>
<p style="margin-top: 40px; padding-top: 20px; border-top: 1px solid #333; color: #888; font-size: 0.9em;">
neopig is open source. The original pig.py was placed in the public domain by Russell Ballestrini.
This project continues that tradition of building useful tools for the community.
<p class="narrative">
A tip of the hat to the original pig.py. A nod to the Matrix's Neo &mdash; seeing through the surface of the web to the underlying content within. And perhaps a warning: this pig has grown teeth.
</p>
</div>
</section>
<!-- FOOTER -->
<footer class="about-footer">
<p>neopig is open source, donated into the public domain.</p>
<p>The beast lives at:</p>
<a href="https://git.unturf.com/engineering/unturf/pig.py" target="_blank">git.unturf.com/engineering/unturf/pig.py</a>
<p style="margin-top: 40px; color: #444; font-size: 0.9em;">
In memory of all the repositories Atlassian killed.<br>
We remember. We archive. We <em>persist</em>.
</p>
</div>
</footer>
</div>
"""

16
tests/unit/test_async_web_fetcher.py
View file

@ -140,10 +140,16 @@ class TestMediaTypeDetection:
assert get_media_type_from_extension("https://example.com/audio.wav") == "audio"
assert get_media_type_from_extension("https://example.com/audio.ogg") == "audio"
def test_get_media_type_from_extension_code(self):
"""Test detecting code from extension."""
assert get_media_type_from_extension("https://example.com/file.txt") == "code"
assert get_media_type_from_extension("https://example.com/script.py") == "code"
assert get_media_type_from_extension("https://example.com/app.js") == "code"
def test_get_media_type_from_extension_unknown(self):
"""Test unknown extension."""
assert get_media_type_from_extension("https://example.com/file.txt") is None
assert get_media_type_from_extension("https://example.com/page.html") is None
assert get_media_type_from_extension("https://example.com/file.xyz") is None
def test_get_media_type_from_mime_image(self):
"""Test detecting image from MIME type."""
@ -161,10 +167,16 @@ class TestMediaTypeDetection:
assert get_media_type_from_mime("audio/mpeg") == "audio"
assert get_media_type_from_mime("audio/wav") == "audio"
def test_get_media_type_from_mime_code(self):
"""Test detecting code from MIME type."""
assert get_media_type_from_mime("application/json") == "code"
assert get_media_type_from_mime("text/plain") == "code"
assert get_media_type_from_mime("application/javascript") == "code"
def test_get_media_type_from_mime_unknown(self):
"""Test unknown MIME type."""
assert get_media_type_from_mime("text/html") is None
assert get_media_type_from_mime("application/json") is None
assert get_media_type_from_mime("application/octet-stream") is None
class TestMediaExtraction: