diff --git a/CLAUDE.md b/CLAUDE.md index 80e4758..104da61 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -251,24 +251,23 @@ venv/bin/python lib/tag_stats.py 5 # only tags with 5+ posts - Root SVG: `width="100%" height="100%"` with `viewBox` (never fixed pt/px) - RST embed: `:width: 100%` and `:align: center` -### Animation Pattern (CSS only, no JS) -```xml - - - -``` +### Animation Pattern (click-to-play) + +SVGs render fully visible by default. A play button overlay triggers sequential fadeIn animation on click. + +**SVG markup:** No `` block (animation handled by page-level JS) +6. Verify: no `fill="black"`, no `stroke="black"`, no `fill="#333333"` ## RUSSELL'S PROJECT TECH STACKS @@ -307,6 +307,24 @@ edge [color="#AAAAAA" arrowsize=0.6 fontcolor="#AAAAAA"] - make_post_sell & remarkbox are **Python/Pyramid**, NOT Django - uncloseai.com is **Node.js**, NOT Python +## BUILD SERVER (build.unturf.com) + +**CI/CD for this blog & all unturf projects.** GitLab Runner with shell executor. + +**Config management:** Salt states in `~/git/foxhop-states/` +- `gitlab/build-host/ubuntu.sls` — runner config, cleanup cron, build deps +- `lxd/build-host.sls` — LXD + ZFS, image cache expiry +- `top.sls` targets `build.unturf.com` with both states + +**Key settings (salt-managed):** +- `concurrent = 4` in `/etc/gitlab-runner/config.toml` (pillar: `gitlab-runner:concurrent`) +- LXD `images.remote_cache_expiry = 3` days, `images.auto_update_interval = 0` +- `/etc/cron.d/build-cleanup` — hourly container + zombie cleanup, weekly image prune, disk alerts at 85% + +**Postmortem:** `russell.ballestrini.net/build-server-postmortem-disk-full/` (2026-03-08) + +**Pipeline for this repo:** `make venv && make html && make formats && make resume` → tar → deploy via `deploy-www.sh` + ## DATE HANDLING FOR NEW CONTENT **CRITICAL: ALWAYS check the current date BEFORE creating ANY new documents** diff --git a/content/2026-03-08-build-server-postmortem-disk-full.rst b/content/2026-03-08-build-server-postmortem-disk-full.rst index cfe005a..8709744 100644 --- a/content/2026-03-08-build-server-postmortem-disk-full.rst +++ b/content/2026-03-08-build-server-postmortem-disk-full.rst @@ -41,7 +41,7 @@ Five independent failures conspired. Each alone stays survivable. Together they ``/etc/gitlab-runner/config.toml`` set ``concurrent = 64``. A shell executor creates a full git checkout per slot per project. 64 slots across multiple repos meant 64 copies of every codebase. The ``unsandbox-all-upgradable`` repo alone stored 3.7G FreeBSD & 710M OpenBSD qcow2 images per slot. Slot 15 alone consumed 4.3G. -**Fix applied:** ``concurrent = 4``. Matches actual workload. Stale slots 4-63 removed. +**Fix applied:** ``concurrent = 4``. Matches actual workload. Stale slots 4-63 removed. Enforced permanently via salt state ``gitlab.build-host.ubuntu`` (pillar-configurable: ``gitlab-runner:concurrent``). 2. LXD images never expired ----------------------------- @@ -50,7 +50,7 @@ CI pipelines launch LXD containers for multi-distro testing (Alpine, Arch, Debia ``images.remote_cache_expiry`` defaulted to 10 days but never cleaned up because ``images.auto_update_interval`` kept refreshing them. No pruning mechanism existed. -**Fix applied:** ``images.remote_cache_expiry = 3``, ``images.auto_update_interval = 0``. Weekly cron prunes unused images. +**Fix applied:** ``images.remote_cache_expiry = 3``, ``images.auto_update_interval = 0``. Weekly cron prunes unused images. Enforced permanently via salt state ``lxd.build-host``. 3. orphaned LXD containers --------------------------- @@ -104,6 +104,34 @@ Installed at ``/etc/cron.d/build-cleanup``: | +salt states (permanent fixes) +============================== + +Manual hotfixes on a server vanish on reprovision. Every fix got codified into salt states in ``foxhop-states`` so a ``salt-call state.highstate`` reproduces them. + +``gitlab/build-host/ubuntu.sls``: + +- Enforces ``concurrent`` in ``config.toml`` via sed (default 4, configurable via pillar ``gitlab-runner:concurrent``) +- Manages ``/etc/cron.d/build-cleanup`` with all five cleanup jobs +- Uses ``/snap/bin/lxc`` full paths (snap-installed LXD) +- Disk alerts go to syslog via ``logger -t disk-alert`` instead of a file + +``lxd/build-host.sls``: + +- Sets ``images.remote_cache_expiry = 3`` (days) +- Sets ``images.auto_update_interval = 0`` (CI pulls fresh images on demand) +- Both idempotent with ``unless`` guards + +``top.sls`` already targets ``build.unturf.com`` with both states: + +.. code-block:: yaml + + 'build.unturf.com': + - gitlab.build-host.ubuntu + - lxd.build-host + +| + disk recovery ============== @@ -132,3 +160,5 @@ lessons **absence of signal stays a signal.** No disk alert fired because no disk alert existed. Silence in monitoring always means one of two things: everything works, or nothing watches. Assume the second until proven otherwise. **build servers need janitors.** CI systems produce waste: cached images, stopped containers, orphaned processes, stale checkouts. Without automated cleanup, waste accumulates until something breaks. The cron job costs nothing. The outage costs a pipeline. + +**manual fixes rot. salt states persist.** Every fix applied manually on the build server got codified into salt states within the same session. ``gitlab/build-host/ubuntu.sls`` manages concurrent limits & the cleanup cron. ``lxd/build-host.sls`` manages image cache expiry. A highstate reproduces the fix. A reprovision preserves it. Manual hotfixes buy time. Configuration management buys permanence.