update postmortem with salt state details, update CLAUDE.md
postmortem now documents permanent fixes via salt states, added salt states section with top.sls targeting. CLAUDE.md updated: build server docs, SVG animation pattern changed from scroll-driven to click-to-play, post-processing checklist updated.
This commit is contained in:
parent
949bfaacc2
commit
524542c33a
2 changed files with 69 additions and 21 deletions
56
CLAUDE.md
56
CLAUDE.md
|
|
@ -251,24 +251,23 @@ venv/bin/python lib/tag_stats.py 5 # only tags with 5+ posts
|
|||
- Root SVG: `width="100%" height="100%"` with `viewBox` (never fixed pt/px)
|
||||
- RST embed: `:width: 100%` and `:align: center`
|
||||
|
||||
### Animation Pattern (CSS only, no JS)
|
||||
```xml
|
||||
<defs>
|
||||
<style type="text/css">
|
||||
@keyframes fadeIn {
|
||||
from { opacity: 0; }
|
||||
to { opacity: 1; }
|
||||
}
|
||||
.anim { opacity: 0; animation: fadeIn 0.8s ease-out forwards; }
|
||||
.d0 { animation-delay: 0s; }
|
||||
.d1 { animation-delay: 3s; }
|
||||
/* ... increment by 2-3s per group */
|
||||
</style>
|
||||
</defs>
|
||||
```
|
||||
### Animation Pattern (click-to-play)
|
||||
|
||||
SVGs render fully visible by default. A play button overlay triggers sequential fadeIn animation on click.
|
||||
|
||||
**SVG markup:** No `<style>` block needed. Just wrap groups in `<g class="anim dN">` where N controls sequence order.
|
||||
|
||||
**JS controller** (added once per page via `raw:: html`):
|
||||
- Finds all SVGs with `.anim` elements, wraps each in a container, adds play button
|
||||
- On click: hides all `.anim` groups (opacity 0), then animates them in sequence using `@keyframes svgFadeIn` with calculated delays based on `dN` class values
|
||||
- Total animation spread scales to ~15s regardless of group count (`step = 15 / (maxD + 1)`)
|
||||
- After animation completes, restores full opacity & shows play button again
|
||||
|
||||
**Previous approach (scroll-driven CSS view-timeline) was removed** because browser support proved unreliable. See royal-we post for the working click-to-play implementation.
|
||||
|
||||
- Wrap each conceptual group in `<g class="anim dN">`
|
||||
- Groups fade in sequentially (3s apart for major concepts, 1-2s for chain elements)
|
||||
- Keep total animation under 30s
|
||||
- N values control sequence order (d0 appears first, d1 second, etc.)
|
||||
- Multiple groups can share the same dN class to appear simultaneously
|
||||
|
||||
### Graphviz Dot Conventions
|
||||
```dot
|
||||
|
|
@ -281,8 +280,9 @@ edge [color="#AAAAAA" arrowsize=0.6 fontcolor="#AAAAAA"]
|
|||
1. Strip `<title>` tags: `sed -i 's/<title>[^<]*<\/title>//g'` (prevents tooltip artifacts)
|
||||
2. Replace fixed dimensions: `width="100%" height="100%"` keeping viewBox
|
||||
3. Change cluster label fontcolor from `#aaaaaa` to `#cccccc`
|
||||
4. Wrap groups with animation classes
|
||||
5. Verify: no `fill="black"`, no `stroke="black"`, no `fill="#333333"`
|
||||
4. Wrap groups with animation classes (`<g class="anim dN">`)
|
||||
5. Remove `<defs><style>...</style></defs>` block (animation handled by page-level JS)
|
||||
6. Verify: no `fill="black"`, no `stroke="black"`, no `fill="#333333"`
|
||||
|
||||
## RUSSELL'S PROJECT TECH STACKS
|
||||
|
||||
|
|
@ -307,6 +307,24 @@ edge [color="#AAAAAA" arrowsize=0.6 fontcolor="#AAAAAA"]
|
|||
- make_post_sell & remarkbox are **Python/Pyramid**, NOT Django
|
||||
- uncloseai.com is **Node.js**, NOT Python
|
||||
|
||||
## BUILD SERVER (build.unturf.com)
|
||||
|
||||
**CI/CD for this blog & all unturf projects.** GitLab Runner with shell executor.
|
||||
|
||||
**Config management:** Salt states in `~/git/foxhop-states/`
|
||||
- `gitlab/build-host/ubuntu.sls` — runner config, cleanup cron, build deps
|
||||
- `lxd/build-host.sls` — LXD + ZFS, image cache expiry
|
||||
- `top.sls` targets `build.unturf.com` with both states
|
||||
|
||||
**Key settings (salt-managed):**
|
||||
- `concurrent = 4` in `/etc/gitlab-runner/config.toml` (pillar: `gitlab-runner:concurrent`)
|
||||
- LXD `images.remote_cache_expiry = 3` days, `images.auto_update_interval = 0`
|
||||
- `/etc/cron.d/build-cleanup` — hourly container + zombie cleanup, weekly image prune, disk alerts at 85%
|
||||
|
||||
**Postmortem:** `russell.ballestrini.net/build-server-postmortem-disk-full/` (2026-03-08)
|
||||
|
||||
**Pipeline for this repo:** `make venv && make html && make formats && make resume` → tar → deploy via `deploy-www.sh`
|
||||
|
||||
## DATE HANDLING FOR NEW CONTENT
|
||||
|
||||
**CRITICAL: ALWAYS check the current date BEFORE creating ANY new documents**
|
||||
|
|
|
|||
|
|
@ -41,7 +41,7 @@ Five independent failures conspired. Each alone stays survivable. Together they
|
|||
|
||||
``/etc/gitlab-runner/config.toml`` set ``concurrent = 64``. A shell executor creates a full git checkout per slot per project. 64 slots across multiple repos meant 64 copies of every codebase. The ``unsandbox-all-upgradable`` repo alone stored 3.7G FreeBSD & 710M OpenBSD qcow2 images per slot. Slot 15 alone consumed 4.3G.
|
||||
|
||||
**Fix applied:** ``concurrent = 4``. Matches actual workload. Stale slots 4-63 removed.
|
||||
**Fix applied:** ``concurrent = 4``. Matches actual workload. Stale slots 4-63 removed. Enforced permanently via salt state ``gitlab.build-host.ubuntu`` (pillar-configurable: ``gitlab-runner:concurrent``).
|
||||
|
||||
2. LXD images never expired
|
||||
-----------------------------
|
||||
|
|
@ -50,7 +50,7 @@ CI pipelines launch LXD containers for multi-distro testing (Alpine, Arch, Debia
|
|||
|
||||
``images.remote_cache_expiry`` defaulted to 10 days but never cleaned up because ``images.auto_update_interval`` kept refreshing them. No pruning mechanism existed.
|
||||
|
||||
**Fix applied:** ``images.remote_cache_expiry = 3``, ``images.auto_update_interval = 0``. Weekly cron prunes unused images.
|
||||
**Fix applied:** ``images.remote_cache_expiry = 3``, ``images.auto_update_interval = 0``. Weekly cron prunes unused images. Enforced permanently via salt state ``lxd.build-host``.
|
||||
|
||||
3. orphaned LXD containers
|
||||
---------------------------
|
||||
|
|
@ -104,6 +104,34 @@ Installed at ``/etc/cron.d/build-cleanup``:
|
|||
|
||||
|
|
||||
|
||||
salt states (permanent fixes)
|
||||
==============================
|
||||
|
||||
Manual hotfixes on a server vanish on reprovision. Every fix got codified into salt states in ``foxhop-states`` so a ``salt-call state.highstate`` reproduces them.
|
||||
|
||||
``gitlab/build-host/ubuntu.sls``:
|
||||
|
||||
- Enforces ``concurrent`` in ``config.toml`` via sed (default 4, configurable via pillar ``gitlab-runner:concurrent``)
|
||||
- Manages ``/etc/cron.d/build-cleanup`` with all five cleanup jobs
|
||||
- Uses ``/snap/bin/lxc`` full paths (snap-installed LXD)
|
||||
- Disk alerts go to syslog via ``logger -t disk-alert`` instead of a file
|
||||
|
||||
``lxd/build-host.sls``:
|
||||
|
||||
- Sets ``images.remote_cache_expiry = 3`` (days)
|
||||
- Sets ``images.auto_update_interval = 0`` (CI pulls fresh images on demand)
|
||||
- Both idempotent with ``unless`` guards
|
||||
|
||||
``top.sls`` already targets ``build.unturf.com`` with both states:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
'build.unturf.com':
|
||||
- gitlab.build-host.ubuntu
|
||||
- lxd.build-host
|
||||
|
||||
|
|
||||
|
||||
disk recovery
|
||||
==============
|
||||
|
||||
|
|
@ -132,3 +160,5 @@ lessons
|
|||
**absence of signal stays a signal.** No disk alert fired because no disk alert existed. Silence in monitoring always means one of two things: everything works, or nothing watches. Assume the second until proven otherwise.
|
||||
|
||||
**build servers need janitors.** CI systems produce waste: cached images, stopped containers, orphaned processes, stale checkouts. Without automated cleanup, waste accumulates until something breaks. The cron job costs nothing. The outage costs a pipeline.
|
||||
|
||||
**manual fixes rot. salt states persist.** Every fix applied manually on the build server got codified into salt states within the same session. ``gitlab/build-host/ubuntu.sls`` manages concurrent limits & the cleanup cron. ``lxd/build-host.sls`` manages image cache expiry. A highstate reproduces the fix. A reprovision preserves it. Manual hotfixes buy time. Configuration management buys permanence.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue