russellballestrini/russell.ballestrini.net
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

update postmortem with salt state details, update CLAUDE.md

Browse source 
postmortem now documents permanent fixes via salt states,
added salt states section with top.sls targeting. CLAUDE.md
updated: build server docs, SVG animation pattern changed
from scroll-driven to click-to-play, post-processing checklist
updated.
This commit is contained in:
russell@unturf.com 2026-03-07 22:16:03 -05:00
parent 949bfaacc2
commit 524542c33a
2 changed files with 69 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

34
content/2026-03-08-build-server-postmortem-disk-full.rst
View file

@ -41,7 +41,7 @@ Five independent failures conspired. Each alone stays survivable. Together they
``/etc/gitlab-runner/config.toml`` set ``concurrent = 64``. A shell executor creates a full git checkout per slot per project. 64 slots across multiple repos meant 64 copies of every codebase. The ``unsandbox-all-upgradable`` repo alone stored 3.7G FreeBSD & 710M OpenBSD qcow2 images per slot. Slot 15 alone consumed 4.3G.
**Fix applied:** ``concurrent = 4``. Matches actual workload. Stale slots 4-63 removed.
**Fix applied:** ``concurrent = 4``. Matches actual workload. Stale slots 4-63 removed. Enforced permanently via salt state ``gitlab.build-host.ubuntu`` (pillar-configurable: ``gitlab-runner:concurrent``).
2. LXD images never expired
-----------------------------
@ -50,7 +50,7 @@ CI pipelines launch LXD containers for multi-distro testing (Alpine, Arch, Debia
``images.remote_cache_expiry`` defaulted to 10 days but never cleaned up because ``images.auto_update_interval`` kept refreshing them. No pruning mechanism existed.
**Fix applied:** ``images.remote_cache_expiry = 3``, ``images.auto_update_interval = 0``. Weekly cron prunes unused images.
**Fix applied:** ``images.remote_cache_expiry = 3``, ``images.auto_update_interval = 0``. Weekly cron prunes unused images. Enforced permanently via salt state ``lxd.build-host``.
3. orphaned LXD containers
---------------------------
@ -104,6 +104,34 @@ Installed at ``/etc/cron.d/build-cleanup``:
|
salt states (permanent fixes)
==============================
Manual hotfixes on a server vanish on reprovision. Every fix got codified into salt states in ``foxhop-states`` so a ``salt-call state.highstate`` reproduces them.
``gitlab/build-host/ubuntu.sls``:
- Enforces ``concurrent`` in ``config.toml`` via sed (default 4, configurable via pillar ``gitlab-runner:concurrent``)
- Manages ``/etc/cron.d/build-cleanup`` with all five cleanup jobs
- Uses ``/snap/bin/lxc`` full paths (snap-installed LXD)
- Disk alerts go to syslog via ``logger -t disk-alert`` instead of a file
``lxd/build-host.sls``:
- Sets ``images.remote_cache_expiry = 3`` (days)
- Sets ``images.auto_update_interval = 0`` (CI pulls fresh images on demand)
- Both idempotent with ``unless`` guards
``top.sls`` already targets ``build.unturf.com`` with both states:
.. code-block:: yaml
'build.unturf.com':
- gitlab.build-host.ubuntu
- lxd.build-host
|
disk recovery
==============
@ -132,3 +160,5 @@ lessons
**absence of signal stays a signal.** No disk alert fired because no disk alert existed. Silence in monitoring always means one of two things: everything works, or nothing watches. Assume the second until proven otherwise.
**build servers need janitors.** CI systems produce waste: cached images, stopped containers, orphaned processes, stale checkouts. Without automated cleanup, waste accumulates until something breaks. The cron job costs nothing. The outage costs a pipeline.
**manual fixes rot. salt states persist.** Every fix applied manually on the build server got codified into salt states within the same session. ``gitlab/build-host/ubuntu.sls`` manages concurrent limits & the cleanup cron. ``lxd/build-host.sls`` manages image cache expiry. A highstate reproduces the fix. A reprovision preserves it. Manual hotfixes buy time. Configuration management buys permanence.