engineering/lumbda
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

factory: heal_orphan_bins nullglob defect — use compgen for inflight check + reducer

Browse source 
shopt -s nullglob (set so outer for-loop tolerates empty *.bin) made
an unmatched .inflight-* glob expand to nothing. Bare "ls >/dev/null"
then succeeded by listing CWD, and our if-branch incorrectly skipped
every orphan whose .inflight-* did not match. Net effect: medium-sized
orphan bins (100 MiB+, valid QECCOPS1 magic, no markers) accumulated
in our queue indefinitely across pool restarts.

Replace with compgen -G which returns success only when our pattern
matches, immune to nullglob.

Adds tests/integration/test-heal-orphan-bins.sh as TCRAUDT reducer
covering our contract: medium-size orphan promotes to .ready,
sub-threshold truncated to .done, bad-magic DLQs to dlq/.

Incident 2026-06-14: 8 vecC-tri-*-w8c.bin orphans (493 MiB each, Jun 12
emit) survived multiple foxhop pool restarts. Live factory queue
visible to operator as a persistent ~8-bin floor that never drained.
This commit is contained in:
russell@unturf.com 2026-06-14 11:47:03 -04:00
parent 346b873247
commit 3e47814cf3
No known key found for this signature in database
2 changed files with 127 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

7
factory/bend-emit-pool.sh
View file

@ -197,7 +197,12 @@ heal_orphan_bins() {
[ -f "$QUEUE_DIR/${tag}.done" ] && continue
[ -f "$QUEUE_DIR/${tag}.emitting" ] && continue
[ -f "$QUEUE_DIR/${tag}.dispatch.log" ] && continue
if ls "$QUEUE_DIR/${tag}.inflight-"* >/dev/null 2>&1; then
# 2026-06-14 defect fix: with shopt -s nullglob (set at function top
# so the outer for loop tolerates empty *.bin), an unmatched
# .inflight-* glob expands to NOTHING. Bare "ls >/dev/null" then
# succeeds (lists CWD) + the if-branch incorrectly skips the orphan.
# Compgen returns success only when pattern matches, immune to nullglob.
if compgen -G "$QUEUE_DIR/${tag}.inflight-*" >/dev/null; then
continue
fi
sz=$(stat -c %s "$bin" 2>/dev/null || echo 0)