engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: bench/load_monitor.py — stdlib request-load monitor for single-slot endpoints

Browse source 
Answers fox's 2026-05-21 question — 'are we being swamped because we're
open to internet?' — before the multi-day GPU bench, where uncontrolled
internet traffic on the single-slot hermes (3090/vLLM) and qwen
(4090/llama.cpp) endpoints would contaminate wattage + throughput.

Three stdlib subcommands (urllib + sqlite3 + hand-rolled SVG, nothing to
install):
  poll   — scrape each endpoint's Prometheus /metrics on an interval into
           SQLite; queue depth (num_requests_waiting / requests_deferred)
           is the swamp signal a single GPU slot exposes. Prunes past
           --retention-days each cycle (bounded store, no cancer growth).
  graph  — multi-panel SVG: queue depth, running, req/s, tok/s, e2e latency.
  access — parse Caddy/nginx access log for the real client IPs the backend
           can't see behind the proxy hop; top talkers + per-IP rate SVG.

Backend /metrics = HOW MUCH; proxy log = WHO. make monitor-poll /
monitor-graph / monitor-access.
This commit is contained in:
russell@unturf.com 2026-05-21 09:20:53 -04:00
parent 9d0015b4d4
commit b5cc970a86
No known key found for this signature in database
2 changed files with 599 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

31
Makefile
View file

@ -39,7 +39,8 @@ SEARCH_Q ?= computer
bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \
bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab control-sweep rapl-access rapl-access-revoke clean clean-db clean-data help \
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
crawl-textbooks crawl-textbooks-stats textbook textbook-list bench-jaggedness
crawl-textbooks crawl-textbooks-stats textbook textbook-list bench-jaggedness \
monitor-poll monitor-graph monitor-access
all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
@ -276,6 +277,34 @@ bench-jaggedness: bootstrap ## #000060: deterministic retrieval jaggedness acros
--conc $(JAGGED_CONC) \
--shards-dir $(SHARDS_DIR)
# Request-load monitor for the single-slot LLM endpoints (fox 2026-05-21:
# "are we being swamped because we're open to internet?"). Stdlib-only
# (urllib + sqlite3 + hand-rolled SVG) — no bootstrap, no venv. The backend
# /metrics answers HOW MUCH (queue depth = the swamp signal); the proxy
# access log answers WHO (real client IPs the backend can't see behind the
# Caddy hop). See bench/load_monitor.py.
# make monitor-poll # poll forever, Ctrl-C to stop
# make monitor-graph MONITOR_HOURS=6 # render last 6h to SVG
# make monitor-access LOG=/var/log/caddy/access.log
MONITOR_DB ?= $(HOME)/.arborist/load_monitor.db
MONITOR_INTERVAL ?= 10
MONITOR_HOURS ?= 6
MONITOR_TOP ?= 20
# GPU sampling: ssh nvidia-smi on the box behind each endpoint so power-draw +
# util graph alongside request rate (the 100%-power-cap saturation signal).
MONITOR_GPU ?= hermes-3090=3090-ai.foxhop.net
monitor-poll: ## poll LLM endpoint /metrics (+GPU power/util) into SQLite [MONITOR_INTERVAL=10 MONITOR_GPU=name=host]
PYTHONUNBUFFERED=1 python3 bench/load_monitor.py --db $(MONITOR_DB) \
poll --interval $(MONITOR_INTERVAL) --gpu $(MONITOR_GPU)
monitor-graph: ## render stored load samples to SVG [MONITOR_HOURS=6]
python3 bench/load_monitor.py --db $(MONITOR_DB) \
graph --since-hours $(MONITOR_HOURS)
monitor-access: ## proxy access-log real-client-IP tally + SVG: make monitor-access LOG=path
python3 bench/load_monitor.py access $(LOG) --top $(MONITOR_TOP) \
--out bench/results/proxy_access.svg
# Progressive-AND / DF-filter fixture: 9 questions chosen to exercise
# the OR-fallback and progressive-AND drop paths. Use this for any
# retrieval-side A/B (alternative search backends, synonym/rerank