docs: document scanner improvement loop in CLAUDE.md

This commit is contained in:
russell@unturf.com 2026-04-12 17:43:44 -04:00
parent 0ac47a5b75
commit 89f094d242

View file

@ -123,6 +123,50 @@ qBittorrent, Transmission, Deluge, aria2.
**No stone unturned complete as of 2026-03-29.** All 320 defect directories have a patch or CLEAN marker. 629 confirmed sites, 571 UNDF assigned. Next wave: deeper scans on single-defect targets (allegro5, caddy, ceph, grafana, etc.) or clone new targets not yet in defects/.
## Test Harness and Scanner — How They Improve
**The scanner does not improve on its own.** It improves when we do real work:
1. **Find a new hit** — scan a target, find CWE-407 or other MOAD pattern
2. **Write the patch**`defects/{project}/patch/{slug}.patch` with `# UNDF: ...` header
3. **Write the tests** — unit in `tests/unit/`, integration in `tests/integration/`, functional in `tests/functional/`
4. **Add intel** — run `generate_undf.py`, push to undefect.com
5. **Update CLAUDE.md** — if a new defect pattern or fix signature was involved, add it to `tests/scan_verify.py` `DEFECTIVE_SIGS` / `FIX_SIGS` so that pattern promotes from WARN to PASS on future scans
**If a patch shows WARN in `make scan-verify`:**
- Check whether its defective pattern needs a new entry in `DEFECTIVE_SIGS`
- Check whether its fix pattern needs a new entry in `FIX_SIGS`
- Add the pattern, re-run, confirm it promotes to PASS
- Commit the pattern addition alongside the new patch/test
**The three feedback loops:**
```
New hit found
→ patch written
→ tests pass
→ scan_verify.py PASS
→ UNDF post on undefect.com
→ scanner coverage grows by 1
WARN entry spotted
→ read the patch to find the unrecognized pattern
→ add pattern to DEFECTIVE_SIGS or FIX_SIGS
→ re-run make scan-verify, confirm PASS
→ scanner improves for all future patches using that pattern
NO_PATCH entry seen
→ write the missing patch for that defect dir
→ coverage-check passes for that entry
→ scanner coverage grows
```
**Current state (2026-04-12):**
- 1258 UNDF entries in registry
- 454 PASS, 441 WARN, 12 FAIL, 300 NO_PATCH, 51 PENDING
- FAIL = real stub patches needing actual diff content (thunderbird 1-6, systemd-0003, etc.)
- NO_PATCH = project dir exists, patch file not yet written
- WARN = patch present but defective/fix pattern not in our lists yet
## UNDF Numbering System
Every defect in this repo gets a **UNDF-2026-XXXXXXXXX** identifier (9-digit, covers 999,999,999 entries).