From e5ffa6c6fdeca80274b4dd22b4b76abdbd7dd15f Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sat, 2 May 2026 14:27:24 -0400 Subject: [PATCH] qa(inspect): wordlist union (US+UK) + configurable supplemental dictionaries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes that shape the same lever: (1) The metaphor-cue wordlist now unions /usr/share/dict/words + /usr/share/dict/american-english + /usr/share/dict/british-english. The Debian split made the prior 'just symlink to american-english' miss British spellings (colour, organisation, realise) which silently became false negatives on British-speaker questions. Union: 102,485 → 104,305 entries on this machine. ~1,820 added British-specific entries. (2) Supplemental dictionary support: operators can layer domain-specific vocabulary into the morphological substrate. Two paths: - Env var: ABORIST_METAPHOR_DICTS=/path/a:/path/b - Programmatic: register_metaphor_dictionary(path) Each supplemental dict is one word per line. The cue suffix tests (-ly stem, -ing stem, -est stem) then resolve domain stems automatically — adding 'aerodynamic' to a custom dict makes 'aerodynamically' classify as adverbial without code changes. Use case: 'a tree with its own vocabulary' — an aviation forest, a medical corpus, a legal-domain shard each carries jargon the standard wordlist doesn't cover. Register once, suffix tests pick up domain stems forever. (3) README gains a 'Sidecar diagnostics' section with a table of the three sidecars (deflection, title-relevance, metaphor- deflection) plus a 'Metaphor-deflection cue dictionary' subsection explaining the derivation rule, the load order, and the per- forest vocabulary configurability. Architecturally documents why the rule is *derived* from the union (Phase-2 lesson) and not hand-curated. 3 new tests in tests/test_inspect.py: - register_metaphor_dictionary unions a custom path's words - ABORIST_METAPHOR_DICTS env var supplements with two paths - re-registering same path is idempotent 763/34 tests pass. --- README.md | 40 +++++++++++++++++++ aborist/qa/inspect.py | 90 ++++++++++++++++++++++++++++++++++++------- tests/test_inspect.py | 80 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 196 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 19765c7..dcdb80f 100644 --- a/README.md +++ b/README.md @@ -502,6 +502,46 @@ The default suite never hits the network. The crawler suite is gated behind `mak `make bench-qa` writes JSONL + markdown into `bench/qa_results/.{jsonl,md}` (gitignored); design-log entries live in `docs/qa-modes-bench-.md`. The bench is stop/start-able via `--resume ` (same `--seed` required for shuffled-task-order alignment). `BENCH_QA_CONCURRENCY=N` Makefile variable overrides the default `4`. See `docs/bench-maxing.md` for the full speed playbook. +## Sidecar diagnostics + +The verifier stays binary; soft signals proliferate as **sidecars** — read-only diagnostic functions that pull the same source chunks the verifier saw, classify spans / detect topic shifts / surface metaphor framing, and never write to `providence_cache` or extend the audit chain. Three live today: + +| sidecar | flags | typical use | +|---------|-------|-------------| +| `diagnose_deflection(question, answer)` | model deflected — answer omits the question's subject anchor | adversarial-premise questions where the model answers an adjacent grounded question | +| `diagnose_title_relevance(claim, cited_titles)` | claim's stems share zero overlap with cited source titles | retrieval-driven hallucinations where a chunk's source is structurally unrelated to the claim | +| `diagnose_metaphor_deflection(question, answer)` | question carried strong poetic framing the answer ignored | metaphorical questions answered with literal taxonomic / definitional facts | + +Run the per-record diagnostic via `aborist inspect --cache-key ` (CLI: `make inspect KEY=...`). Sidecars run on demand, observe state, report — they participate in no proof path. + +### Metaphor-deflection cue dictionary + +`diagnose_metaphor_deflection` derives its poetic-cue vocabulary from a configurable union of wordlists. The default load order: + +1. `/usr/share/dict/words` (OS default) +2. `/usr/share/dict/american-english` (Debian split, if present) +3. `/usr/share/dict/british-english` (Debian split, if present) +4. Paths in `ABORIST_METAPHOR_DICTS` (colon-separated env var) +5. Paths registered via `aborist.qa.inspect.register_metaphor_dictionary(path)` + +A cue is a token that survives one of four morphological tests against the union — closed-class preposition (`amidst, despite, beneath, ...`), `-ly` adverb whose stem is in the union, `-ing` participle whose verb stem is in the union (with consonant-de-doubling for `running → run`), or `-est` superlative whose adjective stem is in the union (with i↔y swap for `rockiest → rocky`). The rule is *derived*, not hand-curated — same Phase-2 lesson the `concept_relations` synonym layer learned. Adding `butterfly` to a hand exception list scales poorly; trusting `butterf` not in `/usr/share/dict/words` to filter `butterfly` scales by definition. + +**A forest with its own vocabulary** — domain-specific jargon (aviation, medical, legal, dynastic) registers its own wordlist once and the suffix tests pick up domain-specific adverbs / participles automatically: + +```sh +# Per-process: env var +export ABORIST_METAPHOR_DICTS=/path/to/aviation-terms.txt:/path/to/medical-terms.txt +.venv/bin/aborist inspect --cache-key +``` + +```python +# Programmatic: register once, applies to all subsequent diagnoses +from aborist.qa.inspect import register_metaphor_dictionary +register_metaphor_dictionary("/path/to/aviation-terms.txt") +``` + +Each supplemental dict is one word per line; blank lines and case-mixed entries are tolerated. The cue suffix tests then resolve domain stems (`aerodynamic → aerodynamically`, `widgetize → widgetizing`) without code changes. + ## License License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists. diff --git a/aborist/qa/inspect.py b/aborist/qa/inspect.py index aaf982e..ebc3d70 100644 --- a/aborist/qa/inspect.py +++ b/aborist/qa/inspect.py @@ -15,6 +15,7 @@ verbs, never feed back into the hard chain. from __future__ import annotations import json +import os import sqlite3 import unicodedata from pathlib import Path @@ -521,28 +522,89 @@ _METAPHOR_PREPOSITION_CUES = frozenset({ "throughout", "alongside", "betwixt", "atop", }) -_DICT_WORDS_PATH = Path("/usr/share/dict/words") +# Default wordlist sources. The standard Unix `/usr/share/dict/words` +# (and its Debian split into american-english + british-english) is +# the morphological substrate. Both spelling regions get unioned so a +# British speaker's metaphor cues (`colouredly`, `realisingly`) land +# alongside American (`coloredly`, `realizingly`). +# +# Operators can supplement with corpus / domain-specific vocabulary +# via the ``ABORIST_METAPHOR_DICTS`` environment variable (colon- +# separated list of paths, one word per line) or the +# ``register_metaphor_dictionary(path)`` helper below. A forest with +# its own jargon (aviation, medical, legal, dynastic) registers the +# domain wordlist once and the suffix tests pick up domain-specific +# adverbs / participles automatically. Same Phase-2 derivation rule +# applies — `aerodynamically` → stem `aerodynamic` lookup → if in the +# union, classified as adverbial. +_DEFAULT_DICT_PATHS = ( + Path("/usr/share/dict/words"), # OS default symlink + Path("/usr/share/dict/american-english"), # Debian split + Path("/usr/share/dict/british-english"), # Debian split +) _english_wordlist_cache: frozenset[str] | None = None +_extra_dict_paths: list[Path] = [] + + +def register_metaphor_dictionary(path: str | Path) -> None: + """Register a supplemental wordlist (one word per line, lowercase + or mixed-case, blank lines OK). Words union into the default set + on next ``_english_wordlist()`` call. Use for corpus / domain + vocabulary so the metaphor sidecar's morphological tests can pick + up domain-specific stems. + + Programmatic equivalent of the ``ABORIST_METAPHOR_DICTS`` env var. + Calling this invalidates the cache so subsequent lookups re-build + the union. Idempotent — re-registering the same path is a no-op. + """ + global _english_wordlist_cache + p = Path(path) + if p not in _extra_dict_paths: + _extra_dict_paths.append(p) + _english_wordlist_cache = None + + +def _load_dict(path: Path) -> set[str]: + """Read ``path`` if it exists; return lowercase token set.""" + if not path.exists(): + return set() + try: + text = path.read_text(encoding="utf-8", errors="replace") + except OSError: + return set() + return {w.strip().lower() for w in text.splitlines() if w.strip()} def _english_wordlist() -> frozenset[str]: - """Load /usr/share/dict/words once; return as a lowercase frozenset. + """Union of the default Unix wordlists + supplemental dictionaries. - Graceful degradation: if the wordlist doesn't exist (some - containers, BSDs, Windows), return an empty set — every cue - returns False, the sidecar quietly returns ``no_signal``. + Sources (in order, all unioned): + 1. ``/usr/share/dict/words`` (OS default) + 2. ``/usr/share/dict/american-english`` (Debian split) + 3. ``/usr/share/dict/british-english`` (Debian split) + 4. Paths in ``ABORIST_METAPHOR_DICTS`` (colon-separated env var) + 5. Paths registered via ``register_metaphor_dictionary()`` + + Cached on first call. Graceful degradation: missing paths skip + silently; empty union returns an empty frozenset and the suffix + tests all return False (sidecar returns ``no_signal`` quietly). """ global _english_wordlist_cache - if _english_wordlist_cache is None: - if not _DICT_WORDS_PATH.exists(): - _english_wordlist_cache = frozenset() - else: - _english_wordlist_cache = frozenset( - w.strip().lower() - for w in _DICT_WORDS_PATH.read_text(encoding="utf-8", errors="replace").splitlines() - if w.strip() - ) + if _english_wordlist_cache is not None: + return _english_wordlist_cache + union: set[str] = set() + for p in _DEFAULT_DICT_PATHS: + union |= _load_dict(p) + env_paths = os.environ.get("ABORIST_METAPHOR_DICTS", "") + if env_paths: + for raw in env_paths.split(":"): + raw = raw.strip() + if raw: + union |= _load_dict(Path(raw)) + for p in _extra_dict_paths: + union |= _load_dict(p) + _english_wordlist_cache = frozenset(union) return _english_wordlist_cache diff --git a/tests/test_inspect.py b/tests/test_inspect.py index c0390ba..f2f45de 100644 --- a/tests/test_inspect.py +++ b/tests/test_inspect.py @@ -684,3 +684,83 @@ def test_metaphor_deflection_under_threshold_returns_no_signal(): ) assert d["kind"] == "no_signal" assert d["cue_count"] < 3 + + +# ───────────────────────────────────────────────────────────────────── +# Wordlist configurability — supplemental dictionaries +# ───────────────────────────────────────────────────────────────────── + + +def test_register_metaphor_dictionary_unions_into_wordlist(tmp_path): + """Custom wordlist registers, unions into the default, and the + suffix tests pick up domain-specific stems.""" + import aborist.qa.inspect as m + + # Reset cache so the test sees a clean slate. + saved_cache = m._english_wordlist_cache + saved_extra = list(m._extra_dict_paths) + m._english_wordlist_cache = None + m._extra_dict_paths = [] + + try: + wl_default = m._english_wordlist() + size_default = len(wl_default) + + custom = tmp_path / "domain.txt" + custom.write_text("foofoogeneous\nzaplet\n") + m.register_metaphor_dictionary(custom) + + wl_after = m._english_wordlist() + assert "foofoogeneous" in wl_after + assert "zaplet" in wl_after + assert len(wl_after) >= size_default + 2 + + # Suffix tests now classify domain-specific adverbs. + assert m._is_adverbial_ly("foofoogeneously") # stem in custom dict + finally: + m._english_wordlist_cache = saved_cache + m._extra_dict_paths = saved_extra + + +def test_aborist_metaphor_dicts_env_var_supplements(monkeypatch, tmp_path): + """Setting ABORIST_METAPHOR_DICTS=path1:path2 unions both into + the default wordlist on first lookup.""" + import aborist.qa.inspect as m + + saved_cache = m._english_wordlist_cache + m._english_wordlist_cache = None + + try: + d1 = tmp_path / "d1.txt" + d1.write_text("widgetspeak\n") + d2 = tmp_path / "d2.txt" + d2.write_text("frizzlebop\n") + monkeypatch.setenv("ABORIST_METAPHOR_DICTS", f"{d1}:{d2}") + + wl = m._english_wordlist() + assert "widgetspeak" in wl + assert "frizzlebop" in wl + finally: + m._english_wordlist_cache = saved_cache + + +def test_register_metaphor_dictionary_idempotent(tmp_path): + """Registering the same path twice is a no-op (cache invalidates + once, second call is a no-op since path already in list).""" + import aborist.qa.inspect as m + + saved_extra = list(m._extra_dict_paths) + saved_cache = m._english_wordlist_cache + m._extra_dict_paths = [] + m._english_wordlist_cache = None + + try: + custom = tmp_path / "d.txt" + custom.write_text("uniquewidget\n") + m.register_metaphor_dictionary(custom) + m.register_metaphor_dictionary(custom) # second call + # Path appears once. + assert m._extra_dict_paths.count(custom) == 1 + finally: + m._extra_dict_paths = saved_extra + m._english_wordlist_cache = saved_cache