qa(inspect): wordlist union (US+UK) + configurable supplemental dictionaries
Three changes that shape the same lever:
(1) The metaphor-cue wordlist now unions /usr/share/dict/words +
/usr/share/dict/american-english + /usr/share/dict/british-english.
The Debian split made the prior 'just symlink to american-english'
miss British spellings (colour, organisation, realise) which
silently became false negatives on British-speaker questions.
Union: 102,485 → 104,305 entries on this machine. ~1,820 added
British-specific entries.
(2) Supplemental dictionary support: operators can layer
domain-specific vocabulary into the morphological substrate.
Two paths:
- Env var: ABORIST_METAPHOR_DICTS=/path/a:/path/b
- Programmatic: register_metaphor_dictionary(path)
Each supplemental dict is one word per line. The cue suffix
tests (-ly stem, -ing stem, -est stem) then resolve domain
stems automatically — adding 'aerodynamic' to a custom dict
makes 'aerodynamically' classify as adverbial without code
changes.
Use case: 'a tree with its own vocabulary' — an aviation
forest, a medical corpus, a legal-domain shard each carries
jargon the standard wordlist doesn't cover. Register once,
suffix tests pick up domain stems forever.
(3) README gains a 'Sidecar diagnostics' section with a table of
the three sidecars (deflection, title-relevance, metaphor-
deflection) plus a 'Metaphor-deflection cue dictionary' subsection
explaining the derivation rule, the load order, and the per-
forest vocabulary configurability. Architecturally documents
why the rule is *derived* from the union (Phase-2 lesson) and
not hand-curated.
3 new tests in tests/test_inspect.py:
- register_metaphor_dictionary unions a custom path's words
- ABORIST_METAPHOR_DICTS env var supplements with two paths
- re-registering same path is idempotent
763/34 tests pass.
This commit is contained in:
parent
5f2d0b3fa7
commit
e5ffa6c6fd
3 changed files with 196 additions and 14 deletions
40
README.md
40
README.md
|
|
@ -502,6 +502,46 @@ The default suite never hits the network. The crawler suite is gated behind `mak
|
|||
|
||||
`make bench-qa` writes JSONL + markdown into `bench/qa_results/<utc-stamp>.{jsonl,md}` (gitignored); design-log entries live in `docs/qa-modes-bench-<date>.md`. The bench is stop/start-able via `--resume <jsonl-path>` (same `--seed` required for shuffled-task-order alignment). `BENCH_QA_CONCURRENCY=N` Makefile variable overrides the default `4`. See `docs/bench-maxing.md` for the full speed playbook.
|
||||
|
||||
## Sidecar diagnostics
|
||||
|
||||
The verifier stays binary; soft signals proliferate as **sidecars** — read-only diagnostic functions that pull the same source chunks the verifier saw, classify spans / detect topic shifts / surface metaphor framing, and never write to `providence_cache` or extend the audit chain. Three live today:
|
||||
|
||||
| sidecar | flags | typical use |
|
||||
|---------|-------|-------------|
|
||||
| `diagnose_deflection(question, answer)` | model deflected — answer omits the question's subject anchor | adversarial-premise questions where the model answers an adjacent grounded question |
|
||||
| `diagnose_title_relevance(claim, cited_titles)` | claim's stems share zero overlap with cited source titles | retrieval-driven hallucinations where a chunk's source is structurally unrelated to the claim |
|
||||
| `diagnose_metaphor_deflection(question, answer)` | question carried strong poetic framing the answer ignored | metaphorical questions answered with literal taxonomic / definitional facts |
|
||||
|
||||
Run the per-record diagnostic via `aborist inspect --cache-key <hex>` (CLI: `make inspect KEY=...`). Sidecars run on demand, observe state, report — they participate in no proof path.
|
||||
|
||||
### Metaphor-deflection cue dictionary
|
||||
|
||||
`diagnose_metaphor_deflection` derives its poetic-cue vocabulary from a configurable union of wordlists. The default load order:
|
||||
|
||||
1. `/usr/share/dict/words` (OS default)
|
||||
2. `/usr/share/dict/american-english` (Debian split, if present)
|
||||
3. `/usr/share/dict/british-english` (Debian split, if present)
|
||||
4. Paths in `ABORIST_METAPHOR_DICTS` (colon-separated env var)
|
||||
5. Paths registered via `aborist.qa.inspect.register_metaphor_dictionary(path)`
|
||||
|
||||
A cue is a token that survives one of four morphological tests against the union — closed-class preposition (`amidst, despite, beneath, ...`), `-ly` adverb whose stem is in the union, `-ing` participle whose verb stem is in the union (with consonant-de-doubling for `running → run`), or `-est` superlative whose adjective stem is in the union (with i↔y swap for `rockiest → rocky`). The rule is *derived*, not hand-curated — same Phase-2 lesson the `concept_relations` synonym layer learned. Adding `butterfly` to a hand exception list scales poorly; trusting `butterf` not in `/usr/share/dict/words` to filter `butterfly` scales by definition.
|
||||
|
||||
**A forest with its own vocabulary** — domain-specific jargon (aviation, medical, legal, dynastic) registers its own wordlist once and the suffix tests pick up domain-specific adverbs / participles automatically:
|
||||
|
||||
```sh
|
||||
# Per-process: env var
|
||||
export ABORIST_METAPHOR_DICTS=/path/to/aviation-terms.txt:/path/to/medical-terms.txt
|
||||
.venv/bin/aborist inspect --cache-key <hex>
|
||||
```
|
||||
|
||||
```python
|
||||
# Programmatic: register once, applies to all subsequent diagnoses
|
||||
from aborist.qa.inspect import register_metaphor_dictionary
|
||||
register_metaphor_dictionary("/path/to/aviation-terms.txt")
|
||||
```
|
||||
|
||||
Each supplemental dict is one word per line; blank lines and case-mixed entries are tolerated. The cue suffix tests then resolve domain stems (`aerodynamic → aerodynamically`, `widgetize → widgetizing`) without code changes.
|
||||
|
||||
## License
|
||||
|
||||
License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists.
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ verbs, never feed back into the hard chain.
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import unicodedata
|
||||
from pathlib import Path
|
||||
|
|
@ -521,28 +522,89 @@ _METAPHOR_PREPOSITION_CUES = frozenset({
|
|||
"throughout", "alongside", "betwixt", "atop",
|
||||
})
|
||||
|
||||
_DICT_WORDS_PATH = Path("/usr/share/dict/words")
|
||||
# Default wordlist sources. The standard Unix `/usr/share/dict/words`
|
||||
# (and its Debian split into american-english + british-english) is
|
||||
# the morphological substrate. Both spelling regions get unioned so a
|
||||
# British speaker's metaphor cues (`colouredly`, `realisingly`) land
|
||||
# alongside American (`coloredly`, `realizingly`).
|
||||
#
|
||||
# Operators can supplement with corpus / domain-specific vocabulary
|
||||
# via the ``ABORIST_METAPHOR_DICTS`` environment variable (colon-
|
||||
# separated list of paths, one word per line) or the
|
||||
# ``register_metaphor_dictionary(path)`` helper below. A forest with
|
||||
# its own jargon (aviation, medical, legal, dynastic) registers the
|
||||
# domain wordlist once and the suffix tests pick up domain-specific
|
||||
# adverbs / participles automatically. Same Phase-2 derivation rule
|
||||
# applies — `aerodynamically` → stem `aerodynamic` lookup → if in the
|
||||
# union, classified as adverbial.
|
||||
_DEFAULT_DICT_PATHS = (
|
||||
Path("/usr/share/dict/words"), # OS default symlink
|
||||
Path("/usr/share/dict/american-english"), # Debian split
|
||||
Path("/usr/share/dict/british-english"), # Debian split
|
||||
)
|
||||
|
||||
_english_wordlist_cache: frozenset[str] | None = None
|
||||
_extra_dict_paths: list[Path] = []
|
||||
|
||||
|
||||
def register_metaphor_dictionary(path: str | Path) -> None:
|
||||
"""Register a supplemental wordlist (one word per line, lowercase
|
||||
or mixed-case, blank lines OK). Words union into the default set
|
||||
on next ``_english_wordlist()`` call. Use for corpus / domain
|
||||
vocabulary so the metaphor sidecar's morphological tests can pick
|
||||
up domain-specific stems.
|
||||
|
||||
Programmatic equivalent of the ``ABORIST_METAPHOR_DICTS`` env var.
|
||||
Calling this invalidates the cache so subsequent lookups re-build
|
||||
the union. Idempotent — re-registering the same path is a no-op.
|
||||
"""
|
||||
global _english_wordlist_cache
|
||||
p = Path(path)
|
||||
if p not in _extra_dict_paths:
|
||||
_extra_dict_paths.append(p)
|
||||
_english_wordlist_cache = None
|
||||
|
||||
|
||||
def _load_dict(path: Path) -> set[str]:
|
||||
"""Read ``path`` if it exists; return lowercase token set."""
|
||||
if not path.exists():
|
||||
return set()
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
return set()
|
||||
return {w.strip().lower() for w in text.splitlines() if w.strip()}
|
||||
|
||||
|
||||
def _english_wordlist() -> frozenset[str]:
|
||||
"""Load /usr/share/dict/words once; return as a lowercase frozenset.
|
||||
"""Union of the default Unix wordlists + supplemental dictionaries.
|
||||
|
||||
Graceful degradation: if the wordlist doesn't exist (some
|
||||
containers, BSDs, Windows), return an empty set — every cue
|
||||
returns False, the sidecar quietly returns ``no_signal``.
|
||||
Sources (in order, all unioned):
|
||||
1. ``/usr/share/dict/words`` (OS default)
|
||||
2. ``/usr/share/dict/american-english`` (Debian split)
|
||||
3. ``/usr/share/dict/british-english`` (Debian split)
|
||||
4. Paths in ``ABORIST_METAPHOR_DICTS`` (colon-separated env var)
|
||||
5. Paths registered via ``register_metaphor_dictionary()``
|
||||
|
||||
Cached on first call. Graceful degradation: missing paths skip
|
||||
silently; empty union returns an empty frozenset and the suffix
|
||||
tests all return False (sidecar returns ``no_signal`` quietly).
|
||||
"""
|
||||
global _english_wordlist_cache
|
||||
if _english_wordlist_cache is None:
|
||||
if not _DICT_WORDS_PATH.exists():
|
||||
_english_wordlist_cache = frozenset()
|
||||
else:
|
||||
_english_wordlist_cache = frozenset(
|
||||
w.strip().lower()
|
||||
for w in _DICT_WORDS_PATH.read_text(encoding="utf-8", errors="replace").splitlines()
|
||||
if w.strip()
|
||||
)
|
||||
if _english_wordlist_cache is not None:
|
||||
return _english_wordlist_cache
|
||||
union: set[str] = set()
|
||||
for p in _DEFAULT_DICT_PATHS:
|
||||
union |= _load_dict(p)
|
||||
env_paths = os.environ.get("ABORIST_METAPHOR_DICTS", "")
|
||||
if env_paths:
|
||||
for raw in env_paths.split(":"):
|
||||
raw = raw.strip()
|
||||
if raw:
|
||||
union |= _load_dict(Path(raw))
|
||||
for p in _extra_dict_paths:
|
||||
union |= _load_dict(p)
|
||||
_english_wordlist_cache = frozenset(union)
|
||||
return _english_wordlist_cache
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -684,3 +684,83 @@ def test_metaphor_deflection_under_threshold_returns_no_signal():
|
|||
)
|
||||
assert d["kind"] == "no_signal"
|
||||
assert d["cue_count"] < 3
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
# Wordlist configurability — supplemental dictionaries
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_register_metaphor_dictionary_unions_into_wordlist(tmp_path):
|
||||
"""Custom wordlist registers, unions into the default, and the
|
||||
suffix tests pick up domain-specific stems."""
|
||||
import aborist.qa.inspect as m
|
||||
|
||||
# Reset cache so the test sees a clean slate.
|
||||
saved_cache = m._english_wordlist_cache
|
||||
saved_extra = list(m._extra_dict_paths)
|
||||
m._english_wordlist_cache = None
|
||||
m._extra_dict_paths = []
|
||||
|
||||
try:
|
||||
wl_default = m._english_wordlist()
|
||||
size_default = len(wl_default)
|
||||
|
||||
custom = tmp_path / "domain.txt"
|
||||
custom.write_text("foofoogeneous\nzaplet\n")
|
||||
m.register_metaphor_dictionary(custom)
|
||||
|
||||
wl_after = m._english_wordlist()
|
||||
assert "foofoogeneous" in wl_after
|
||||
assert "zaplet" in wl_after
|
||||
assert len(wl_after) >= size_default + 2
|
||||
|
||||
# Suffix tests now classify domain-specific adverbs.
|
||||
assert m._is_adverbial_ly("foofoogeneously") # stem in custom dict
|
||||
finally:
|
||||
m._english_wordlist_cache = saved_cache
|
||||
m._extra_dict_paths = saved_extra
|
||||
|
||||
|
||||
def test_aborist_metaphor_dicts_env_var_supplements(monkeypatch, tmp_path):
|
||||
"""Setting ABORIST_METAPHOR_DICTS=path1:path2 unions both into
|
||||
the default wordlist on first lookup."""
|
||||
import aborist.qa.inspect as m
|
||||
|
||||
saved_cache = m._english_wordlist_cache
|
||||
m._english_wordlist_cache = None
|
||||
|
||||
try:
|
||||
d1 = tmp_path / "d1.txt"
|
||||
d1.write_text("widgetspeak\n")
|
||||
d2 = tmp_path / "d2.txt"
|
||||
d2.write_text("frizzlebop\n")
|
||||
monkeypatch.setenv("ABORIST_METAPHOR_DICTS", f"{d1}:{d2}")
|
||||
|
||||
wl = m._english_wordlist()
|
||||
assert "widgetspeak" in wl
|
||||
assert "frizzlebop" in wl
|
||||
finally:
|
||||
m._english_wordlist_cache = saved_cache
|
||||
|
||||
|
||||
def test_register_metaphor_dictionary_idempotent(tmp_path):
|
||||
"""Registering the same path twice is a no-op (cache invalidates
|
||||
once, second call is a no-op since path already in list)."""
|
||||
import aborist.qa.inspect as m
|
||||
|
||||
saved_extra = list(m._extra_dict_paths)
|
||||
saved_cache = m._english_wordlist_cache
|
||||
m._extra_dict_paths = []
|
||||
m._english_wordlist_cache = None
|
||||
|
||||
try:
|
||||
custom = tmp_path / "d.txt"
|
||||
custom.write_text("uniquewidget\n")
|
||||
m.register_metaphor_dictionary(custom)
|
||||
m.register_metaphor_dictionary(custom) # second call
|
||||
# Path appears once.
|
||||
assert m._extra_dict_paths.count(custom) == 1
|
||||
finally:
|
||||
m._extra_dict_paths = saved_extra
|
||||
m._english_wordlist_cache = saved_cache
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue