aborist now writes one JSONL session per `make query` invocation
and per `bench-emergent` cycle to:
~/.aborist/unfirehose/{project-slug}/{session-uuid}.jsonl
Unfirehose's native-harness auto-discovery picks up any
~/.{name}/unfirehose/ directory (see ingest.ts:discoverNativeHarnesses)
without registration — once a session lands, the unfirehose watcher
debounces, ingests, and exposes it in the dashboard alongside
Claude Code / Fetch / uncloseai sessions.
Schema: unfirehose/1.0 (per ~/git/unfirehose-nextjs-logger/docs/
unfirehose-schema.md). Each session file:
line 1 type=session (header — id, projectId, firstPrompt,
harness="aborist", harnessVersion)
line 2 type=message role=user
line 3 type=message role=assistant
content=[text]
model=hermes-3-llama-3.1-8b-fp8-dynamic
provider=hermes
durationMs=<wall>
aborist_meta={audit_mode, n_verified/n_quotes,
cache_key, cache_status, lookup_path,
violations, sources, timings_ms, answer_mode}
line 4 type=message role=system subtype=session_end durationMs
aborist-specific extras (verifier verdict, sources, timings) ride
under namespaced ``aborist_meta`` so the canonical fields stay clean
for off-the-shelf consumers; per the spec, unknown fields are
ignored downstream.
Bench-emergent cycles emit an additional system init message at
the start of each session noting the 3 random words, marking the
session as a generator-driven cycle vs a normal user query.
Failure-isolation: journal write is wrapped in a broad try/except
at every call site. A journaling bug must NEVER break the query
or bench loop.
Tests: 10 new in tests/test_journal.py (slug encoding, session
header, parent-id chain, session_end on close, aborist_meta
passthrough, usage block, idempotent close). Full suite: 663 passed.
Live verified: `make query Q="what is photosynthesis?"` produced
a 4-line JSONL with STRICT 3/3, all sources + timings populated,
ready for unfirehose ingestion.
Empirically motivated by the 2026-05-02 emergent log:
Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
the rockiest terrain, remain undeterred by the upbraiding
winds...'
A: 'The Macleay's Swallowtail butterfly is found in Eastern
Australia including the ACT, New South Wales, Queensland...'
The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.
Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.
Detection rule:
1. Extract metaphor cues from the question:
- -ly adverbs (gracefully, defiantly), excluding common
-ly nouns (butterfly, italy, july) via blocklist
- -ing present participles >=6 chars (upbraiding,
fluttering, brooding), excluding common verb -ing forms
- -est superlatives >=6 chars (rockiest, harshest)
- prepositional cues (amidst, despite, against, beneath)
2. Count overlap with answer's content tokens.
3. Fire metaphor_deflection when:
cue_count >= 3 AND answer_overlap_count == 0
The threshold is conservative; the smell only triggers on
STRONGLY poetic questions with PURELY literal answers.
Wire-up:
- aborist/qa/inspect.py:diagnose_metaphor_deflection
- bench/qa_sweep.py: rows gain metaphor_deflection_kind +
metaphor_cue_count + metaphor_overlap_count
- scripts/bench_emergent.py: same fields on emergent log rows
5 new tests in tests/test_inspect.py:
- swallowtail canary case fires metaphor_deflection
- literal questions (mona lisa) return no_signal
- questions whose answer engages cues return no_signal
- common -ly nouns (butterfly, italy, july, family) filtered
- sub-threshold cue counts return no_signal
756/34 tests pass (5 new + 751 prior).
scripts/bench_emergent.py + make bench-emergent + design doc.
Random word triangulation surfaces failure modes the curated
bench/qa_questions.txt doesn't reach.
Loop:
/usr/share/dict/words → random.sample(3) →
Hermes @ temp=0.8 weaves a creative question →
aborist student answers via query() →
append journey to bench/emergent_log.jsonl
(teacher review = separate manual step, fox brings entries to
Opus & gets judgment to append)
Word filter: ^[a-z]{5,12}$ after lowercasing. Skips short words
(too vague) + very long words (Hermes can't weave them).
Cadence: NOT every commit. ~20s per cycle (Hermes generator +
aborist student); N=10 ≈ 4 min, N=50 ≈ 17 min. Most cycles land
UNGROUNDED-by-corpus-design (random triplets rarely overlap with
2010-11 Wikipedia coverage); the interesting cases are STRICT/
HYBRID surprises and the verifier-disagreement cases the teacher
catches.
Teacher review is intentionally out of the bench script:
- separation of concerns: generation is automated, judgment is
contextual & needs the corpus-knowledge frame ("is this a
2010 Wikipedia gap or a substrate failure?")
- future flexibility: today the teacher is Claude Opus 4.7
in this conversation; tomorrow GPT-5 or a review committee.
Swapping teachers is a workflow change, not a code change.
Teacher output schema (appended to the same JSONL line):
teacher.match bool
teacher.audit_agreement agree|disagree|unsure
teacher.novelty_class known_truth_grounding | emergent_synthesis
| novel_claim | no_signal
teacher.score_0_5 0..5
teacher.bench_max_signal retrieval | warrant | prompt | nil
teacher.reasoning one sentence
teacher.reviewed_by model id
teacher.reviewed_ts unix ts
Smoke verified (N=2, seed=42): 41s wall-clock, both UNGROUNDED
(expected — random triplets rarely overlap 2010 Wikipedia).
Append-only log seeded with the smoke entries.
Future flag (not yet wired): --generator-endpoint &
--student-endpoint to swap LLM upstreams per role.
Full design + teacher protocol: docs/bench-emergent-design.md.
Three dev-loop speedups:
(1) `make test` already on -n auto via pytest-xdist (was implicit
serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
suite. Big inner-loop win.
(2) `make test-live` now also uses -n auto (live tests are
independent against the Hermes endpoint; concurrency=4 doesn't
overload it on the 17-test fixture set).
(3) `make backfill-concepts` (new) replaces the ad-hoc
`python -c "from aborist.concepts.extract import …"` invocations
fox was running by hand for the post-2026-05-02 concept-layer
backfills. Parallelizes per-shard work via multiprocessing.Pool
with CONCEPTS_WORKERS=4 (env-tunable).
Driven by scripts/backfill_concepts.py — runs every registered
extractor in EXTRACTORS (link_reciprocity, token_idf,
documents_fts) across every numeric-stem shard. Skips qa.db /
snapshots.db / crawl_*.db by default; --include-non-numeric
opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
~260s serial estimate; modest 1.4× speedup because SQLite WAL
+ FTS5 vocab queries are I/O-bound on a single SSD (4 workers
contend), but the unified UX & structured progress output are
the real wins.
(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
between bench-qa-smoke (n=1, ~30s) and full bench-qa
(~70min). Use as the inner-loop pre-commit signal.
Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.
No behavior change in the test suite or LLM pipeline; pure tooling.
