arborist/scripts
russell@unturf.com d3ad520529
journal: emit unfirehose/1.0 JSONL for queries + bench cycles
aborist now writes one JSONL session per `make query` invocation
and per `bench-emergent` cycle to:

    ~/.aborist/unfirehose/{project-slug}/{session-uuid}.jsonl

Unfirehose's native-harness auto-discovery picks up any
~/.{name}/unfirehose/ directory (see ingest.ts:discoverNativeHarnesses)
without registration — once a session lands, the unfirehose watcher
debounces, ingests, and exposes it in the dashboard alongside
Claude Code / Fetch / uncloseai sessions.

Schema: unfirehose/1.0 (per ~/git/unfirehose-nextjs-logger/docs/
unfirehose-schema.md). Each session file:

    line 1   type=session  (header — id, projectId, firstPrompt,
                            harness="aborist", harnessVersion)
    line 2   type=message role=user
    line 3   type=message role=assistant
                          content=[text]
                          model=hermes-3-llama-3.1-8b-fp8-dynamic
                          provider=hermes
                          durationMs=<wall>
                          aborist_meta={audit_mode, n_verified/n_quotes,
                            cache_key, cache_status, lookup_path,
                            violations, sources, timings_ms, answer_mode}
    line 4   type=message role=system subtype=session_end durationMs

aborist-specific extras (verifier verdict, sources, timings) ride
under namespaced ``aborist_meta`` so the canonical fields stay clean
for off-the-shelf consumers; per the spec, unknown fields are
ignored downstream.

Bench-emergent cycles emit an additional system init message at
the start of each session noting the 3 random words, marking the
session as a generator-driven cycle vs a normal user query.

Failure-isolation: journal write is wrapped in a broad try/except
at every call site. A journaling bug must NEVER break the query
or bench loop.

Tests: 10 new in tests/test_journal.py (slug encoding, session
header, parent-id chain, session_end on close, aborist_meta
passthrough, usage block, idempotent close). Full suite: 663 passed.

Live verified: `make query Q="what is photosynthesis?"` produced
a 4-line JSONL with STRICT 3/3, all sources + timings populated,
ready for unfirehose ingestion.
2026-05-02 15:19:39 -04:00
..
backfill_concepts.py make: speed up tests + automate concept backfill (test 38s→11s) 2026-05-02 10:11:16 -04:00
bench_emergent.py journal: emit unfirehose/1.0 JSONL for queries + bench cycles 2026-05-02 15:19:39 -04:00