journal: emit unfirehose/1.0 JSONL for queries + bench cycles
aborist now writes one JSONL session per `make query` invocation
and per `bench-emergent` cycle to:
~/.aborist/unfirehose/{project-slug}/{session-uuid}.jsonl
Unfirehose's native-harness auto-discovery picks up any
~/.{name}/unfirehose/ directory (see ingest.ts:discoverNativeHarnesses)
without registration — once a session lands, the unfirehose watcher
debounces, ingests, and exposes it in the dashboard alongside
Claude Code / Fetch / uncloseai sessions.
Schema: unfirehose/1.0 (per ~/git/unfirehose-nextjs-logger/docs/
unfirehose-schema.md). Each session file:
line 1 type=session (header — id, projectId, firstPrompt,
harness="aborist", harnessVersion)
line 2 type=message role=user
line 3 type=message role=assistant
content=[text]
model=hermes-3-llama-3.1-8b-fp8-dynamic
provider=hermes
durationMs=<wall>
aborist_meta={audit_mode, n_verified/n_quotes,
cache_key, cache_status, lookup_path,
violations, sources, timings_ms, answer_mode}
line 4 type=message role=system subtype=session_end durationMs
aborist-specific extras (verifier verdict, sources, timings) ride
under namespaced ``aborist_meta`` so the canonical fields stay clean
for off-the-shelf consumers; per the spec, unknown fields are
ignored downstream.
Bench-emergent cycles emit an additional system init message at
the start of each session noting the 3 random words, marking the
session as a generator-driven cycle vs a normal user query.
Failure-isolation: journal write is wrapped in a broad try/except
at every call site. A journaling bug must NEVER break the query
or bench loop.
Tests: 10 new in tests/test_journal.py (slug encoding, session
header, parent-id chain, session_end on close, aborist_meta
passthrough, usage block, idempotent close). Full suite: 663 passed.
Live verified: `make query Q="what is photosynthesis?"` produced
a 4-line JSONL with STRICT 3/3, all sources + timings populated,
ready for unfirehose ingestion.
This commit is contained in:
parent
529434ec67
commit
d3ad520529
4 changed files with 504 additions and 0 deletions
|
|
@ -203,6 +203,51 @@ def append_log(entry: dict, log_path: Path) -> None:
|
|||
f.write(json.dumps(entry, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
def emit_unfirehose_session(entry: dict, model_id: str) -> None:
|
||||
"""Write one unfirehose/1.0 session for this bench cycle.
|
||||
|
||||
Same auto-discovery path as `aborist.cli._emit_query_journal`:
|
||||
~/.aborist/unfirehose/{slug}/{session_uuid}.jsonl. Each cycle gets
|
||||
its own session file (one prompt → one answer = one session).
|
||||
Failures must NEVER break the bench loop — wrapped at the call
|
||||
site.
|
||||
"""
|
||||
from aborist.journal import SessionWriter
|
||||
aborist_meta = {
|
||||
"audit_mode": entry.get("audit_mode"),
|
||||
"verifier_method": entry.get("verifier_method"),
|
||||
"n_quotes": entry.get("n_quotes"),
|
||||
"n_verified": entry.get("n_verified"),
|
||||
"cache_key": entry.get("cache_key"),
|
||||
"violation_kinds": entry.get("violation_kinds"),
|
||||
"metaphor_deflection_kind": entry.get("metaphor_deflection_kind"),
|
||||
"answer_mode": entry.get("answer_mode"),
|
||||
"sources": entry.get("sources"),
|
||||
"timings": {
|
||||
"question_gen_seconds": entry.get("question_gen_seconds"),
|
||||
"answer_seconds": entry.get("answer_seconds"),
|
||||
"total_seconds": entry.get("total_seconds"),
|
||||
},
|
||||
"bench": "emergent",
|
||||
"words": entry.get("words"),
|
||||
"student_error": entry.get("student_error"),
|
||||
}
|
||||
with SessionWriter(first_prompt=entry.get("question") or "") as s:
|
||||
s.system_message(
|
||||
"bench-emergent cycle: 3-word triangulation",
|
||||
subtype="init",
|
||||
aborist_meta={"words": entry.get("words"), "harness_role": "generator"},
|
||||
)
|
||||
s.user_message(entry.get("question") or "")
|
||||
s.assistant_message(
|
||||
entry.get("answer") or "",
|
||||
model=model_id,
|
||||
provider="hermes",
|
||||
duration_ms=int((entry.get("answer_seconds") or 0) * 1000) or None,
|
||||
aborist_meta=aborist_meta,
|
||||
)
|
||||
|
||||
|
||||
def print_pending(log_path: Path) -> int:
|
||||
"""Print every log entry with `teacher: None` — what fox should
|
||||
bring to a teacher model for review."""
|
||||
|
|
@ -294,6 +339,14 @@ def main(argv: list[str] | None = None) -> int:
|
|||
top_k=ns.top_k,
|
||||
)
|
||||
append_log(entry, ns.log_path)
|
||||
# Mirror to unfirehose-compatible journal so the bench cycles
|
||||
# show up in the unfirehose dashboard alongside Claude Code /
|
||||
# Fetch sessions. Best-effort — never break the bench on a
|
||||
# journal write failure.
|
||||
try:
|
||||
emit_unfirehose_session(entry, ns.model)
|
||||
except Exception: # pragma: no cover
|
||||
pass
|
||||
# Compact stdout summary so a long sweep is observable.
|
||||
audit = entry.get("audit_mode") or "?"
|
||||
ratio = (
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue