9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
7bb11ed62f
|
#000048 step 2.4 — parse_pointer_claims clause segmentation
Closes the 8 mis-segments #000046 left in formulate-hard-v1.jsonl. The parser was line/bullet-only — one line ⇒ one claim — so a line that crammed several pointered claims onto one row ("Water is wet [E1]; fire is hot [E2]", "X happened [E1]. Y followed [E2]") became one monolithic claim with all the pointers, and a wrapped bullet became two. arborist/qa/parse_claims.py: _SEGMENT_SEP_RE splits a line on ';', sentence boundaries ('. '/'! '/'? ' then a Capital), spaced dashes (' - '/' — '/' – '), ' and '/' or '/' because '/' although '/' since '/' while ', inline '(N)' enumeration markers, and commas — with '(?![^\[]*\])' so a comma inside a [E1, E2] bracket never splits it. _segment_line keeps the split ONLY IF every resulting non-empty segment is a well-pointered claim — a legit single claim ("The cat is black and white [E1].", "The cast: A, B, C [E1].") is never broken because splitting it would manufacture pointer-less prose fragments → guard rejects; a leading colon-terminated header with no pointer ("Two facts:", "Key points:") is allowed and dropped. Plus a wrapped-bullet join: a continuation line (leading whitespace then a lowercase letter, no bullet glyph) folds its text + pointers into the previous claim. Effect: formulate-hard rate 4/12 → 12/12 (the pack is now at ceiling — a harder Formulate tier would re-open below-ceiling headroom; a #000046 follow-up). Remaining #000048 headroom: 2 STRICT_PARAPHRASE recombinations in falsification-hard (Mercury, Einstein — step 2.2). Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells; parse_pointer_claims feeds the 450 claim_lattice_pointer + claim_lattice cells) after (bench/qa_results/2026-05-11T20-26-37Z) vs the pre-step-2.4 baseline (...T17-12-41Z = HEAD's parse_claims.py). STRICT-rate quote 0.54→0.55, pointer 0.22→0.22, lattice 0.43→0.45 — all within the 5-pp noise floor. Per-row diff: the segmenter changed the parsed-claim count on the SAME answer text for 7 of the 450 lattice cells (0 in claim_lattice, 7 in claim_lattice_pointer); of those, 2 caused an audit_mode change — both correct: a wrap-join recovered an answer's intended structure (4 claims, 2 pointer-less wrap-fragments → HYBRID) into 2 well-pointered claims → STRICT; and a crammed-one-line blob (1 monolithic claim, all pointers → STRICT) split into 8 claims, some not individually verifying → HYBRID (the honest verdict — false- positive STRICT was the corruption). Every other lattice/quote delta is LLM re-answer variance. No regression — the segmenter's only visible effects on real traffic are honest improvements. Summarized in qa-modes-bench.md Addendum 7 + ticket-000048 §5 step 2.4. Tests: 8 new in test_claim_lattice.py (semicolon/sentence/conjunction splits; pointerless-fragment + cast-list guards; leading-colon-header drop; wrapped-bullet join; pointer-order/multi-pointer); existing parse_pointer_claims tests pass untouched; test_5f_formulate_hard_pack re-pinned 4/12 → 12/12. make test 2358 passed, 28 skipped. #000048 → steps 2.1 + 2.4 landed; #000046 / #000012 §8 / TICKETS.md / Makefile / fixture _meta + notes updated. |
|||
|
9899a33b7b
|
#000048 step 2.1 — verify_quotes entity salient-token-disagreement gate
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a multi-word proper noun matches the source — but "Insulin was discovered by Alexander Fleming" against "Penicillin was discovered by Alexander Fleming" matches on the shared "Alexander Fleming" while the swapped subject "Insulin" (the falsehood) is ignored. arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx) flags a >4-char Capitalized content token (stopword-filtered) or a digit-number in the answer absent from the source. _is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in verify_quotes' entity branch (proximity policy) in the weakest-grounding slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is what keeps a structured multi-claim summary untouched — the Matrix cast list (many entities, a tight cluster) and the TMNT answer (a numbered list with parenthetical nicknames the source omits): model-added accurate detail in a real summary isn't a contradiction, only the single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid entity-path regression tests still pass, pinned untouched. Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin / 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard pack around) and 5f-fal-live-028 — had expected_reason updated HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE recombinations (Mercury, Einstein — step 2.2) + 8 Formulate mis-segments (step 2.4). Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after (bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline (...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54, pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise floor. Per-row diff (675 common cells, 30 quote-mode rows changed audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity path — the gate fired on 0 legitimate QA answers in the whole bench. Every transition was LLM re-answer variance (verifier quote→quote with the verdict flipping); pointer/lattice deltas are noise too (the gate is in verify_quotes / quote mode, not the claim-lattice verifier). No regression — the gate is provably narrow on real traffic. Summarized in qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1. Tests: 4 new in test_verify.py (_is_single_sentence helper, _entity_salient_disagrees helper, swapped-subject → UNGROUNDED, gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated (positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5 dilution; ACCEPT via a degraded-parent sub-scenario). make test 2343 passed, 28 skipped. #000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md / Makefile / fixture _meta + notes / baseline JSON updated. |
|||
|
724f433790
|
#000046 closed — verify_quotes paraphrase numeric-agreement gate (Phase 3)
Closes the harder-fixture-tier ticket: a real verify_quotes
tightening lifts the falsification-hard rate 4/12 → 6/12, bench-gated,
ForkScore's bench-Δ goes positive on it — the loop is closed
end-to-end.
arborist/qa/verify.py: _numeric_signature(text) extracts comma-
stripped digit-runs ('8,849' and '8849' collapse; '300' stays
distinct from '300000' ← '300,000'). _check_each_with_paraphrase
gains a gate: a span that token-covers the source ≥ paraphrase_coverage
but asserts a digit-number the source lacks (modulo thousands-comma)
is no longer paraphrase-grounded — it goes to unverified. Catches the
near-miss the lexical coverage check is blind to ("Water boils at 50
degrees" against a source saying 100 token-covers 100% because
'50'/'100' aren't >4-char content tokens). Narrow by construction:
fires only on the paraphrase fallback (verbatim/span/entity/claim-
lattice paths untouched), only on a digit-number. A rounding-
paraphrase demoting here is the honest verdict — it isn't a verbatim
grounding.
Hard pack: lifts 5f-fal-hard-004 (50 vs 100) and -007 (300 vs
300,000) to UNGROUNDED → falsification-hard rate 4/12 → 6/12 = 0.5.
The other 6 over-grounds (4 HYBRID_ENTITY + 2 recombined-no-number
STRICT_PARAPHRASE) and the Formulate hard pack are unaffected —
headroom for a bigger order/dependency-aware verifier upgrade, an
optional follow-up, not a #000046 blocker.
Bench gate: make bench-qa (n=3 × 75 questions × 3 modes = 675 cells)
before / after. STRICT-rate quote 0.53→0.50, pointer 0.23→0.25,
lattice 0.44→0.45 — all within the 5-pp noise floor. Per-row diff
(675 common cells, 74 changed audit_mode): the only clearly
gate-attributable QA shift was the fictional "our cold fusion
breakthrough" year-claim demoting STRICT→HYBRID (×3 samples) — a
correct demotion; every other transition was quote→quote /
claim_lattice→claim_lattice LLM re-answer variance. No regression on
legit answers. Artifacts: bench/qa_results/2026-05-11T13-42-38Z (before)
and ...T14-19-51Z (after) — gitignored; summarized in qa-modes-bench.md
Addendum 5 + ticket-000046 §5 Phase 3.
Worked example: fork_score on the real change — parent {5f/
falsification: 4/12} → child {5f/falsification: 6/12} → γ·Δ5f =
(1/6)/5 ≈ +0.033 > 0 (positive; a single improvement of this size is
MARGINAL by the ÷5 dilution, the test pins the full-lift-to-1.0 case
at ACCEPT).
Tests: 5 new in tests/test_verify.py (numeric_signature normalization
+ subset-matches-comma-variant + disagreement-rejected +
gate-is-narrow + number-present-still-verifies);
test_5f_falsification_hard_pack_below_ceiling re-pinned 4/12 → 6/12.
make test 2339 passed, 28 skipped.
#000046 → closed; #000012 §8 §4 + TICKETS.md row + falsification-hard
_meta / hard-004,007 notes + Makefile 8/12→6/12 comments updated.
(Makefile also carries an uncommitted chain-check-SQL improvement from
the concurrent session — NOT in this commit; staged only the
#000046-comment hunks.)
|
|||
|
8d6961fcc1
|
aborist/arborist
modified: .gitlab-ci.yml modified: bench/qa_questions.txt modified: bench/qa_sweep.py modified: bench/run.sh modified: docs/TICKETS.md modified: docs/_source/README.md modified: docs/_source/_ext/makefile_targets.py modified: docs/_source/api/cli.rst modified: docs/_source/api/distill.rst modified: docs/_source/api/mesh.rst modified: docs/_source/api/qa.rst modified: docs/_source/api/retrieval.rst modified: docs/_source/api/storage.rst modified: docs/_source/api/substrate.rst modified: docs/_source/concepts.rst modified: docs/_source/conf.py modified: docs/_source/cookbook.rst modified: docs/_source/index.rst modified: docs/_source/license.rst modified: docs/_source/quickstart.rst modified: docs/bench-maxing.md modified: docs/benchmarks.md modified: docs/cti-architecture.md modified: docs/diagrams/aborist-modules.dot modified: docs/diagrams/aborist-modules.svg modified: docs/diagrams/mesh-data-flow.dot modified: docs/diagrams/mesh-epoch-lifecycle.dot modified: docs/diagrams/mesh-epoch-lifecycle.svg modified: docs/diagrams/mesh-group-decisions.dot modified: docs/diagrams/mesh-group-decisions.svg modified: docs/diagrams/mesh-identity-stack.dot modified: docs/diagrams/mesh-secret-envelope.dot modified: docs/mesh.md modified: docs/qa-modes-bench.md modified: docs/seven-point-program.md modified: docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md modified: docs/tickets/ticket-000002-reference-frame-polarity-contract.md modified: docs/tickets/ticket-000003-anchor-class-warrant.md modified: docs/tickets/ticket-000005-label-ladder-migration.md modified: docs/tickets/ticket-000006-bench-emergent-findings.md modified: docs/tickets/ticket-000007-query-layer-hyphen-fold.md modified: docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md modified: docs/tickets/ticket-000009-quantifier-preflight-dag-binding.md modified: docs/tickets/ticket-000010-metacognition-preflight-guard.md modified: docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md modified: scripts/backfill_concepts.py modified: scripts/bench_emergent.py modified: tests/crawler/test_async_web_fetcher.py modified: tests/crawler/test_bridge.py modified: tests/crawler/test_web_fetch.py modified: tests/test_bench_qa_sweep.py modified: tests/test_burn.py modified: tests/test_burn_doc.py modified: tests/test_claim_lattice.py modified: tests/test_cli_render.py modified: tests/test_compress.py modified: tests/test_concepts.py modified: tests/test_dag.py modified: tests/test_directives.py modified: tests/test_distill.py modified: tests/test_distill_recursive.py modified: tests/test_evict.py modified: tests/test_frame.py modified: tests/test_grok_source.py modified: tests/test_html_source.py modified: tests/test_ingest.py modified: tests/test_inspect.py modified: tests/test_journal.py modified: tests/test_keys.py modified: tests/test_llm_context_base.py modified: tests/test_merkle.py modified: tests/test_mesh.py modified: tests/test_mesh_aead.py modified: tests/test_mesh_chain.py modified: tests/test_mesh_cli.py modified: tests/test_mesh_cli_pull.py modified: tests/test_mesh_wire.py modified: tests/test_mesh_wire_e2e.py modified: tests/test_metacognition.py modified: tests/test_migration_audit_mode.py modified: tests/test_providence_source.py modified: tests/test_qa.py modified: tests/test_qa_quality_live.py modified: tests/test_quantifier_caps.py modified: tests/test_quantifier_classifier.py modified: tests/test_quantifier_phase4.py modified: tests/test_quantifier_reminder.py modified: tests/test_query.py modified: tests/test_reclassify.py modified: tests/test_repair.py modified: tests/test_resume.py modified: tests/test_snapshot.py modified: tests/test_soft_preflight.py modified: tests/test_tfidf.py modified: tests/test_vcs_source.py modified: tests/test_verify.py modified: tests/test_verify_json.py modified: tests/test_versioned_ingest.py modified: tests/test_warrant.py modified: tests/test_wikipedia_old.py modified: tests/test_wikipedia_xml.py modified: tests/test_wikitext.py |
|||
|
621f0b2cda
|
docs+code: 5-task fan-out — preflight_hash field, --show-preflight CLI, frame plumbing, metacog bench, #000011
Fan-out execution of the deferred-but-not-blocking pile from prior status reports. #000009 §7.2 — bench harness preflight_hash field: - aborist/qa/query.py surfaces `preflight_hash` on result dict (miss path, reject path, and cache-hit path via new helper `_extract_preflight_hash_from_blob` that pulls the stage hash out of persisted run_dag_blob). - bench/qa_sweep.py adds 12-char preflight_hash prefix to bench rows. Mirrors cache_key truncation pattern. Operators can grep / SQL-filter bench JSONL by preflight policy state. - 3 new tests in tests/test_dag.py for the extract helper. #000009 §7.2 — `aborist providence --show-preflight CACHE_KEY_PREFIX`: - New CLI flag pulls the preflight stage payload from a row's run_dag_blob. Match by 12-char prefix. Renders preflight stage hash + run-DAG stage list. Operator tool for inspecting which policy state governed a cached row. - Live verified on a real cache row (8a212fecb2a9 — current CEO of OpenAI question, 10-stage CTI shape with preflight at idx 1). - Legacy rows (predating #000009) report a clean fall-through message: "run_dag has no preflight stage (predates #000009)". #000010 §12.6 — reference-frame plumbing into QuestionState: - Pre-retrieval preflight runs with reference_frames=() (frame_detection needs source titles, not available yet). Post-retrieval, query.py re-runs preflight_question() with the detected frames so the result-dict + run-DAG QuestionState carry frame-aware logical_statuses (specifically `reference_frame_ambiguous` when 2+ frames match). - Live verified on Orwell-style question; logical_statuses now correctly includes `reference_frame_ambiguous` in the result. Metacog-trigger bench fixture (#000010 §13.3): - bench/qa_questions_metacog_subset.txt — 6 questions, one per detector kind plus a well-formed control. - Bench artifact 2026-05-04T02-18-42Z. Detector accuracy 6/6 on fixture; 2 of 5 trigger questions return STRICT on lattice mode despite metacog warning (JSON STRICT on George-Washington-stop-being-president-of-France false-premise + uploaded-contract out-of-corpus questions). Audit-line tails correctly surface the warnings. - qa-modes-bench.md Addendum 4 captures the per-question matrix + interpretation. #000010 §13.3 cross-references with bench artifact stamp. #000011 SOFT_PREFLIGHT_HINT design ticket opened: - docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md captures the design proposal per #000010 §18 / source doc. Implementation deferred — design only. - Sidecar would add model-assisted preflight as a soft signal (`SOFT_FALSE_PREMISE_SUSPECTED` etc.) that NEVER enters the verifier hard path. Strict guardrail: cannot create PREFLIGHT_OK or PREFLIGHT_BLOCKED without deterministic support. - Validated by §13.3 finding: deterministic detectors flag correctly; corpus-accidental grounding produces 2/5 STRICT on trigger questions; soft sidecar would add independent semantic skepticism. - TICKETS.md index row added; Next ID bumped to 000012. 996 tests passing (3 new for the extract helper). Cross-doc consistency: - qa-modes-bench.md Addenda 1+2+3+4 chronological - #000010 §13.1 (broad subset) + §13.2 (full bench) + §13.3 (metacog trigger subset) - #000011 design captured but not implemented |
|||
|
3bd36f36c8
|
bench: full 75-question regression check — no regression from #000010
Addendum 3 to qa-modes-bench.md. Validates that flipping quantifier_reminder_enabled=True for lattice modes (per #000010 §12.10 / §13.1) doesn't regress narrow-question performance. Prior validation (Addendum 2) covered the 9-question broad subset only. This run sweeps the full 75-question bench/qa_questions.txt (~10% broad, ~89% narrow), 225 runs per mode, comparing against the frozen 2026-05-02T15-07Z baseline. Findings: Mode | Pre-flip SR | Post-flip SR | Δ ----------+-------------+--------------+-------- quote | 0.54 | 0.52 | -2pp (within 5pp floor) pointer | 0.20 | 0.21 | +1pp JSON | 0.42 | 0.44 | +2pp Mean ratio: -1/+2/+3pp — all within noise band. Substrate-level wins beyond the headline metrics: - pointer FORMAT_COLLAPSED: 0/225 across the full sweep. Reminder eliminates collapse mode globally, not just on broad. - pointer NO_EVIDENCE_POINTER: 13% (vs 33% on broad-only when reminder was off). Citation discipline propagates beyond the rows where the reminder text actually fires — the model's session attention reinforces. - JSON mean ratio +3pp consistent with broad-subset finding. Quote-mode is essentially unchanged because it's mode-gated off the guard by default. Latency 33-35s/call this evening (vs 17-19s in prior runs) is endpoint contention, not a substrate regression — preflight adds zero LLM calls. Verdict: no regression. The default flip ships clean across the full corpus. Defaults stay on. The substrate is strictly more honest (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without sacrificing throughput on non-broad questions. Bench artifact: bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}. |
|||
|
cd0e8ef64f
|
bench(#000010): preflight on-vs-off validation — defaults stay on
Post-landing validation cell. Same 9-question broad subset as §12.6 reminder-only baseline; this run flips both metacognition_enabled and quantifier_reminder_enabled to False so we can isolate the preflight contribution. Metric | OFF | ON | Δ --------------------------+-----------+----------+---------- pointer mean ratio | 0.483 | 0.643 | +16pp JSON mean ratio | 0.570 | 0.735 | +17pp JSON UNGROUNDED rate | 7/27 | 1/27 | -22pp pointer FORMAT_COLLAPSED | 2/27 | 0/27 | -100% any-mode STRICT-rate | within ±11pp noise (27-sample n=3 floor) Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all clear the 5pp signal floor on lattice modes. STRICT-rate moves are within Hermes nondeterminism. The #000010 default flip is doing what was claimed. On this subset none of the metacognition detectors fire (no temporal / contradiction / false-premise / out-of-corpus shapes), so the delta effectively isolates the reminder contribution from #000008. A metacog-trigger subset bench is deferred. Documentation: - docs/qa-modes-bench.md Addendum 2 captures the comparison table + interpretation + verdict. - docs/tickets/ticket-000010-... §13.1 cross-references with bench artifact stamp. Bench artifact: bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}. |
|||
|
9780cca4d3
|
docs: §12.8/§12.9 cap+reminder verdict + cross-doc updates
#000008 §12.8 — Cap+reminder A/B (2026-05-03T12-54-11Z, 81 runs): Metric | Base | Rem | Cap | Cap+Rem --------------------+------+------+------+-------- JSON SR | 0.19 | 0.22 | 0.33 | 0.30 ← cap-only wins SR JSON UNGROUNDED | 7 | 1 | 6 | 1 ← rem dominates U-rescue pointer mean ratio | 0.473| 0.643| 0.516| 0.684 ← cap+rem best pointer FORMAT_COLL | 2 | 0 | 2 | 0 ← rem-driven §10.8 strict gate "compound beats either alone by ≥5pp on every metric" NOT cleanly met. Cap+reminder beats reminder-only by +8pp on JSON SR (clears floor) and beats cap-only by +17pp on pointer mean ratio (clears floor), but is -3pp vs cap-only on JSON SR (regression, within noise). #000008 §12.9 — Final verdict + recommendation: Mechanism asymmetry (clean signal): Reminder rescues UNGROUNDED → HYBRID Cap rescues HYBRID → STRICT Recommendation: Option A — single-knob defaults, per-mode tailored: claim_lattice (JSON): apply_caps=True + reminder=True claim_lattice_pointer: apply_caps=False + reminder=True (cap can't rescue pointer-tag discipline upstream of cap; cap fires 20× without verdict gain) quote: guard mode-gated off (already default) Caveat: n=3 × 9 = 27/cell variance is ~3-4pp; recommend n=5 verification on cap+reminder before flipping defaults. Cross-doc updates: - CLAUDE.md: architecture diagram now lists the three new Phase 1-3 modules (quantifier.py, model_profiles.py, quantifier_reminder.py). New "Broad-quantifier preflight guard" conventions entry covers the 7 policy fields, six-level disable hierarchy, dry-run defaults, CLI flags, and §12 bench summary. - docs/qa-modes-bench.md: addendum at end pointing at #000008 §12 four-cell A/B + 4 bench artifact stamps. Original 2026-05-02 journal frozen; 2026-05-03 broad-subset findings flagged as question-mix-dependent narrative on top of the global per-mode recommendation. |
|||
|
2bbd12a270
|
docs(bench): rename qa-modes-bench-2026-05-02.md → qa-modes-bench.md
The 2026-05-02 journal is the LIVING bench doc — it absorbs each
day's bench results and rolls forward. Pinning a date in the
filename made it look like a frozen snapshot when it is in fact
the working journal.
Naming pattern moving forward:
docs/qa-modes-bench.md — living journal (latest run)
docs/qa-modes-bench-2026-04-30.md — historical snapshot
(frozen for the JSON-mode
hardening day)
Future dated snapshots stay dated. The bare 'qa-modes-bench.md'
is always the current state of the substrate.
Updated references in:
- CLAUDE.md (× 2)
- aborist/qa/prompts.py
- aborist/qa/query.py
- docs/bench-maxing.md
The 2026-04-30 references in docs/cti-architecture.md,
docs/test-coverage-audit-2026-05-01.md, docs/verifier-semantic-gap-design.md,
docs/TICKETS.md correctly point at the historical snapshot and
stay as-is.
|
Renamed from docs/qa-modes-bench-2026-05-02.md (Browse further)