3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
8d6961fcc1
|
aborist/arborist
modified: .gitlab-ci.yml modified: bench/qa_questions.txt modified: bench/qa_sweep.py modified: bench/run.sh modified: docs/TICKETS.md modified: docs/_source/README.md modified: docs/_source/_ext/makefile_targets.py modified: docs/_source/api/cli.rst modified: docs/_source/api/distill.rst modified: docs/_source/api/mesh.rst modified: docs/_source/api/qa.rst modified: docs/_source/api/retrieval.rst modified: docs/_source/api/storage.rst modified: docs/_source/api/substrate.rst modified: docs/_source/concepts.rst modified: docs/_source/conf.py modified: docs/_source/cookbook.rst modified: docs/_source/index.rst modified: docs/_source/license.rst modified: docs/_source/quickstart.rst modified: docs/bench-maxing.md modified: docs/benchmarks.md modified: docs/cti-architecture.md modified: docs/diagrams/aborist-modules.dot modified: docs/diagrams/aborist-modules.svg modified: docs/diagrams/mesh-data-flow.dot modified: docs/diagrams/mesh-epoch-lifecycle.dot modified: docs/diagrams/mesh-epoch-lifecycle.svg modified: docs/diagrams/mesh-group-decisions.dot modified: docs/diagrams/mesh-group-decisions.svg modified: docs/diagrams/mesh-identity-stack.dot modified: docs/diagrams/mesh-secret-envelope.dot modified: docs/mesh.md modified: docs/qa-modes-bench.md modified: docs/seven-point-program.md modified: docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md modified: docs/tickets/ticket-000002-reference-frame-polarity-contract.md modified: docs/tickets/ticket-000003-anchor-class-warrant.md modified: docs/tickets/ticket-000005-label-ladder-migration.md modified: docs/tickets/ticket-000006-bench-emergent-findings.md modified: docs/tickets/ticket-000007-query-layer-hyphen-fold.md modified: docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md modified: docs/tickets/ticket-000009-quantifier-preflight-dag-binding.md modified: docs/tickets/ticket-000010-metacognition-preflight-guard.md modified: docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md modified: scripts/backfill_concepts.py modified: scripts/bench_emergent.py modified: tests/crawler/test_async_web_fetcher.py modified: tests/crawler/test_bridge.py modified: tests/crawler/test_web_fetch.py modified: tests/test_bench_qa_sweep.py modified: tests/test_burn.py modified: tests/test_burn_doc.py modified: tests/test_claim_lattice.py modified: tests/test_cli_render.py modified: tests/test_compress.py modified: tests/test_concepts.py modified: tests/test_dag.py modified: tests/test_directives.py modified: tests/test_distill.py modified: tests/test_distill_recursive.py modified: tests/test_evict.py modified: tests/test_frame.py modified: tests/test_grok_source.py modified: tests/test_html_source.py modified: tests/test_ingest.py modified: tests/test_inspect.py modified: tests/test_journal.py modified: tests/test_keys.py modified: tests/test_llm_context_base.py modified: tests/test_merkle.py modified: tests/test_mesh.py modified: tests/test_mesh_aead.py modified: tests/test_mesh_chain.py modified: tests/test_mesh_cli.py modified: tests/test_mesh_cli_pull.py modified: tests/test_mesh_wire.py modified: tests/test_mesh_wire_e2e.py modified: tests/test_metacognition.py modified: tests/test_migration_audit_mode.py modified: tests/test_providence_source.py modified: tests/test_qa.py modified: tests/test_qa_quality_live.py modified: tests/test_quantifier_caps.py modified: tests/test_quantifier_classifier.py modified: tests/test_quantifier_phase4.py modified: tests/test_quantifier_reminder.py modified: tests/test_query.py modified: tests/test_reclassify.py modified: tests/test_repair.py modified: tests/test_resume.py modified: tests/test_snapshot.py modified: tests/test_soft_preflight.py modified: tests/test_tfidf.py modified: tests/test_vcs_source.py modified: tests/test_verify.py modified: tests/test_verify_json.py modified: tests/test_versioned_ingest.py modified: tests/test_warrant.py modified: tests/test_wikipedia_old.py modified: tests/test_wikipedia_xml.py modified: tests/test_wikitext.py |
|||
|
4f2b5a6685
|
qa(#000008): §12.10 n=5 verification + §12.11 defaults flipped (Option A)
n=5 verification of cap+reminder cell (135 runs):
Metric | n=3 | n=5
--------------------+-----------+------------
JSON SR | 0.30 | 0.33 ← matches cap-only
JSON UNGROUNDED rate| 1/27 (4%) | 2/45 (4%) ← matches reminder-only
pointer SR | 0/27 | 0/45 ← unchanged across all cells
The §12.8 0.30 was Hermes nondeterminism. n=5 confirms cap+reminder
delivers cap-only's STRICT-rate AND reminder-only's UNGROUNDED-rescue.
§10.8 strict gate met at n=5:
vs reminder-only on JSON SR: +11pp (clears floor)
vs cap-only on JSON UNGROUNDED: -18pp (clears floor)
vs cap-only on ptr mean ratio: +12pp (clears floor)
Defaults flipped — Option A landing (per-mode tailored):
quantifier_reminder_enabled False → True
(load-bearing on both lattice modes)
NEW field: quantifier_apply_caps_modes = ["claim_lattice"]
(allowlist for which modes apply caps
when apply_caps=True; JSON-only since
cap-on-pointer is wasted noise per
§12.10 0/45 STRICT data)
quantifier_guard_apply_caps False → False (UNCHANGED)
(operator opts in via
--apply-quantifier-caps; preserves
§10.11.3 dry-run discipline)
Cap-application gate now reads:
if apply_caps AND mode in apply_caps_modes AND cap is not None:
effective_max_claims = looked_up_cap
quantifier_apply_caps_modes folded into _VERIFIER_POLICY_FIELDS so
flipping the allowlist invalidates prior cache records.
5 new tests pin: reminder default ON for both runner.DEFAULT_POLICY
and query.DEFAULT_QUERY_POLICY; apply_caps_modes default
["claim_lattice"]; governance-hash invalidation on allowlist flip;
apply_caps default still False (dry-run preserved).
920 tests passing (5 new); 36 skipped.
Operator behavior:
$ aborist query "winners of all major sports?"
→ reminder ON, cap OFF (default after this commit)
$ aborist query --apply-quantifier-caps "..."
→ cap applies on claim_lattice (JSON) only
$ aborist query --apply-quantifier-caps \
--policy quantifier_apply_caps_modes='["claim_lattice","claim_lattice_pointer"]' "..."
→ Option D for one call
Phase 5 (run-DAG node binding for quantifier_preflight) and
cross-model Qwen/GPT-4 verification remain as follow-ups per §11.11.
|
|||
|
84d5b5cd76
|
qa(#000008): Phase 2 — model-profile caps + governance hash (dry-run)
Lands aborist/qa/model_profiles.py with two profiles:
- adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic
tight caps for broad intensities (ALL=8, COMPREHENSIVE=5,
OPEN_REQUEST=5) reflecting the 2026-05-02 runaway case.
- default
large-reasoner-class fallback (ALL=12, COMPREHENSIVE=15,
OPEN_REQUEST=12). Used when model_profile_id has no entry.
cap_for_intensity() resolves caps via three-source chain:
1. policy_overrides (per-call dict, highest priority)
2. per-model profile from PROFILES
3. "default" profile fallback
EXPLICIT_COUNT sentinel handles SMALL_NUM_EXPLICIT and
COMPARATIVE_BOUND — cap is the question's explicit count, not a
profile-set value. Defensive fallback to MANY cap if classifier
fired the rung without extracting a count.
Four new policy fields, all folded into governance_policy_hash via
_VERIFIER_POLICY_FIELDS:
- quantifier_guard_enabled master kill (default True)
- quantifier_guard_apply_caps dry-run gate (default False per
§10.11.3 — cap LOOKED UP and reported
on result, but NOT applied to the
verifier until operator flips True)
- quantifier_caps_by_intensity per-call override dict
- quantifier_guard_modes per-mode opt-in list (default
["claim_lattice_pointer",
"claim_lattice"]; quote opts out)
Six-level disable hierarchy (§10.11.2) implemented:
- Per-test: policy={"quantifier_guard_enabled": False}
- Per-call: --no-quantifier-guard (Phase 4)
- Per-phase: each policy switch is independent
- Per-mode: quantifier_guard_modes filter
- Per-model: model_profiles.py lookup
- Master: governance_policy_hash invalidation on flip
Wired through both query() and runner.ask() — both compute
effective_max_claims from the (classifier_intensity, model_profile,
policy_overrides) triple and pass it as max_claims_per_answer to
the verifier. Dry-run mode keeps effective_max_claims at the policy
default (12) until apply_caps flips True.
Result dict surfaces claim_cap_applied (the LOOKED-UP cap, even in
dry-run) plus all Phase-1 quantifier fields on miss-path AND
cache-hit path so bench rows stay column-aligned.
19 new tests pin: per-model selection, EXPLICIT_COUNT sentinel,
override precedence, governance-hash invalidation on every cap
field, profile shape (all ten rungs covered), default profile
presence.
|
