Commit graph

5 commits

Author SHA1 Message Date
8d6961fcc1
aborist/arborist
modified:   .gitlab-ci.yml
	modified:   bench/qa_questions.txt
	modified:   bench/qa_sweep.py
	modified:   bench/run.sh
	modified:   docs/TICKETS.md
	modified:   docs/_source/README.md
	modified:   docs/_source/_ext/makefile_targets.py
	modified:   docs/_source/api/cli.rst
	modified:   docs/_source/api/distill.rst
	modified:   docs/_source/api/mesh.rst
	modified:   docs/_source/api/qa.rst
	modified:   docs/_source/api/retrieval.rst
	modified:   docs/_source/api/storage.rst
	modified:   docs/_source/api/substrate.rst
	modified:   docs/_source/concepts.rst
	modified:   docs/_source/conf.py
	modified:   docs/_source/cookbook.rst
	modified:   docs/_source/index.rst
	modified:   docs/_source/license.rst
	modified:   docs/_source/quickstart.rst
	modified:   docs/bench-maxing.md
	modified:   docs/benchmarks.md
	modified:   docs/cti-architecture.md
	modified:   docs/diagrams/aborist-modules.dot
	modified:   docs/diagrams/aborist-modules.svg
	modified:   docs/diagrams/mesh-data-flow.dot
	modified:   docs/diagrams/mesh-epoch-lifecycle.dot
	modified:   docs/diagrams/mesh-epoch-lifecycle.svg
	modified:   docs/diagrams/mesh-group-decisions.dot
	modified:   docs/diagrams/mesh-group-decisions.svg
	modified:   docs/diagrams/mesh-identity-stack.dot
	modified:   docs/diagrams/mesh-secret-envelope.dot
	modified:   docs/mesh.md
	modified:   docs/qa-modes-bench.md
	modified:   docs/seven-point-program.md
	modified:   docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md
	modified:   docs/tickets/ticket-000002-reference-frame-polarity-contract.md
	modified:   docs/tickets/ticket-000003-anchor-class-warrant.md
	modified:   docs/tickets/ticket-000005-label-ladder-migration.md
	modified:   docs/tickets/ticket-000006-bench-emergent-findings.md
	modified:   docs/tickets/ticket-000007-query-layer-hyphen-fold.md
	modified:   docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md
	modified:   docs/tickets/ticket-000009-quantifier-preflight-dag-binding.md
	modified:   docs/tickets/ticket-000010-metacognition-preflight-guard.md
	modified:   docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md
	modified:   scripts/backfill_concepts.py
	modified:   scripts/bench_emergent.py
	modified:   tests/crawler/test_async_web_fetcher.py
	modified:   tests/crawler/test_bridge.py
	modified:   tests/crawler/test_web_fetch.py
	modified:   tests/test_bench_qa_sweep.py
	modified:   tests/test_burn.py
	modified:   tests/test_burn_doc.py
	modified:   tests/test_claim_lattice.py
	modified:   tests/test_cli_render.py
	modified:   tests/test_compress.py
	modified:   tests/test_concepts.py
	modified:   tests/test_dag.py
	modified:   tests/test_directives.py
	modified:   tests/test_distill.py
	modified:   tests/test_distill_recursive.py
	modified:   tests/test_evict.py
	modified:   tests/test_frame.py
	modified:   tests/test_grok_source.py
	modified:   tests/test_html_source.py
	modified:   tests/test_ingest.py
	modified:   tests/test_inspect.py
	modified:   tests/test_journal.py
	modified:   tests/test_keys.py
	modified:   tests/test_llm_context_base.py
	modified:   tests/test_merkle.py
	modified:   tests/test_mesh.py
	modified:   tests/test_mesh_aead.py
	modified:   tests/test_mesh_chain.py
	modified:   tests/test_mesh_cli.py
	modified:   tests/test_mesh_cli_pull.py
	modified:   tests/test_mesh_wire.py
	modified:   tests/test_mesh_wire_e2e.py
	modified:   tests/test_metacognition.py
	modified:   tests/test_migration_audit_mode.py
	modified:   tests/test_providence_source.py
	modified:   tests/test_qa.py
	modified:   tests/test_qa_quality_live.py
	modified:   tests/test_quantifier_caps.py
	modified:   tests/test_quantifier_classifier.py
	modified:   tests/test_quantifier_phase4.py
	modified:   tests/test_quantifier_reminder.py
	modified:   tests/test_query.py
	modified:   tests/test_reclassify.py
	modified:   tests/test_repair.py
	modified:   tests/test_resume.py
	modified:   tests/test_snapshot.py
	modified:   tests/test_soft_preflight.py
	modified:   tests/test_tfidf.py
	modified:   tests/test_vcs_source.py
	modified:   tests/test_verify.py
	modified:   tests/test_verify_json.py
	modified:   tests/test_versioned_ingest.py
	modified:   tests/test_warrant.py
	modified:   tests/test_wikipedia_old.py
	modified:   tests/test_wikipedia_xml.py
	modified:   tests/test_wikitext.py
2026-05-07 09:31:49 -04:00
de07ad9392
docs: distill #000008+#000009+#000010 into core docs + diagrams
Three Explore agents fanned out in parallel for a docs/ + diagrams/
+ code-comment audit against the shipped state of the three
preflight tickets. This commit lands all the alignment fixes.

Core docs updates:

  CLAUDE.md
    - dag.py module description: stage counts now read
      "7/8 quote · 9/10 CTI · 3 reject" reflecting #000009 preflight
      stage + reject-broad early-return shape.

  docs/cti-architecture.md §2.2 + §2.3
    - §2.3 Merkle-AGI-DAG section rewritten: documents all five DAG
      shapes (legacy 7/9, post-#000009 8/10, reject-broad 3),
      describes the preflight stage's 5 nested CTI clauses
      (classifier / answer_contract / prompt_contract /
      evidence_contract / policy_refs), pins
      PREFLIGHT_NODE_VERSION = "preflight-node-v1", states the
      audit-replay payoff.
    - §2.2 CTI section: adds the four new modules
      (quantifier, model_profiles, quantifier_reminder,
      metacognition) as code anchors. Notes that pre-answer
      preflight contract extends CTI upstream of retrieval.

  docs/seven-point-program.md
    - D3 status ½ → ¾ — pre-answer preflight contract landed via
      #000008 + #000010. Code anchors + pinning tests updated.
    - D4 status ½ → ¾ — preflight stage adds upstream control
      commitment to the run-DAG. Code anchors include
      build_reject_run_dag + preflight_node_hash.
    - Status snapshot table: tickets column now references
      #000008/#000009/#000010 against D1/D3/D4 directives.
    - "Post-landing addendum (2026-05-03 / 2026-05-04)" subsection
      summarises all three tickets + their commit shas + final
      test count (993 passing, up from 734).

  docs/modules.md
    - Q&A pipeline table: added 4 new modules (quantifier.py,
      model_profiles.py, quantifier_reminder.py, metacognition.py).
      dag.py row updated to "7/8 quote · 9/10 CTI · 3 reject".
    - dag.py subsection rewritten: documents all 5 DAG shapes,
      describes the preflight payload's 5 clauses + question_state.
    - 4 new module subsections (quantifier / model_profiles /
      quantifier_reminder / metacognition) explaining each
      module's purpose, signature, and how it feeds the run-DAG
      preflight clause.

Diagram updates:

  docs/diagrams/query-pipeline.dot + .svg
    - New "PREFLIGHT (#000008 + #000010)" node inserted between
      cache_check and concepts_lookup.
    - New "REJECT-BROAD" node showing the 3-stage minimal DAG
      escape path.
    - render node label extended with the audit-line tail token
      catalog.

  docs/diagrams/aborist-modules.dot + .svg
    - 4 new qa_* nodes in the retrieval & verifier cluster.
    - 8 new edges: qa_query/qa_runner each call into all 4
      preflight modules; qa_dag has dotted edges to qa_quantifier
      + qa_metacognition (preflight clause sources).
    - qa_dag label updated to mention preflight_node_hash + 5 clauses.

  docs/diagrams/verifier-ladder.dot + .svg
    - Soft-demote violations list extended: BROAD_QUANTIFIER_RUNAWAY
      / CAP_APPLIED / SCOPE_UNBOUND, FORMAT_COLLAPSED, BARE_NAME_CLAIM.
    - New "AUDIT-LINE TAILS" annotation node listing all 11 tail
      tokens (#000008 broad-* + #000010 metacog + classic verifier).
    - Dashed edges from each rung to tails note showing tails
      compose onto labels.

Code-side stale-comment fixes (caught by 3rd Explore agent):

  aborist/qa/keys.py:218
    - "The four fields" → "The seven fields"; mention #000010 adds
      six more for metacognition.
  aborist/qa/query.py:2644
    - 7-stage / 9-stage comment expanded to enumerate all four
      base+preflight shapes plus the 3-stage reject path.
  aborist/qa/runner.py:835
    - same expansion as query.py for runner.ask() callsite.

mesh-*.dot, ingest-pipeline.dot, qa-modes-bench.md, bench-maxing.md,
bench-emergent-design.md, verifier-semantic-gap-design.md,
self-reference-design.md, concept-relations-design.md confirmed
orthogonal — no edits needed.

993 tests still passing (no behavior change). 7 files modified
across docs/ + 3 dot diagrams + 3 SVGs + 4 code-comment fixes.
2026-05-03 19:18:22 -04:00
7444f106fc
docs: tickets/ subdir + delete naming-deferral, ref-frame, coverage-audit
Two cleanup operations bundled (separate scopes, single commit
since they share the doc-tree settle):

1. Move ticket-NNNNNN-<slug>.md files into docs/tickets/. The
   directory makes browsing the design log easier; the index
   stays at docs/TICKETS.md (top-level pointer). Convention text
   in TICKETS.md updated to spell the new path.

2. Delete three docs whose load-bearing content has either been
   absorbed into the codebase or distilled into closed tickets:

   - docs/naming-deferral.md (147 lines) — explained why we
     don't rename claim_lattice → CTI/PROMETHEUS-Σ. Decision
     stays in place; the rationale is no longer worth a
     dedicated doc. Inline citations removed from
     cti-architecture.md (4 refs), warrant.py, ticket-000003
     (closed-ticket internal ref).
   - docs/reference-frame-failure-class.md (169 lines) — Orwell
     case journal that motivated the phrase-pattern retrieval
     route. The route shipped; the analysis is now duplicate
     with the closed Ticket #000002. Inline citation removed
     from CLAUDE.md retrieval pipeline section + frame.py.
   - docs/test-coverage-audit-2026-05-01.md (46 lines) —
     point-in-time audit checking 16/16 of fox's §11 list. Tests
     themselves live in tests/; the audit was a one-shot
     checkmark exercise.

References updated:
  CLAUDE.md, aborist/qa/frame.py, aborist/qa/retrieval_plan.py,
  aborist/qa/warrant.py, docs/cti-architecture.md, docs/TICKETS.md,
  docs/tickets/ticket-000003 + ticket-000004 (internal links).

Net: -362 lines + tickets/ subdir. 751/34 tests still pass.
2026-05-02 12:51:09 -04:00
39c3652e0a
docs: consolidate self-reference design — flat MVP + fact-Core distillation
The two self-reference design docs were a sequential pair:

  self-reference-thought-chains-design.md  (96 lines, v1 MVP that
                                            shipped via 8de0044)
  self-reference-distillation-design.md    (208 lines, v2 fact-Core
                                            successor, design-only)

v2 explicitly named v1 as 'flat MVP' that 'falls short' and itself
as 'successor' — they're one story told in two files. Merge into

  docs/self-reference-design.md  (230 lines)

Structure: §1 problem statement, §2 v1 flat MVP (shipped),
§3 v2 fact-Core distillation (proposal), §4 v2 implementation
plan, §5 deliberate non-goals, §6 bench impact, §7 architectural
payoff. One narrative; the v1/v2 demarcation stays clear.

Net: 304 → 230 lines (-74), one file instead of two.

References updated in:
  CLAUDE.md, aborist/cli.py, aborist/qa/query.py,
  aborist/qa/runner.py, aborist/sources/providence.py,
  docs/TICKETS.md, docs/cti-architecture.md (3 refs),
  tests/test_providence_source.py

10/10 test_providence_source tests still pass.
2026-05-02 12:39:42 -04:00
5990834ce8
arch: layer-cake docs + verifier_policy_hash + old-maps principle
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).

NEW DOCS
--------
docs/cti-architecture.md
  Maps today's modules onto the architectural layer cake fox named
  in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
  reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
  (evidence direction) / v9.8 Providence (admissibility ledger) /
  Hermes (weak proposer). Per-layer responsibility table + module
  map showing which existing files implement which layer. The
  architecture is real; the labels above name what's already there.

docs/naming-deferral.md
  Argues why we keep internal terms (claim_lattice,
  claim_lattice_pointer, verify_claim_lattice) instead of renaming
  to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
  cache invalidation, ~150 test fixture references, schema CHECK
  migration, blame-history disconnect, mesh peer coordination.
  The bridge is the cti-architecture mapping doc — read it once,
  then read code in code's vocabulary and prose in prose's
  vocabulary. Lists four triggers that would invert the decision.

docs/self-reference-distillation-design.md
  Successor to docs/self-reference-thought-chains-design.md (the
  flat-source MVP). Maps STRICT claims onto the existing Distiller
  / Core / derivations infrastructure: each STRICT record becomes
  a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
  Merkle inclusion proofs back to Wikipedia source_roots. The
  fact-graph grows; new claims attach by inheriting the proof
  chain. CompositionDistiller (combining facts into new claims)
  is explicitly deferred — that's reasoning machinery, not
  infrastructure.

docs/test-coverage-audit-2026-05-01.md
  Maps fox's §11 test list (16 items) against the codebase. 16/16
  covered. Documents item #9's manual-quote-rule lifecycle: the
  rule was retired from pointer mode in commit 224bfd6 after the
  bench journey identified it was rejecting factually correct
  claims; retained in JSON variant where the punctuation-rationale
  argument doesn't apply. The audit doc itself is the requirements-
  drift defense.

CODE
----
aborist/qa/keys.py:verifier_policy_hash
  New pure function. Hashes the verifier-relevant subset of policy
  (answer_mode + claim_lattice_* verifier knobs + entity_policy
  fields + base_version). Folded into cache_key as an optional
  9th dimension via a new keyword arg with backward-compat default
  None — legacy 8-dim callers continue to work and produce the
  pre-2026-05-01 hash result.

aborist/qa/{runner,query}.py
  Compute verifier_policy_hash from the policy_variant and pass
  to cache_key as the 9th dim. Records written from this commit
  forward bind to the verifier-policy identity. Pre-existing 8-dim
  records become cache-misses on next lookup — same migration cost
  as any governance_policy_hash bump. The 9-dim form makes the
  question "did the verifier rules change?" answerable from
  cache_key diff alone, separate from "did the prompt change?"

CLAUDE.md
  Adds "old maps vs runtime maps" engineering discipline to the
  bench-maxing section. Codifies the principle: every base model
  carries old maps from training time; the runtime carries the
  fresh map; when they disagree, the runtime wins. Move authority
  OUT of the model's prior and INTO runtime artifacts (pointer IDs
  the runtime mints, source spans the runtime interpolates by
  offset, evidence maps assembled per query, policy hashes that
  fold prompt/verifier/retrieval into cache_key, hard checks run
  by the verifier). Hermes' content-addressed-evidence-id
  hallucination (commit bb8450d) is the canonical case study.

TESTS
-----
tests/test_keys.py
  Six new tests for verifier_policy_hash:
    - only hashes verifier subset (non-verifier fields don't change it)
    - changes when verifier-relevant field changes
    - empty-policy → stable
    - 9-dim cache_key distinct from 8-dim
    - 9-dim distinct under different verifier hashes
    - 8-dim form preserved for legacy callers (None == omit)

DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
  logic in runner.py + query.py already IS PROMETHEUS-Σ; an
  explicit prometheus.py is a refactor with no behavior change.
  Defer until a §5-rename-trigger fires.

507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
2026-05-01 12:10:29 -04:00